Re: Crashes under Xen with Radeon graphics card

[Juergen Gross <jgross@xxxxxxxx>]

On 15.12.23 17:04, Deucher, Alexander wrote:
> [Public]
>
> > -----Original Message-----
> > From: Juergen Gross <jgross@xxxxxxxx>
> > Sent: Friday, December 15, 2023 6:57 AM
> > To: lkml <linux-kernel@xxxxxxxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; amd-
> > gfx@xxxxxxxxxxxxxxxxxxxxx
> > Cc: Deucher, Alexander <Alexander.Deucher@xxxxxxx>; Koenig, Christian
> > <Christian.Koenig@xxxxxxx>; Pan, Xinhui <Xinhui.Pan@xxxxxxx>
> > Subject: Crashes under Xen with Radeon graphics card
> >
> > Hi,
> >
> > I recently stumbled over a test system which showed crashes probably
> > resulting from memory being overwritten randomly.
> >
> > The problem is occurring only in Dom0 when running under Xen. It seems to
> > be present since at least kernel 6.3 (I didn't go back further yet), and it 
> > seems
> > NOT to be present in kernel 5.14.
> >
> > I tracked the problem down to the initialization of the graphics card (the
> > problem might surface only later, but at least an early initialization failure 
> > made
> > the problem go away).
> >
> > # lspci
> > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > Caicos XTX [Radeon HD 8490 / R5 235X OEM]
> > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI
> > Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
> >
> > I had a working .config and one which did produce the crashes, so I narrowed
> > the problem down to detect that the important difference was in the area of
> > firmware loading (the working .config didn't have
> > CONFIG_FW_LOADER_COMPRESS_XZ set, causing firmware loading for the
> > card to fail). This was of course not the real problem, but it caused the card
> > initialization to fail.
> >
> > I manually decompressed the firmware files one by one to see whether the
> > problem would be in the decompressor or probably in the driver of the card.
> >
> > The last step without crash was:
> >
> > # dmesg | grep radeon
> > [   10.106405] [drm] radeon kernel modesetting enabled.
> > [   10.106455] radeon 0000:01:00.0: vgaarb: deactivate vga console
> > [   10.222944] radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000
> > -
> > 0x000000003FFFFFFF (1024M used)
> > [   10.252921] radeon 0000:01:00.0: GTT: 1024M 0x0000000040000000 -
> > 0x000000007FFFFFFF
> > [   10.278255] [drm] radeon: 1024M of VRAM memory ready
> > [   10.295828] [drm] radeon: 1024M of GTT memory ready.
> > [   10.295867] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_pfp.bin succeeded
> > [   10.330846] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_me.bin succeeded
> > [   10.330858] radeon 0000:01:00.0: Direct firmware load for
> > radeon/BTC_rlc.bin
> > succeeded
> > [   10.330870] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_mc.bin failed with error -2
> > [   10.380979] ni_cp: Failed to load firmware "radeon/CAICOS_mc.bin"
> > [   10.381006] [drm:evergreen_init [radeon]] *ERROR* Failed to load
> > firmware!
> > [   10.405765] radeon 0000:01:00.0: Fatal error during GPU init
> > [   10.432107] [drm] radeon: finishing device.
> > [   10.439179] [drm] radeon: ttm finalized
> > [   10.463203] radeon: probe of 0000:01:00.0 failed with error -2
> >
> > And with decompressing radeon/CAICOS_mc.bin I got:
> >
> > # dmesg | grep radeon
> > [   10.266491] [drm] radeon kernel modesetting enabled.
> > [   10.266552] radeon 0000:01:00.0: vgaarb: deactivate vga console
> > [   10.456047] radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000
> > -
> > 0x000000003FFFFFFF (1024M used)
> > [   10.470270] radeon 0000:01:00.0: GTT: 1024M 0x0000000040000000 -
> > 0x000000007FFFFFFF
> > [   10.566946] [drm] radeon: 1024M of VRAM memory ready
> > [   10.576891] [drm] radeon: 1024M of GTT memory ready.
> > [   10.586971] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_pfp.bin succeeded
> > [   10.611886] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_me.bin succeeded
> > [   10.611909] radeon 0000:01:00.0: Direct firmware load for
> > radeon/BTC_rlc.bin
> > succeeded
> > [   10.611938] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_mc.bin succeeded
> > [   10.660599] radeon 0000:01:00.0: Direct firmware load for
> > radeon/CAICOS_smc.bin failed with error -2
> > [   10.660601] smc: error loading firmware "radeon/CAICOS_smc.bin"
> You also need to make sure CAICOS_smc.bin is available.
Of course. But with all firmware files loadable the system is crashing, too.

I thought it might help to see after which firmware the crashes are starting.

> > [   10.661676] [drm] radeon: power management initialized
> > [   10.713666] radeon 0000:01:00.0: Direct firmware load for
> > radeon/SUMO_uvd.bin
> > failed with error -2
> > [   10.713668] radeon 0000:01:00.0: radeon_uvd: Can't load firmware
> > "radeon/SUMO_uvd.bin"
> > [   10.713669] radeon 0000:01:00.0: failed UVD (-2) init.
> And SUMO_uvd.bin.
Sure.

> > [   10.714787] [drm] enabling PCIE gen 2 link speeds, disable with
> > radeon.pcie_gen2=0
> > [   10.809213] radeon 0000:01:00.0: WB enabled
> > [   10.817528] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr
> > 0x0000000040000c00
> > [   10.833755] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr
> > 0x0000000040000c0c
> > [   10.850330] radeon 0000:01:00.0: radeon: MSI limited to 32-bit
> > [   10.862154] radeon 0000:01:00.0: radeon: using MSI.
> > [   10.871930] [drm] radeon: irq initialized.
> > [   11.062028] [drm] Initialized radeon 2.50.0 20080528 for 0000:01:00.0 on
> > minor 0
> > [   11.119723] [drm:radeon_dvi_detect [radeon]] *ERROR* DVI-I-1: probed a
> > monitor but no|invalid EDID
> > [   11.411370] fbcon: radeondrmfb (fb0) is primary device
> > [   11.507252] radeon 0000:01:00.0: [drm] fb0: radeondrmfb frame buffer
> > device
> > [   11.674028] [drm:radeon_dvi_detect [radeon]] *ERROR* DVI-I-1: probed a
> > monitor but no|invalid EDID
> > [   11.834317] [drm:radeon_dvi_detect [radeon]] *ERROR* DVI-I-1: probed a
> > monitor but no|invalid EDID
> > [   28.313041] snd_hda_intel 0000:01:00.1: bound 0000:01:00.0 (ops
> > radeon_audio_component_bind_ops [radeon])
> > [   44.371991] [drm:radeon_dvi_detect [radeon]] *ERROR* DVI-I-1: probed a
> > monitor but no|invalid EDID
> > [   44.428068] [drm:radeon_dvi_detect [radeon]] *ERROR* DVI-I-1: probed a
> > monitor but no|invalid EDID
> >
> > followed by a crash some seconds after the system was up.
> >
> > The crashes vary, but often the kernel accesses non-canonical addresses or
> > tries to map illegal physical addresses. Sometimes the system is just hanging,
> > either with softlockups or without any further signs of being alive.
> >
> > I can easily reproduce the problem, so any debug patches to narrow down the
> > problem are welcome.
> There are still missing firmware required for proper operation.  Please fix 
> them up.
That was the starting point, of course!

BTW, meanwhile I have tested kernel 5.19, which is working. I suspected that
the patch series merging swiotlb and swiotlb-xen could be to blame, but that
went into v5.19.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature