Re: [Xen-devel] AMD GPU passthrough in Xen

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Wednesday, September 24, 2014, 5:10:41 PM, you wrote:

> This is making more sense now.  I didn't read your first message close enough.

> I didn't realize that you are running a Linux DomU.  I run Win7 in my domU. 
> We have two completely different drivers (and groups) that support Windows vs 
> Linux.  I don't have any problems restarting a DomU with Win7.  I haven't 
> recently been using Linux in a DomU, that may be why I am not seeing the same 
> problems.

> My roots were in the Windows kernel driver team. I made the move over to 
> Xen/Linux about a year ago.  I still work very closely with the Windows team 
> and the Windows driver.  I don't work directly on the Linux Radeon driver, I 
> concentrate more on Xen/QEMU.  I will have to ask around the Linux team if 
> they can provide me with any insight.

> Thanks,
> Kelly

(i also use Xen/QEMU (i'm using a HVM guest for this)

Perhaps a difference could be the windows driver always re-initinting the 
critical parts ?

The plus side of the linux radeon driver is that it is:
- quite verbose
- opensource .. so you can tinker with it

I have tried windows guests in the past with mixed results, but since it's more 
or less a black box i changed to first trying to get vga passthroug to a linux 
guest working :-)

However i think in general it would be best if there is a way to get the device 
in a not-posted state again by xen/dom0, instead of relying on the guest to do 
the right 
thing.

--
Sander



>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@xxxxxxxxxxxxxx]
>> Sent: Wednesday, September 24, 2014 10:16 AM
>> To: Zytaruk, Kelly
>> Cc: Peter Kay; xen-devel@xxxxxxxxxxxxx; Gordan Bobic; Konrad Rzeszutek Wilk
>> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> 
>> 
>> Wednesday, September 24, 2014, 3:58:57 PM, you wrote:
>> 
>> > Sander,
>> 
>> > You have given me some interesting information.  First off, I would
>> > like to ask you about one of your comments;
>> 
>> >> Yes, from what i see the problem seems to be that the Radeon card is
>> >> not reset to a state were it will regard itself as "unposted". (on
>> >> the first boot of the guest i see the driver report it hasn't posted
>> >> (which is correct), it starts to past, and it works. On subsequent
>> >> boots of the domU i don't see the unposted message
>> 
>> > I am curious how you can tell if the driver is posting or not?  Which 
>> > message
>> are you looking at?
>> 
>> With domU kernel 3.17-rc5 (with older kernels (i forgot the version number
>> where it got fixed) there was a problem with the secondary card getting the
>> shadow rom from the primary(emulated cirrus) ( see
>> https://lkml.org/lkml/2014/2/13/109)), so a recent kernel domU kernel is 
>> best :)
>> 
>> In the domU on first boot it says it's not posted:
>> 
>> [    4.346649] [drm] Initialized drm 1.1.0 20060810
>> [    4.359722] [drm] radeon kernel modesetting enabled.
>> [    4.380435] xen: --> pirq=32 -> irq=36 (gsi=36)
>> [    4.383891] [drm] initializing kernel modesetting (TURKS 0x1002:0x6759
>> 0x174B:0xE193).
>> [    4.405597] [drm] register mmio base: 0xF3060000
>> [    4.418262] [drm] register mmio size: 131072
>> [    4.559911] ATOM BIOS: ELIXIR
>> [    4.569063] [drm] GPU not posted. posting now...
>> [    4.585542] radeon 0000:00:05.0: VRAM: 1024M 0x0000000000000000 -
>> 0x000000003FFFFFFF (1024M used)
>> [    4.609664] radeon 0000:00:05.0: GTT: 1024M 0x0000000040000000 -
>> 0x000000007FFFFFFF
>> ....
>> 
>> In the domU on subsequent boots that message is not there, but i end up with
>> errors like (and garbage on the screen (looks somewhat like white noise)):
>> 
>> [    4.321952] [drm] Initialized drm 1.1.0 20060810
>> [    4.335246] [drm] radeon kernel modesetting enabled.
>> [    4.355349] xen: --> pirq=32 -> irq=36 (gsi=36)
>> [    4.359541] [drm] initializing kernel modesetting (TURKS 0x1002:0x6759
>> 0x174B:0xE193).
>> [    4.385655] [drm] register mmio base: 0xF3060000
>> [    4.401814] [drm] register mmio size: 131072
>> [    4.599903] ATOM BIOS: ELIXIR
>> [    4.609170] radeon 0000:00:05.0: VRAM: 1024M 0x0000000000000000 -
>> 0x000000003FFFFFFF (1024M used)
>> [    4.632895] radeon 0000:00:05.0: GTT: 1024M 0x0000000040000000 -
>> 0x000000007FFFFFFF
>> ....
>> [    5.126844] [drm:radeon_pm_init_dpm] *ERROR* radeon: dpm initialization
>> failed
>> ....
>> [    5.555322] [drm:r600_ring_test] *ERROR* radeon: ring 0 test failed
>> (scratch(0x8504)=0xCAFEDEAD)
>> ....
>> 
>> (complete dmesg from both boots attached)
>> 
>> --
>> Sander
>> 
>> > Thanks,
>> > Kelly
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@xxxxxxxxxxxxxx]
>> >> Sent: Wednesday, September 24, 2014 9:22 AM
>> >> To: Zytaruk, Kelly
>> >> Cc: Peter Kay; xen-devel@xxxxxxxxxxxxx; Gordan Bobic; Konrad
>> >> Rzeszutek Wilk
>> >> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> >>
>> >>
>> >> Wednesday, September 24, 2014, 2:12:38 PM, you wrote:
>> >>
>> >> >> Good to know there is interest from AMD into this area :-)
>> >>
>> >> > I am taking a personal interest in this and would like to improve
>> >> > AMD support
>> >> and presence within the Xen community.
>> >>
>> >> Thanks for that !
>> >>
>> >> > Gordan has also reported problems restarting a guest.
>> >>
>> >> Added Konrad to the CC as pciback maintainer.
>> >>
>> >> Yes, from what i see the problem seems to be that the Radeon card is
>> >> not reset to a state were it will regard itself as "unposted". (on
>> >> the first boot of the guest i see the driver report it hasn't posted
>> >> (which is correct), it starts to past, and it works. On subsequent
>> >> boots of the domU i don't see the unposted message but some failures
>> >> that are probably due to the driver regarding the card as already been 
>> >> posted
>> and skipping some init logic and reusing wrong values.
>> >>
>> >> So current xen-pciback logic doesn't seem to be able to reset the
>> >> card in a "non- posted" state.
>> >>
>> >> I don't know if you know what kind of reset is required for radeon
>> >> cards to regard itself as "not posted" ?
>> >>
>> >> Current xen-pciback logic uses the "__pci_reset_function_locked(dev);"
>> >> functions (see pcistub.c pcistub_device_release()) to try to reset the 
>> >> device ..
>> >> that function in turn tries some possible reset functions in a
>> >> specific order and bails out at the first one reporting "succes".
>> >> However it could be that this level of reset is not enough for this
>> >> specific case. (in my case it always uses and succeeds at the first
>> (pci_dev_specific_reset(dev, probe);).
>> >>
>> >> When looking at the code for vfio/kvm in "drivers/vfio/pci/vfio_pci.c
>> >> vfio_pci_disable()", it seems to use:
>> >> - Another order for disabling resetting and config save/restore
>> >> - Always try a slot/bus reset on the way out ..
>> >>
>> >> I'm trying to experiment with that in 2 ways:
>> >> 1) overrulling the logic in the domU radeon driver to do the reposting no
>> matter
>> >>    in what state it thinks it is.
>> >> 2) Trying to change the xen-pciback reset logic, but at present that 
>> >> delivers
>> >>    bad irq's to dom0 for completly different irq's as the passedthrough 
>> >> device
>> >>    has (which i have seen before .. and is a bit worrying)
>> >>
>> >> > I have been trying to reproduce the problem but have not had any
>> >> > luck.  As a
>> >> secondary it restarts for me every time.  I don't know if I
>> >> inadvertently made a change that indirectly fixed it in my code base or 
>> >> what
>> the difference might be.
>> >>
>> >> Perhaps in the older qemu-traditional there is already some sort of reset
>> done ?
>> >>
>> >> > What Xen version are you testing with?
>> >> I'm almost always living on the edge with Xen-unstable ;) (the latest
>> >> and greatest, which is actually pretty stable)
>> >>
>> >> --
>> >> Sander
>> >>
>> >> > Thanks,
>> >> > Kelly
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@xxxxxxxxxxxxxx]
>> >> >> Sent: Tuesday, September 23, 2014 9:45 AM
>> >> >> To: Zytaruk, Kelly
>> >> >> Cc: Peter Kay; xen-devel@xxxxxxxxxxxxx
>> >> >> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> >> >>
>> >> >> Good to know there is interest from AMD into this area :-)
>> >> >>
>> >> >> I'm experimenting for a while with:
>> >> >>
>> >> >> - xen-unstable (and thus xl)
>> >> >> - latest kernels (both dom0 and domU)
>> >> >> - qemu-xen
>> >> >> - Radeon HD 6570
>> >> >> - secondary passthrough
>> >> >> - Debian linux (sid) with the opensource (in kernel) radeon driver
>> >> >>   (i also tried fglrx with succes, but it's a real PITA to build with 
>> >> >> every
>> >> >>   new kernel, so i ditched that)
>> >> >>
>> >> >> It used to work, but something broke at the moment, but that could
>> >> >> also be the changes to the systemd cruft that Debian jessie/sid is
>> >> >> currently undergoing (or something else since i regularly update
>> >> >> all
>> >> components).
>> >> >>
>> >> >> The problems are mostly with restarting the domU, it differs a bit:
>> >> >> - sometimes screen goes ok, sometimes it's garbage.
>> >> >> - the radeon powercontrols only seem to work on the first boot and
>> >> >> give errors on any subsequent one.
>> >> >>
>> >> >> But when it works it does:
>> >> >> - the powercontrol.
>> >> >> - opengl and opencl benchmarks with (near) native results.
>> >> >> - hardware video acceleration in xbmc for instance.
>> >> >>
>> >> >> So one of the main problems at present seems to be proper
>> >> >> resetting of the whole device on domain shutdown/start. I did do
>> >> >> some experiments with the opensource radeon driver, but didn't get
>> >> >> conclusive
>> >> results out of that yet.
>> >> >>
>> >> >> --
>> >> >> Sander
>> >> >>
>> >> >> Tuesday, September 23, 2014, 3:19:41 PM, you wrote:
>> >> >>
>> >> >> > Hi Peter / Sander,
>> >> >>
>> >> >> > Yes, I have AMD GPU passthru working as both primary and
>> >> >> > secondary
>> >> >> passthru.  Secondary was easy but primary is a bit tricky.
>> >> >>
>> >> >> > Getting on to your questions;
>> >> >>
>> >> >> >> Is there any specific reason you're using Xen 4.2 rather than 4.4.1?
>> >> >>
>> >> >> > I am working on a project that is based on Xen 4.2 (I can't say
>> >> >> > any more than
>> >> >> that).  I have looked at some of the more recent versions just to
>> >> >> check if some of the bugs that I have seen have been fixed but I
>> >> >> have not studied the newer versions in detail.  At some point in
>> >> >> time in the future I would like will make the jump to a more
>> >> >> recent version but I don't
>> >> know the scheduling of that.
>> >> >>
>> >> >> >> In 4.2, using xl or xm?
>> >> >>
>> >> >> > xl
>> >> >>
>> >> >> >> qemu-traditional (with rombios) or "upstream"
>> >> >>
>> >> >> > qemu-traditional
>> >> >>
>> >> >> >> Primary or secondary passthrough?
>> >> >>
>> >> >> > Both but I am focusing on secondary right now.
>> >> >>
>> >> >> >> Presumably 64 bit versions of Windows?
>> >> >>
>> >> >> > 32 bit and 64 bit Win7.  I have tested Win8.1 and it works but
>> >> >> > my focus is currently Win7
>> >> >>
>> >> >> >> I am quite willing to test various scenarios. I've a 6950, 6450 and 
>> >> >> >> 5450.
>> >> >>
>> >> >> > Awesome.  My goal right now is obtaining stability on Xen 4.2.
>> >> >> > Since 4.2 is
>> >> >> past its feature cutoff I won't be able to submit any open source
>> >> >> changes for
>> >> it.
>> >> >> I would like to eventually work with the community to get passthru
>> >> >> working with a recent version of "upstream".
>> >> >>
>> >> >> > Thanks,
>> >> >> > Kelly
>> >> >>
>> >> >>
>> >> >>
>> >> >> >> -----Original Message-----
>> >> >> >> From: Sander Eikelenboom [mailto:linux@xxxxxxxxxxxxxx]
>> >> >> >> Sent: Monday, September 22, 2014 8:38 AM
>> >> >> >> To: Peter Kay
>> >> >> >> Cc: xen-devel@xxxxxxxxxxxxx; Zytaruk, Kelly
>> >> >> >> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> >> >> >>
>> >> >> >>
>> >> >> >> Monday, September 22, 2014, 2:16:58 PM, you wrote:
>> >> >> >>
>> >> >> >> > Hi Kelly, list
>> >> >> >>
>> >> >> >> > I see you're having AMD GPU success with Xen 4 2 and Linux 3.4.9.
>> >> >> >> > I've been
>> >> >> >> less than successful getting passthrough working at all in Xen
>> >> >> >> (although it's fine in KVM primary passthrough as long as the
>> >> >> >> BIOS is supplied as a file). Could I confirm the following :
>> >> >> >>
>> >> >> >> > Is there any specific reason you're using Xen 4.2 rather than
>> >> >> >> > 4.4.1? I know in
>> >> >> >> some ways 4.4 suffers as it's now xl only and some of the xm
>> >> >> >> functionality has not come across.
>> >> >> >>
>> >> >> >> > In 4.2, using xl or xm?
>> >> >> >>
>> >> >> >> Another interesting question/aspect would be qemu-traditional
>> >> >> >> (with
>> >> >> >> rombios) or "upstream" (with seabios) ?
>> >> >> >>
>> >> >> >> > Primary or secondary passthrough?
>> >> >> >>
>> >> >> >> > Presumably 64 bit versions of Windows?
>> >> >> >>
>> >> >> >> > My system is a bit old (Core2Quad) but as mentioned AMD
>> >> >> >> > passthrough works
>> >> >> >> in KVM but I've found it tricky in Xen.
>> >> >> >>
>> >> >> >> > I am quite willing to test various scenarios. I've a 6950, 6450 
>> >> >> >> > and
>> 5450.
>> >> >> >>
>> >> >> >> > Thanks
>> >> >> >>
>> >> >> >> > Peter
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel