Re: [Xen-users] [Xen VGA Passthrough] AMD R9 290X GPU???

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 2014-09-12 13:14, Peter Kay wrote:
> It's possible that the reason mixing multiple cards doesn't work is
> because ATI drivers allegedly try and initialise cards, even if
> they're claimed elsewhere.
Can you elaborate on that? Do you mean that fglrx driver in dom0
tries to initialize the devices that are bound to xen-pciback?

Or do you mean the ATI driver in domU tries to initialize the ATI
GPUs that haven't even been passed to that domU?

Or something else entirely?

> Most of my attempts so far have been with KVM. With that a HD6950
> passes through just fine once a NoSnoop patch is applied, but having a
> low end Nvidia card in the host Linux breaks things badly.
Are you saying that passthrough of an ATI card causes the host
to crash even though the host is running an Nvidia card?

> I've not managed to get the 6950 working in Xen at all, possibly
> because its BIOS is >64KB. Under KVM it was necessary to supply the
> extracted BIOS in a file.
AFAIK this is only necessary for VGA passthrough (i.e. for
seeing the SeaBIOS splash screen in the guest and getting
video output before the GPU driver loads in the guest.

I've never even tried using VGA passthrough, only secondary
PCI passthrough - I can live with the video output being confined
to the emulated GPU with VNC output and only getting output from
the GPU once the driver loads.

> A GTX480 soft modded to a Quadro 6000 is
> working fine with both qemu-traditional and qemu upstream, although
> upstream seems decidedly less stable. That's with an x64 unpatched
> Windows 8.
I never tried qemu-traditional, and I only used XP x64 and 7 x64.
Unfortunately, NF200 bridges break IOMMU operation so I have to
make sure the domU memory doesn't overlap the IOMEM regions. I
have a bodged patch that achieves this but am hoping to switch
to using the patch that limits RAM below 4GB soon which should
achieve the same thing (I must prevent the domU from accessing
memory above 2688MB as this is where my IOMEM areas are).

In fairness, I could make do without either patch by simply
using the Windows 7 bcdedit badmem option in domU to mark memory
between 2688MB and 4096MB as bad. I'd lose 1408MB per domU, but
with only a handful of domUs and 96GB of RAM I can live with
that.

> I've spent a large amount of time messing around with motherboards,
> kernels and suchlike and my notes so far are :
>
> 1) If your criteria is passthrough of any type, KVM is a better option
> than Xen. It works and it's also easily possible to identify iommu
> isolation groups, aiding stability.
I'm going to guess this requires IOMMU ACL to work (the correct
name/acronym for it escapes me at the moment) - which it doesn't
on some PCIe bridges (NF200), and is broken on many BIOS-es even
when the hardware itself isn't impossibly broken.

> 2) If you want AMD GPU passthrough, use KVM, it's solid and you'll
> save yourself a huge amount of pain.
Guest reboots work without side effects?

> 3) Quadro passthrough on Xen is stable, but the official NVidia
> drivers must be installed, and the adapter must be assigned at VM
> bootup.
>
> 4) Xen's virtualisation seems better, if possibly slower than KVM.
> Less exceptions and the virtual USB is fairly decent.
Last time I tested Xen was somewhat faster than KVM. I would
have preferred to use KVM because unlike the Xen dom0, with
KVM the host domain isn't running as a virtual domain which
has performance and driver compatibility benefits for the host
domain.

> 5) The number of issues and broken BIOSes in motherboards is huge.
This is really the key issue, and coincides with my experience, too.
Worse, apart from a handful of big brand, expensive servers/workstations
that are certified for VGA/PCI passthrough, there is practically no
comprehensive list of motherboards that are known to work properly
with one or more GPUs passed to different VMs.

The impression I get is that on the whole AMD motherboards fare
a little better, but that could be purely down to more Xen users
using them.

> For my next system, I will probably be buying an Intel workstation or
> server board. It's clear that the more exciting virtualisation
> improvements are happening on Intel first and being Intel, their
> BIOSes generally work, even if there are various quirks. I've just
> been mildly stung by a xw4600 motherboard, which claims to support
> VT-d, but is broken. Back to the S3210SHLC it is, then - I'm not keen
> on the maximum 4x PCI-e speed I'm limited to graphics wise, but it
> does actually work.
In fairness, it has been demonstrated many times that PCIe speed is
not particularly relevant for gaming loads. For compute loads that
are heavily reliant on shipping data to/from the GPU it will make
a difference, but if you your compute vs. I/O ratio is so low
the performance will be pretty horrible anyway.

> (yes, I should run something much more modern, but I'm looking at how
> the latest Haswell/Haswell EP chips work out before wasting money..)
I have had a similarly problematic experience with EVGA's flagship
SR-2 motherboard. None of the issues were insurmountable, but the
NF200 bridges bypassing the IOMMU for DMA and thus clobbering the
host's IOMEM region was particularly annoying. But once the Xen
developers here came up with a hunch that this was what was happening,
working around the problem wasn't particularly difficult.

I will write up an extensive howto with the exact patches, software
versions and hardware modifications required to make it all work
properly at some point (probably as soon as I upgrade to a version
of Xen with the max-mem-below-4GB option implemented).

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users