Re: [Xen-users] [Xen VGA Passthrough] AMD R9 290X GPU???

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 2014-09-12 15:24, Peter Kay wrote:
> On 12 September 2014 14:46, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
>
> > On 2014-09-12 13:14, Peter Kay wrote:
> >
> > > It's possible that the reason mixing multiple cards doesn't work
> > > is
> > > because ATI drivers allegedly try and initialise cards, even if
> > > they're claimed elsewhere.
> Â
>
> > > Most of my attempts so far have been with KVM. With that a HD6950
> > > passes through just fine once a NoSnoop patch is applied, but
> > > having a
> > > low end Nvidia card in the host Linux breaks things badly.
> > Are you saying that passthrough of an ATI card causes the host
> > to crash even though the host is running an Nvidia card?
> No. The lower end NVidia cards (GT210) have a (dom0/KVM host) Linux
> driver that causes instability if passthrough is used, due to VGA
> arbitration. Using the official NVidia driver is a bad idea. Nouveau
> is slightly better iirc, but has other issues. Not sure if higher end
> Nvidia cards fix this.
That's weird - until a couple of weeks ago I've been using an 8800GT
for my dom0 GPU (780Tis for domUs) and had no problems. Just made
sure that the xen-pciback was loaded and bound to the 780Tis before
the nvidia driver for the 8800GT card was loaded. Never had a
problem. Quadro 2000 also works, as did, IIRC, GTX470 and GTX480.

GT630 causes the system to lock-up about 80% of the time as soon
as Xorg starts. Same driver, same configuration. I am at a loss
to explain it.

> > I've never even tried using VGA passthrough, only secondary
> > PCI passthrough - I can live with the video output being confined
> > to the emulated GPU with VNC output and only getting output from
> > the GPU once the driver loads.
> I haven't managed to get a 6950 working full stop, either primary or
> secondary in Xen. It's fine in KVM.
That's odd. I had a HD6450 and HD7970 working in domU as well
as can be expected (works fine on the first domU boot, with
artifacting, slowdown, or just plain system-wide lock-up after
domU is rebooted or stopped and started again).

I wanted a passively cooled dom0 GPU and wanted to use the GT630,
but since that just wouldn't work, I opted for the HD6450. I don't
need that much GPU power for dom0, it's just that GT630 is faster
and has lower TDP than the HD6450 (and is 2-3 generations newer).

> > > 1) If your criteria is passthrough of any type, KVM is a better
> > > option
> > > than Xen. It works and it's also easily possible to identify
> > > iommu
> > > isolation groups, aiding stability.
> > I'm going to guess this requires IOMMU ACL to work (the correct
> > name/acronym for it escapes me at the moment) - which it doesn't
> > on some PCIe bridges (NF200), and is broken on many BIOS-es even
> > when the hardware itself isn't impossibly broken.
> To be fair, you'll have just as many issues with KVM with a NF200
> motherboard.. I think there are workarounds there, but I haven't been
> keeping up.
The main problem is that I need to prevent the guest domains from
using the memory addresses within the guest memory map that are used
for PCI IOMEM regions. Ideally this should be done by presending the
e820 map that marks those regions as reserved. As a plan B, however,
marking the regions as bad/broken memory in the guest should work,
even if it does mean some of the RAM becomes unusable/wasted.

> > > 2) If you want AMD GPU passthrough, use KVM, it's solid and you'll
> > > save yourself a huge amount of pain.
> > Guest reboots work without side effects?
> Yes. It's solid.
Are you using a kernel that implements the bus reset to reset
the card? Or is KVM capable of side-loading the GPU BIOS and
re-POST-ing that for recent ATI cards (with VBIOS > 64KB)?

> > Last time I tested Xen was somewhat faster than KVM. I would
> > have preferred to use KVM because unlike the Xen dom0, with
> > KVM the host domain isn't running as a virtual domain which
> > has performance and driver compatibility benefits for the host
> > domain.
> I think the benchmarks differ but show KVM having a bit of an edge. I
> prefer the manageability of Xen. KVM is currently better at hot plug.
Neither of those aspects are particularly important to me, I just
need the guests to be stable and work properly with GPU passthrough,
and survive a reboot every week or two when Windows decides it needs
a reboot to apply the patches, without having to reboot the host.

> It's a bit of a mess how Xen has migrated from xm to xl without
> maintaining all the functionality, never mind in a solid manner.
I know what you mean. It seems to have been done without due
diligence.

> > In fairness, it has been demonstrated many times that PCIe speed is
> > not particularly relevant for gaming loads. For compute loads that
> > are heavily reliant on shipping data to/from the GPU it will make
> > a difference, but if you your compute vs. I/O ratio is so low
> > the performance will be pretty horrible anyway.
> This is true to some extent. My testing seems to show the difference
> between PCI-e 1.x 16x and 8x is minimal, that the difference between
> 8x and 4x is (sadly) noticeable but not catastrophic, and then below
> that things become a bit treacle like - although a fast card at 1x may
> fare a lot better than a slow card at 8x.
Are you still running a PCIe 1.1 system? Or are you using a GPU that
is PCIe 1.1? IIRC most PCIe motherboards and devices since circa 2008
are PCIe 2.0+.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users