[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Multiple VMs VGA Passthrough Success Report

On Sun, 7 Jul 2013 01:28:45 -0700, Casey DeLorme <cdelorme@xxxxxxxxx> wrote:
Hello Gordan,

I would like to hear more, any chance you can share the xen version,
and whether you applied any patches?

As mentioned in the original email, I'm running on EL6 (Scientific)
and I'm running the latest rpms of kernel and xen packages from the
CRC repository (kernel 3.9.9, xen 4.2.2 with up to date XSA patches

I know you were looking into
the primary passthrough patch and also MMIO (in relation to memory
stomping). ÂDid you apply those patches and are you using upstream
qemu and/or primary passthrough? ÂAny tips on applying the patches
would also be great.

I have not applied any extra patches. The MMIO patch I didn't bother
with - it involves statically defining BAR regions, so until I have
time to write a better patch that auto-detects BARs at runtime and
maps them all, I left it out. It might help with stability in some
cases and avoid the PCI memory stomp, but I seem to have gotten
this stable without it. I'll probably still do a dynamic
pBAR=vBAR patch at some point because I'm hoping it'll mean I
don't have to doctor my GeForce cards into Quadros to get them to
work (and get working PhysX as a bonus since Quadro drivers don't
seem to enable it). But for now all my GeForces are modified into
Quadros, so until upgrade time, my motivation is running low.
How long those GTX480->Quadro 6000 cards can keep up remains to
be seen. :)

The other part of the Nvidia primary passthrough patch is the BIOS
initializer, which is currently limited to 64KB BIOSes. All but one
GTX680 cards (a Gigabyte one) and all Titan/7xx cards and most other
6xx cards have UEFI in BIOS which bloats them past 64KB. This can
be stripped out (done it, needs more testing before I'm 100%
confident I didn't break anything - a weekend of various
benchmarking ought to do it). But that requires card modifying,
and if I've got to do that anyway, it kind of reduces the appeal
of the patch, and Quadro drivers no doubt handle a few extra edge
cases more gracefully.


On Sat, Jul 6, 2013 at 5:11 PM, Gordan Bobic  wrote:
 I just wanted to drop a note to say that I have managed to get VGA
passthrough working with multiple cards and domUs.

 The cards I am using are quasi Quadro 5000 (modified GTX470) and
quasi Quadro 6000 (modified GTX480). (Here is an older article about
modified GTS450 modified to Quadro 2000 if anyone is interested:

[2] )

 This succeeded despite all the PCIe slots being behind NF200 bridges
without ACS support.

I now have two VMs with VGA passthrough running, with one card passed
through to each one. The OS is EL6, and I am using Xen packages and
kernel from the CRC repository.

 What is strange, however, is that whereas I had issues with a single
VM crashing on reboot with VGA passthrough (and usually taking the
host down with it), with two VMs running, I seem to be able to reboot
them dozens of times without any crashes, problems, or performance
degradation (yes, rebooting VMs with VGA passthrough without rebooting
the host and there appears to be no effect on performance with Nvidia
Quadro cards). This new found stability could be down to some of the
PCI changes, though, with different USB controllers being passed
through, and NIC being bridged rather than PCI passthrough since there
are now two domUs that have to share it, so it is possible that the
PCI memory stomp I am suspecting to be the root cause is getting
coincidentally avoided (not that I'm complaining!).

 I'm happy to share more details on and off list if anybody is


 Xen-users mailing list
 Xen-users@xxxxxxxxxxxxx [3]
 http://lists.xen.org/xen-users [4]

[1] mailto:gordan@xxxxxxxxxx

[3] mailto:Xen-users@xxxxxxxxxxxxx
[4] http://lists.xen.org/xen-users

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.