Re: [Xen-users] Multiple VMs VGA Passthrough Success Report

[Gordan Bobic <gordan@xxxxxxxxxx>]

On Sun, 7 Jul 2013 01:28:45 -0700, Casey DeLorme <cdelorme@xxxxxxxxx> 
wrote:
> Hello Gordan,
>
> I would like to hear more, any chance you can share the xen version,
> and whether you applied any patches?
As mentioned in the original email, I'm running on EL6 (Scientific)
and I'm running the latest rpms of kernel and xen packages from the
CRC repository (kernel 3.9.9, xen 4.2.2 with up to date XSA patches
applied).

> I know you were looking into
> the primary passthrough patch and also MMIO (in relation to memory
> stomping). ÂDid you apply those patches and are you using upstream
> qemu and/or primary passthrough? ÂAny tips on applying the patches
> would also be great.
I have not applied any extra patches. The MMIO patch I didn't bother
with - it involves statically defining BAR regions, so until I have
time to write a better patch that auto-detects BARs at runtime and
maps them all, I left it out. It might help with stability in some
cases and avoid the PCI memory stomp, but I seem to have gotten
this stable without it. I'll probably still do a dynamic
pBAR=vBAR patch at some point because I'm hoping it'll mean I
don't have to doctor my GeForce cards into Quadros to get them to
work (and get working PhysX as a bonus since Quadro drivers don't
seem to enable it). But for now all my GeForces are modified into
Quadros, so until upgrade time, my motivation is running low.
How long those GTX480->Quadro 6000 cards can keep up remains to
be seen. :)

The other part of the Nvidia primary passthrough patch is the BIOS
initializer, which is currently limited to 64KB BIOSes. All but one
GTX680 cards (a Gigabyte one) and all Titan/7xx cards and most other
6xx cards have UEFI in BIOS which bloats them past 64KB. This can
be stripped out (done it, needs more testing before I'm 100%
confident I didn't break anything - a weekend of various
benchmarking ought to do it). But that requires card modifying,
and if I've got to do that anyway, it kind of reduces the appeal
of the patch, and Quadro drivers no doubt handle a few extra edge
cases more gracefully.

Gordan

> On Sat, Jul 6, 2013 at 5:11 PM, Gordan Bobic  wrote:
>  I just wanted to drop a note to say that I have managed to get VGA
> passthrough working with multiple cards and domUs.
>
>  The cards I am using are quasi Quadro 5000 (modified GTX470) and
> quasi Quadro 6000 (modified GTX480). (Here is an older article about
> modified GTS450 modified to Quadro 2000 if anyone is interested:
>
>
> http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/
> [2] )
>
>  This succeeded despite all the PCIe slots being behind NF200 bridges
> without ACS support.
>
>  I now have two VMs with VGA passthrough running, with one card 
> passed
> through to each one. The OS is EL6, and I am using Xen packages and
> kernel from the CRC repository.
>
>  What is strange, however, is that whereas I had issues with a single
> VM crashing on reboot with VGA passthrough (and usually taking the
> host down with it), with two VMs running, I seem to be able to reboot
> them dozens of times without any crashes, problems, or performance
> degradation (yes, rebooting VMs with VGA passthrough without 
> rebooting
> the host and there appears to be no effect on performance with Nvidia
> Quadro cards). This new found stability could be down to some of the
> PCI changes, though, with different USB controllers being passed
> through, and NIC being bridged rather than PCI passthrough since 
> there
> are now two domUs that have to share it, so it is possible that the
> PCI memory stomp I am suspecting to be the root cause is getting
> coincidentally avoided (not that I'm complaining!).
>
>  I'm happy to share more details on and off list if anybody is
> interested.
>
>  Gordan
>
>  _______________________________________________
>  Xen-users mailing list
>  Xen-users@xxxxxxxxxxxxx [3]
>  http://lists.xen.org/xen-users [4]
>
>
>
> Links:
> ------
> [1] mailto:gordan@xxxxxxxxxx
> [2]
>
> http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/
> [3] mailto:Xen-users@xxxxxxxxxxxxx
> [4] http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users