Xen project Mailing List

I spent the last 8 hours testing VGA passthrough with a AMD Radeon 7770 card. Here the setup:

Hardware: Asus Sabertooth X79 with latest BIOS and VT-d enabled; i7 3930K CPU; Sapphire Radeon 6450 for dom0, Sapphire Radeon 7770 for domU.

1st trial - Linux Mint 16 64 bit with kernel 3.11.0-18 and Xen 4.3.0; Windows 7 Pro 64 bit; both installed on external USB3 disk:

In short, a disaster. I first tried the xl toolstack I was using with my regular configuration (same m/b and CPU, but AMD Radeon 7770 for dom0 and Nvidia Quadro 2000 for domU).

I tried to make a new Windows installation in domU. After each shutdown of the domU I had to reboot the whole machine, else I would get libxl... errors and nothing. dom0 wouldn't even shut down, had to hard reset. I somehow managed to install Windows, but once I tried to install the graphics driver for the AMD card Windows sooner or later gave me a blue screen.

Next I tried with xm. I started from scratch, setting up a new Windows LVM volume. xm create ... failed with "error 22". That sounds familiar. I had the same issue with my regular Nvidia setup some time ago with Xen 4.1.3. But I thought that by Xen 4.3 this was fixed?!

In the bug report someone suggests to pass through only the first function of the graphics card, in my case PCI ID 02:00.0, and not specify the second (HDMI audio) function 02:00.1. I tried but no luck. Just to make sure, removed the graphics card from the PCI passthrough devices and only left a USB controller to pass through, which worked!

Summary: Xen 4.3.0 still has some of the old bugs that I that had gone long ago. Or is that a kernel issue?

2nd trial - Linux Mint 13 64 bit with kernel 3.2 and Xen 4.1.2; Windows 7 Pro 64 bit; both installed on external USB3 disk:

I installed the long term release Linux Mint 13 that worked well when I started out with Xen VGA passthrough. After installation and upgrading all packages and the kernel, I installed Xen and then downgraded using "Force version" in Synaptic to the earliest Xen release available in the repos, which was 4.1.2 if I remember correctly.

This time I only used xm. Installation went smooth, no hickups no nothing after multiple reboots of Windows. Passthrough works just fine with the AMD Radeon 7770. I ran the WEI, both prior and after installing the GPLPV drivers in Windows - it made no difference - 5.9 lowest score for disk (a 2.5" HDD attached via USB3), 2D and 3D graphics are 7.5, memory 7.9 (with 8GB for Windows), and CPU 7.8 (I gave it 8 of 12 VCPUS). I even installed the Unigine demo and benchmark - it ran fine though the 7770 is really not up for it with 21fps.

Summary: Xen 4.1.2 with an old 3.2 kernel works perfect with xm and VGA passthrough.

To me this looks like some ancient bugs are still not fixed (with regard to xm), and probably never will since it's been deprecated. The sad story, however, is that xl has its own serious problems with VGA passthrough.

Whatever it is, it's not an AMD driver issue. The problems start long before I even managed to install the AMD driver.

Hope someone from Xen development has a look at this. Although this means reconfiguring my hardware, I'd be happy to help in debugging if anyone is willing to try and fix it.

On Thursday, March 13, 2014 9:23 AM, Gordan Bobic <gordan@xxxxxxxxxx> wrote:

On 03/13/2014 07:06 AM, H. Sieger wrote:
> I can't (and won't) argue with trying it out. However, the issue here
> (performance loss/instability after domU shutdown/reboot) has nothing to
> do with blacklisting the graphics driver.

On the contrary, it is related because it affects the state of the card.
If the state of the ATI card has been "tainted" by being touched by the
driver, once you detach it from the driver and to pciback it will react
the same as it does after a reboot of the domU it was passed to.

> Blacklisting the graphics
> driver(s) is not only necessary for AMD/ATI graphics cards, but also for
> Nvidia cards.

As I said before, with an Nvidia card, I can move the card around
between domains without rebooting the machine. I haven't tried it in a
few months, but I had been able to shut down the domU with the card,
detach it from pciback, re-load the driver in dom0, use the card in
dom0, remove the nvidia driver and re-attach the card to pciback, and
then fire up another domU with the same card. So blacklisting is not
strictly necessary because Nvidia cards don't seem to be affected by
being in a pre-initialized state.

> This has to do with how and when pciback (or pci-stub) is
> trying to grab the graphics card - if that happens after a graphics
> driver took possession, bad luck (according to my experience).

In my experience this can be made to work. Off the top of my head, you
can detach the card from it's current driver using something like:

echo $device_id > /sys/bus/pci/drivers/nvidia/unbind

and then attack it to pciback using

echo $device_id > /sys/bus/pci/drivers/pciback/bind

(don't quote me on the exact incantation, but that is the gist of it).

Or you could use xl pci-assignable-add which essentially does the same
thing.

I do recall this was more problematic before (maybe around xen 4.1/4.2
days), but I'm sure last time I tried it did work on my setup.

> Now that you mention pci-stub, have you tried pciback? Perhaps the whole
> issue is pci-stub related?

When I said pci-stub I was of course referring to xen-pciback. pci-stub
is what the equivalent KVM driver is called.

Re: [Xen-users] VGA passthrough with Xen 4.3 and xl toolstack - performance degradation resolved?