Re: [Xen-users] VGA/PCI Passthrough of Secondary Graphics Adapter

[Alex Karaoui <code3studios@xxxxxxxxx>]

On Apr 30, 2013, at 8:41 AM, "Gordan Bobic [via Xen]" <[hidden email]> wrote:

> On 04/30/2013 07:27 PM, Ole Johan VÃringstad wrote:
>
> > I too got the atikmpag.sys BSOD initially, I got around it by installing
> > the 12.104 drivers. It also took me a while to realize I had a "working"
> > solution:
> >
> > I would install the drivers, then reboot the dom0, start the domU,
> > connect vnc, and then windows would hang on "Starting Windows". Turns
> > out windows completes the login (I set up auto-login) on the output of
> > the card I passed through. In device manager the emulated VGA adapter
> > shows an error, while the ATI card is fine. If I reboot domU, the
> > emulated VGA is fine, while the ATI card shows an error. For a while I
> > thought I had to do a dom0 reboot, which fixes it, but ejecting the card
> > also works (although I will have to find a way to shut down the domU "in
> > the dark" which is cleaner than xl destroy).
>
> I haven't found any correlation between dom0 reboots and domU working. 2
> days ago, I was rebooting domU multiple times without rebooting dom0.
> Yesterday it didn't matter how many times I rebooted either, domU just
> wouldn't start up and BSOD-ed every time before the login screen appeared.
>
> > My passthrough card is an Asus HD7970. I initially had an Asus HD7770 as
> > my dom0 card, but that caused problems. I have the 7970 in the first
> > PCIe slot and the dom0 in the second, but I can set in bios which prints
> > POST. I could not get an ATI driver to work in dom0. I tried the open
> > source Radeon driver, which needed mesa 9.1+ for acceleration with
> > Southern Islands/HD7xxx, which again needed a patched llvm I had to pull
> > from the git-repo of what I actually believe is an AMD engineer. Still,
> > glxgears gave me 5-7 fps. It might have worked nicely without
> > acceleration but I needed that for monitor rotation. Note that you only
> > need mesa 9.1+ for SI/HD7xxx. The AMD's official driver gave me an
> > amputated xorg.log and ugly segfaults. I pulled out the card and
> > installed my old nVidia GTX275, which now works like a charm with the
> > binary driver. I will order something like an nVidia GTX650 because my
> > 275 only supports 2 monitors and I need 3. I got the HD7770 as a dom0
> > card because I thought it would simplify matters, turns out the opposite
> > was true. But I do not know if this is related to xen, could be a lot of
> > things.
>
> My experience is that ATI cards rarely simplify matters. The moment you
> go off the straight and narrow (single monitor, nothing weird like
> virtualization) things start to fall apart very quickly, especially in
> Windows. FGLRX driver is actually pretty decent in Linux, but it's lack
> of ability to build for dom0 is a major failing, and likely an
> unacceptable one for people on this list.
>
> > I hope this information will be useful to someone beyond myself, because
> > I had to spend quite a bit of time to get it to work. Time is money too,
> > so there comes a point where buying working solutions becomes cheap. I
> > would like to see a database of working hardware combinations. (Atleast
> > I would have loved to when I was setting it up)
> >
> > Setup:
> > Gigabyte GA-X79S-UP5 with i7-3820
> > Gentoo dom0 at 3.7.10, 3.8.10, will now test 3.9.0
> > Xen 4.2.0 from portage, xl toolstack
> > nVidia GTX275 as dom0 VGA
> > Asus Radeon HD7770 DCU-II as domU Secondary Passthrough
> > Windows 7 64bit with PV drivers.
> > AMD 12.104 drivers
> > xen kernel options:
> >    dom0_mem=8192M,max:8192M
> >    dom0_max_vpcus=4
> >    dom0_vcpus_pin
> >    iommu=1
> >    xsave=1
> > linux kernel options:
> >    xen-pciback.permissive
> >    xen-pciback.hide=(USBbus and VGA card)
>
> What does xen-pciback.permissive do?
>
> I found that regardless of xen-pciback module options, I have to
> manually detach the devices from dom0 after loading the xen-pciback module.

I found that using pciback as a module would result in similar behavior.  Since my graphics cards are both AMD Radeons, the open-source 'radeon' driver would load with the kernel due to the new requirement for kernel-mode switching.  The FGLRX drivers do not do this, and can be disabled with pciback as a module.

Since I couldn't' get the FGLRX drivers to work with the Xen kernel (in Fedora, w/3.8 kernel, whereas in Ubuntu 12.04 with 3.5 it worked fine), I recompiled 3.8 with pciback statically linked.  This gave me the ability to hide the device immediately upon kernel loading and prevents the radeon driver from initializing the device.

However, I never got a display on my monitor; but I think I will the methods posted in this thread (regarding rebooting at certain points).  I think they may help, and also explain some of the erratic behavior I've witnessed.

I just wish this were a little more straightforward, and I question the long-term reliability of using PCI passthrough with Radeon cards.  I'm assuming the Quadro cards work flawlessly?

---

View this message in context: Re: VGA/PCI Passthrough of Secondary Graphics Adapter
Sent from the Xen - User mailing list archive at Nabble.com.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users