Re: [Xen-users] ATI VGA Passthrough / Xen 4.2 / Linux 3.8.6

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 04/27/2013 09:25 PM, Alex Karaoui wrote:
> Gordan Bobic wrote
> > On 04/22/2013 10:46 AM, Gizmo Chicken wrote:
> > > Not sure if I can offer any wisdom, but I'll just note that secondary
> > > GPU passthrough  has been pretty stable for me with Windows 7 combined
> > > with a Radeon HD 6670 and the Catalyst 13.3beta3 driver.  But I confess
> > > that I cheat a bit:  For this setup, I'm using XCP 1.6 as dom0 and am
> > > managing guests with XenCenter.  I had GPU passthrough working with
> > > vanilla Xen 4.x in the past.  But XCP 1.6 (which includes Xen 4.1) is
> > > just easier.  But whatever your setup, you may wish to give the Catalyst
> > > 13.3beta3 driver a try.  It works great for me.
> > It got to the point where the VM won't actually start - it BSODs saying:
> >
> > "Attempt to reset the display driver and recover from timeout failed."
> >
> > in atikmpag.sys
> >
> > The only way I can boot it is either into safe mode, or by removing the
> > ATI card from the VM configuration. In safe mode the 13.3-beta driver
> > fails saying it failed to load the detection driver.
[...]
> > Thus far the only noteworthy thing I've changed (to no obvious positive
> > effect) is putting the VGA passthrough GPU into a different slot so that
> > it is the only thing on that particular PCIe bridge.
> I've signed up to this mailing list simply because your situation mirrors my
> own.  Allow me to elaborate, my setup is as follows:
>
> Processors = 2x Intel E5440 Xeons (Harpertown)
> Motherboard = SuperMicro X7DWA-N
> Graphics =
> 1x ATI Radeon 5870 1GB (PCIe Slot 1 -- Primary card in BIOS)
> 1x ATI Radeon 5570 1GB (PCIe Slot 6 -- Secondary; No POST messages on boot)
I don't think it matters what you use for your primary (Dom0) GPU. I 
have an Nvidia 8800GT and an ATI 6450. Passing through the Nvidia card 
didn't work at all (yellow exclamation mark in device manager). I have 
had a little more success with the ATI card passthrough recently. More 
details below.
> I'm running Xen 4.2.1 with patches applied to qemu-dm for ATI primary
> pass-through.
> My dom0 is Fedora 18.  My grub commandlines are:
>
> GRUB_DEFAULT=
> GRUB_CMDLINE_LINUX="xen-pciback.permissive
> xen-pciback.hide=(02:00.0)(02:00.1)(00:1d.7)
> pci=resource_alignment=02:00.0;02:00.1"
> GRUB_CMDLINE_XEN="iommu=1 dom0_mem=10240M"
I'm using kernels and the xen stack from here since I use EL6:

http://xen.crc.id.au/support/guides/

xen-pciback on that is a module rather than built in, so I have the 
options for it in modprobe.d.
> The lspci listing for my cards has the 5870 (used by dom0) at 01:00.0, and
> 01:00.1 (HDMI audio)
> The same listing has the 5570 (hidden by pciback) at 02:00:0, and 02:00.1
>
> I've tried Windows 7 in both x86 and x64 as domU and my results are sporadic
> but neither yields a display on my second monitor.  The bitness seems not to
> matter, but the results vary when using different driver versions as well as
> using CCC.
I have managed to get further than this, but there are still issues.

See here:
https://lists.wireless.org.au/pipermail/kernel-xen/2013-April/000213.html

I'm using xen-hypervisor 4.2.1-6, and everything else is 4.2.2-1, with a 
modified 3.8.8-1 kernel (3.8.8-1.0, pciehp is a module and NR_CPUS 
boosted to 32).
In this setup the 6450 _almost_ works in the guest OS. I am running a 
bare driver (13.3beta3), no CCC. I also don't have .NET installed in my 
Win7 64-bit guest OS.
The VM boots and generally works OK as long as there is no full-screen 
3D usage. The moment you try to fire up a 3D application in full screen 
mode, it causes the app crash. Sometimes the driver manages a device 
reset. Most of the time it results in a 
"PAGING_REQUEST_IN_NONPAGED_AREA" BSOD, though.
Windowed 3D operations seem to work fine, though. I can run GPU-Z PCIe 
test and OCCT GPU test (windows, full screen causes above mentioned 
crashes) without any obvious problems.
> 1) I'm using a Dual CPU system, and this is somehow causing problems with
> ownership and separation of PCIe devices on the same bus?  I don't know if
> this is possible.
I'm not entirely convinced about this.
Note that I am running a GPU in a slot where it is the only device 
behind a particular PCIe bridge. I don't actually know whether that 
makes a difference or not (or whether it only might make a difference on 
my hardware since I am stuck with ALL 8 PCIe slots behind the Nvidia 
NF200 bridge (EVGA in their infinite wisdom decided that somebody might 
need 8 PCIe slots and crammed them all behind the NF200 routers).
> 3) Xen cannot pass-through one card in a two card configuration when dom0 is
> using one of the cards.
Not sure if this might be dual-ATI related. I have no such issues with 
Nvidia+ATI card in my system.
> 4) Xen cannot pass-through the secondary card in a two card configuration.
> It must be the primary card, and the OS must use the secondary card.
> Perhaps a hardware limitation of VT-d?
Not the case - I only ever tried passing the secondary GPU. As mentioned 
above, this mostly seems to work now, albeit not quite usable yet for 
any particularly meaningful purpose.
What made the difference for me since my last post is the Xen stack 
update to the versions mentioned above. The xen-hypervisor 
issue/regression I mentioned is something I am still investigating. 
Hoping to gather some logs and data and file a bug report later today.
Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users