Re: [Xen-users] VGA Passthrough of AMD FirePro W7000 to Windows 8: Not enough resources

[Gordan Bobic <gordan@xxxxxxxxxx>]

On Wed, 02 Oct 2013 18:12:11 +0100, Toby Miller <tobycmiller@xxxxxxxxx> 
wrote:
> On 02/10/13 15:17, Gordan Bobic wrote:
> > On 10/02/2013 03:01 PM, Toby Miller wrote:
> > > > On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
> > > >
> > > > > On 10/02/2013 01:58 PM, Toby Miller wrote:
> > > > > > > gordan@xxxxxxxxxx wrote:
> > > > > >   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
> > > > > > Miller<tobycmiller@xxxxxxxxx>   wrote:
> > > > > > > > I'm using Xen 4.3, installed from source on Ubuntu 13.04. I've
> > > > > > > > successfully installed a Windows 8 guest, and had Xen pass it 
> > > > > > > > an AMD
> > > > > > > > FirePro W7000 GPU (technically 2 PCI devices - the second is 
> > > > > > > > for the
> > > > > > > > sound), as well as the virtualised graphics adapter. Windows 
> > > > > > > > finds
> > > > > > > > the
> > > > > > > > card fine, and I've installed AMD's drivers. Device Manager 
> > > > > > > > says that
> > > > > > > > the card cannot find enough resources (code 12). I've checked 
> > > > > > > > the IO,
> > > > > > > > IRQ, and Memory details in DM, and there don't seem to be any
> > > > > > > > clashes.
> > > > > > > >
> > > > > > > > I did try disabling the virtualised card from Windows, but it 
> > > > > > > > seemed
> > > > > > > > to keep using it anyway, even after a restart. It didn't make 
> > > > > > > > any
> > > > > > > > difference to the FirePro in any case.
> > > > > > > >
> > > > > > > > I'd be grateful for some help.
> > > > > >   I haven't tried Windows 8, but on XP64 disabling the Cirrus 
> > > > > > card in
> > > > > >   device manager fixed the problem for me with Nvidia card 
> > > > > > passthrough.
> > > > > >   You could try setting gfx_passthru=1, which should disable the
> > > > > >   Cirrus emulation alltogether. That may make Windows domU 
> > > > > > completely
> > > > > >   fail to boot, though.
> > > > > >
> > > > > >   Did you do a clean Win8 install and reboot the host fully? Is 
> > > > > > the
> > > > > >   FirePro the primary GPU on the host? Or is it completely 
> > > > > > untouched
> > > > > >   in dom0 (no dom0 or BIOS output on it)?
> > > > > >
> > > > > >   Gordan
> > > > > Thanks for your reply. It turns out the problem was that I'd 
> > > > > forgotten
> > > > > to put device_model_version="qemu-xen-traditional", so I was 
> > > > > using a
> > > > > qemu that didn't support gfx_passthrough. The FirePro is not used 
> > > > > in
> > > > > dom0 at all. I'm using pciback to hide it.
> > > > >
> > > > > I now have another problem though. The Ubuntu host crashes a lot 
> > > > > when
> > > > > the Windows 8 guest is running. Mostly it happens when I try to 
> > > > > shut
> > > > > down Windows - I think it might be a result of the FirePro card 
> > > > > being
> > > > > released -, but sometimes it happens randomly too. The screen 
> > > > > completely
> > > > > freezes, and no ssh or anything. I should mention that I'm not 
> > > > > passing
> > > > > through the audio pci device for the FirePro, although it is 
> > > > > bound to
> > > > > pciback so the host can't be using it either. If I try to pass 
> > > > > that
> > > > > through it never actually boots. I've had a lot of blue screens, 
> > > > > and a
> > > > > lot of host crashes. I'm passing through a USB device too (i.e. 
> > > > > the PCI
> > > > > USB device), and that works fine.
> > > > >
> > > > > Googling doesn't seem to come up with much for host crashes. What 
> > > > > can I
> > > > > do to diagnose the problem? There's nothing in the syslog.
> > > > Can you try it with domU limited to 1GB of RAM? Also, can you add 
> > > > your physical e820 memory map as reported in dmesg?
> > > > It is plausible you might be running into the same issue I had 
> > > > with domU memory overwriting the physical IOMEM without getting 
> > > > remapped by the IOMMU.
> > > > Does your motherboard feature NF200 PCIe bridges?
> > > >
> > > > Gordan
> > > Your overwriting memory suggestion sounds likely. I've tried with 
> > > 1GiB
> > > but the host still crashes a lot. The first time I ran it it 
> > > booted,
> > > and then shutdown fine, but starting it again crashed the host.
> > That sounds like normal behaviour. Rebooting domU with ATI cards in 
> > passthrough does tend to result in crashing the host. I had this 
> > problem, too, without the memory overwriting issue.
> > This is one of the main reasons why I gave up on using ATI cards and 
> > switched to Nvidia. I don't consider ATI cards fit for the purpose of 
> > VGA passthrough.
> > > The
> > > second and third tries resulted in a host crash before Windows had
> > > booted even once. Sometimes as the host crashes the mouse movement
> > > slows down, before stopping altogether. Would this make sense if 
> > > the
> > > host's memory were being overwritten?
> > No, that sounds like "normal" broken behaviour with ATI cards.
> >
> > > Is this the memory map you wanted?:
> > >
> > > [root@yeti /etc/xen]% dmesg | grep e820
> > > [    0.000000] e820: BIOS-provided physical RAM map:
> > > [    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> 
> > > reserved
> > > [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
> > > [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
> > > [    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
> > > [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI 
> > > devices
> > > [    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
> > > [    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
> > > [    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
> > > [    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]
> > Not quite, the block you are looking for is:
> > e820: BIOS-provided physical RAM map:
> > Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
> > Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
> > Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
> > Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
> > Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
> > Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
> > Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
> > Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> > Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
> > Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
> >
> >
> > > The motherboard is an EVGA SR-X, and I'm not sure how to tell 
> > > whether
> > > it has NF200 PCIe bridges.
> > I fear you'll find it does. I have an SR-2. The SR-X almost 
> > certainly also has the same memory overwrite issue (NF200 bridges 
> > bypass the IOMMU) that I've been fighting. My advice, unless you are 
> > very bloody minded are prepared to either help write patches or use 
> > highly unsupported patches on self-built Xen packages, is to trade in 
> > your EVGA board for one made by a competent manufacturer. Otherwise 
> > you are in for a whole world of frustration and time wasting. I speak 
> > from experience. I'm persevering with the SR-2 because it has some 
> > tuning potential to it, unlike the SR-X, which makes the SR-X very 
> > much not worth bothering with. Persevere only if you deem your time 
> > worthless.
> > On the off-chance you do decide to persevere, despite the well 
> > intentioned advice above, you may want to look through the archives 
> > for a thread about HVM e820_host support. The patches on that thread 
> > are very much unfit for public consumption, but they do get around the 
> > memory stomp problem for me.
> > Unfortunately, I don't think the symptoms you are seeing (domU 
> > crashing on reboot) are to do with the memory stomp. It is a common 
> > Xen issue, somewhat less bad with certain quadrified Nvidia cards than 
> > with ATI cards. For example, I have 2 domUs with GPU passthrough. They 
> > are based off the same cloned image. One always handled rebooting OK 
> > (GTX470 modified to Quadro 5000), one often seemed to fail to 
> > initialize the GPU (GTX480 modified to Quadro 6000). It _may_ have 
> > been related to what slot which card was in, I never quite got to the 
> > bottom of it. When it fails to init the Nvidia card, it would come up 
> > with emulated VGA output on VNC. I since upgraded the faux-Quadro 6000 
> > to a faux-Quadro K5000 (modified GTX680) and the problem went away - I 
> > can now reboot both domUs, but the GTX680 in domU has other issues 
> > (e.g. no output when using a dual-link DVI mode - something seemingly 
> > unique to my setup since David doesn't have the same issue with his 
> > modified GTX680).
> > I have never managed to get an ATI passthrough setup to usably 
> > survive a domU reboot.
> > Gordan
> Thanks so much for all the advice. It seems to be stable enough once
> booted, so I think I'll just accept that a reboot of Windows requires
> a reboot of dom0 too. I don't really need to be able to turn it off!
One thing that some people have found helps on Windows 7 and later
is to eject the GPU as a hot-plug device (like you would dismount
a USB stick, from the removable hardware icon in the tray) before
rebooting. The driver seems to know how to reset the card into a
clean state if you do that. The only problem is that you then lose
GPU output which makes shutting down difficult.

What you might be able to do is write something in powershell that
hooks on shutdown and ejects the GPU.

I never got as far as working around that level of issues with ATI
cards because their limitations proved to be completely unworkable
with my T221 monitors.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users