[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] VGA Passthrough of AMD FirePro W7000 to Windows 8: Not enough resources

On 04/10/13 14:36, Gordan Bobic wrote:
On Wed, 02 Oct 2013 18:12:11 +0100, Toby Miller <tobycmiller@xxxxxxxxx> wrote:
On 02/10/13 15:17, Gordan Bobic wrote:
On 10/02/2013 03:01 PM, Toby Miller wrote:
On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@xxxxxxxxxx> wrote:

On 10/02/2013 01:58 PM, Toby Miller wrote:
gordan@xxxxxxxxxx wrote:
  On Sun, 29 Sep 2013 15:01:01 +0100, Toby
Miller<tobycmiller@xxxxxxxxx>   wrote:
I'm using Xen 4.3, installed from source on Ubuntu 13.04. I've
successfully installed a Windows 8 guest, and had Xen pass it an AMD FirePro W7000 GPU (technically 2 PCI devices - the second is for the sound), as well as the virtualised graphics adapter. Windows finds
card fine, and I've installed AMD's drivers. Device Manager says that the card cannot find enough resources (code 12). I've checked the IO,
IRQ, and Memory details in DM, and there don't seem to be any

I did try disabling the virtualised card from Windows, but it seemed
to keep using it anyway, even after a restart. It didn't make any
difference to the FirePro in any case.

I'd be grateful for some help.
I haven't tried Windows 8, but on XP64 disabling the Cirrus card in device manager fixed the problem for me with Nvidia card passthrough.

  You could try setting gfx_passthru=1, which should disable the
Cirrus emulation alltogether. That may make Windows domU completely
  fail to boot, though.

  Did you do a clean Win8 install and reboot the host fully? Is the
FirePro the primary GPU on the host? Or is it completely untouched
  in dom0 (no dom0 or BIOS output on it)?

Thanks for your reply. It turns out the problem was that I'd forgotten
to put device_model_version="qemu-xen-traditional", so I was using a
qemu that didn't support gfx_passthrough. The FirePro is not used in
dom0 at all. I'm using pciback to hide it.

I now have another problem though. The Ubuntu host crashes a lot when
the Windows 8 guest is running. Mostly it happens when I try to shut
down Windows - I think it might be a result of the FirePro card being released -, but sometimes it happens randomly too. The screen completely freezes, and no ssh or anything. I should mention that I'm not passing through the audio pci device for the FirePro, although it is bound to
pciback so the host can't be using it either. If I try to pass that
through it never actually boots. I've had a lot of blue screens, and a lot of host crashes. I'm passing through a USB device too (i.e. the PCI
USB device), and that works fine.

Googling doesn't seem to come up with much for host crashes. What can I
do to diagnose the problem? There's nothing in the syslog.

Can you try it with domU limited to 1GB of RAM? Also, can you add your physical e820 memory map as reported in dmesg?

It is plausible you might be running into the same issue I had with domU memory overwriting the physical IOMEM without getting remapped by the IOMMU.

Does your motherboard feature NF200 PCIe bridges?


Your overwriting memory suggestion sounds likely. I've tried with 1GiB
but the host still crashes a lot. The first time I ran it it booted,
and then shutdown fine, but starting it again crashed the host.

That sounds like normal behaviour. Rebooting domU with ATI cards in passthrough does tend to result in crashing the host. I had this problem, too, without the memory overwriting issue.

This is one of the main reasons why I gave up on using ATI cards and switched to Nvidia. I don't consider ATI cards fit for the purpose of VGA passthrough.

second and third tries resulted in a host crash before Windows had
booted even once. Sometimes as the host crashes the mouse movement
slows down, before stopping altogether. Would this make sense if the
host's memory were being overwritten?

No, that sounds like "normal" broken behaviour with ATI cards.

Is this the memory map you wanted?:

[root@yeti /etc/xen]% dmesg | grep e820
[    0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
[ 0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices
[    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
[    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
[    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
[    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]

Not quite, the block you are looking for is:
e820: BIOS-provided physical RAM map:
Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable

The motherboard is an EVGA SR-X, and I'm not sure how to tell whether
it has NF200 PCIe bridges.

I fear you'll find it does. I have an SR-2. The SR-X almost certainly also has the same memory overwrite issue (NF200 bridges bypass the IOMMU) that I've been fighting. My advice, unless you are very bloody minded are prepared to either help write patches or use highly unsupported patches on self-built Xen packages, is to trade in your EVGA board for one made by a competent manufacturer. Otherwise you are in for a whole world of frustration and time wasting. I speak from experience. I'm persevering with the SR-2 because it has some tuning potential to it, unlike the SR-X, which makes the SR-X very much not worth bothering with. Persevere only if you deem your time worthless.

On the off-chance you do decide to persevere, despite the well intentioned advice above, you may want to look through the archives for a thread about HVM e820_host support. The patches on that thread are very much unfit for public consumption, but they do get around the memory stomp problem for me.

Unfortunately, I don't think the symptoms you are seeing (domU crashing on reboot) are to do with the memory stomp. It is a common Xen issue, somewhat less bad with certain quadrified Nvidia cards than with ATI cards. For example, I have 2 domUs with GPU passthrough. They are based off the same cloned image. One always handled rebooting OK (GTX470 modified to Quadro 5000), one often seemed to fail to initialize the GPU (GTX480 modified to Quadro 6000). It _may_ have been related to what slot which card was in, I never quite got to the bottom of it. When it fails to init the Nvidia card, it would come up with emulated VGA output on VNC. I since upgraded the faux-Quadro 6000 to a faux-Quadro K5000 (modified GTX680) and the problem went away - I can now reboot both domUs, but the GTX680 in domU has other issues (e.g. no output when using a dual-link DVI mode - something seemingly unique to my setup since David doesn't have the same issue with his modified GTX680).

I have never managed to get an ATI passthrough setup to usably survive a domU reboot.


Thanks so much for all the advice. It seems to be stable enough once
booted, so I think I'll just accept that a reboot of Windows requires
a reboot of dom0 too. I don't really need to be able to turn it off!

One thing that some people have found helps on Windows 7 and later
is to eject the GPU as a hot-plug device (like you would dismount
a USB stick, from the removable hardware icon in the tray) before
rebooting. The driver seems to know how to reset the card into a
clean state if you do that. The only problem is that you then lose
GPU output which makes shutting down difficult.

What you might be able to do is write something in powershell that
hooks on shutdown and ejects the GPU.

I never got as far as working around that level of issues with ATI
cards because their limitations proved to be completely unworkable
with my T221 monitors.

That's a good idea. I'll try that.



Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.