[Xen-users] HVM PCI Passthrough: Code 12: Undersized PCI MMIO region?

[Stephen Oberholtzer <stevie@xxxxxxxxx>]

All,

Please bear with me.  I don't fully understand all of the mechanisms
involved here, so this message contains a large quantity of data,
including some links to even larger quantities of data.
The short version is: I think something isn't reserving enough (GPFN)
address space for MMIO.

I'm trying to do a three-part PCI passthrough to a HVM domU running Windows 8.1:
00:14.0, An xHCI (USB 3.0) controller
01:00.0, Function 0 of an ATI card (VGA controller, non-primary passthrough)
01:00.1, Function 1 of the same ATI card (HDMI audio)

The USB controller seems to work fine.  USB mouse/keyboard are
functional, and I plugged in a few Mass Storage devices and they work
fine.
The HDMI audio controller is harder to vet, because I don't have
anything that will process audio without an associated video stream.
However, I don't get any complains from the software.

For the the VGA controller (01:00.0), Windows reports "Code 12" and
complains about insufficient resources being available for the device.

I tried to gather all of the information I could on the memory layouts
involved, from three sources:
(1) From dom0 and Xen itself
(2) From domU running Windows, using a program called "SIV" (System
Information Viewer)
(3) From domU running Linux (by booting a Knoppix live DVD image)

---

Information from dom0:
http://www.klozoff.org/xen/dom0-xl-info.txt - 'xl info'
http://www.klozoff.org/xen/dom0-xl-dmesg.txt - 'xl dmesg'
http://www.klozoff.org/xen/dom0-proc_iomem.txt - 'cat /proc/iomem'
http://www.klozoff.org/xen/dom0-lspci-output.txt - 'lspci -vvv' for
the three devices I'm forwarding.

Information from domU (Windows):
http://www.klozoff.org/xen/domU-device-manager-info.txt - Text
transcribed from Windows' "Device Manager"
http://www.klozoff.org/xen/domU-siv-usb-controller-info.txt -
Information on 00:14.0 from SIV
http://www.klozoff.org/xen/domU-siv-video-card-info.txt - Information
on 01:00.0 from SIV
http://www.klozoff.org/xen/domU-siv-hdmi-audio-info.txt - Information
on 01:00.1 from SIV

Information from domU (Linux/Knoppix):
http://www.klozoff.org/xen/domU-knoppix-proc-iomem.txt - 'cat /proc/iomem'
http://www.klozoff.org/xen/domU-knoppix-lspci-output.txt - 'lspci
-vvv' for the three devices.


---

Of particular interest to me is domU-knoppix-lspci-output.txt, which
for the VGA controller, says:

    Region 0: Memory at <ignored> (64-bit, prefetchable)
    Region 2: Memory at f3040000 (64-bit, non-prefetchable) [size=256K]

In dom0, this showed as:

    Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
    Region 2: Memory at efc00000 (64-bit, non-prefetchable) [size=256K]


I looked closer by running the following script:

perl -lne '/^([0-9a-f]+)-([0-9a-f]+)/ && print $_, " (", ((hex($2) -
hex($1)) >> 20), " MB)"' /proc/iomem

In dom0, I see this:

00100000-99f53fff : System RAM (2462 MB)
bf200000-feafffff : PCI Bus 0000:00 (1016 MB)   <- Region 0 is in this range
100000000-83edfffff : System RAM (29677 MB)


In domU, I see this:
00100000-effecfff : System RAM (3838 MB)
f0000000-fbffffff : PCI Bus 0000:00 (191 MB)    <- Definitely not big
enough to hold the 256MB we need
100000000-20f7fffff : System RAM (4343 MB)

Now, I don't fully understand how this stuff all works, but here's
what appears to be happening: Xen isn't reserving enough contiguous
GPFN (virtual machine address space) for all of the devices on the
virtual PCI bus -- in particular, for what is probably a big honkin'
window into the card's onboard RAM.

So my two questions are:
(1) Is my guess correct?  Or at least close?
(2) What the heck do I do about it? I don't see any options in
xl.cfg(5) for specifying the size of the MMIO region for the virtual
PCI bus, or anything similar to that.


NOTE: If I assign 2048MB or less RAM to this VM, the VM crashes at
bootup, and has a nasty tendency to take the host with it (lockup).

-- 
-- Stevie-O
Real programmers use COPY CON PROGRAM.EXE

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users