[Xen-devel] PCI passthrough to QEMU traditional stubdom not working when option ROM present

[Eric Shelton <eshelton@xxxxxxxxx>]

I am running Xen 4.6.3 (with Marek's recent patches at
https://lists.xen.org/archives/html/xen-devel/2016-10/msg01122.html)
and Linux 4.4.14.

qemu-traditional (running in a stubdom) exits due to an error when I
try doing passthrough of a PCI device that includes an
option/expansion ROM.  Specifically, the console log for the dm domain
shows:

= = = =
pcifront_watches: waiting for backend to get into the right state
/local/domain/0/backend/pci/19/0

******************* PCIFRONT for device/pci/0 **********


backend at /local/domain/0/backend/pci/19/0
**************************
pcifront_watches: waiting for backend events
/local/domain/0/backend/pci/19/0/state
xs_read_watch() -> device-model/18/command dm-command
dm-command: hot insert pass-through pci dev
warning: pt_iomul not supported in stubdom 05:00.0
ERROR: PCI region size must be pow2 type=0x8, size=0xdf080000
close(0)
= = = =

The size shown in the error message is actually the base address for
the expansion ROM:

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
    Subsystem: Intel Corporation Gigabit CT Desktop Adapter
    Flags: fast devsel, IRQ 19
    Memory at df0c0000 (32-bit, non-prefetchable) [disabled] [size=128K]
    Memory at df000000 (32-bit, non-prefetchable) [disabled] [size=512K]
    I/O ports at e000 [disabled] [size=32]
    Memory at df0e0000 (32-bit, non-prefetchable) [disabled] [size=16K]
    Expansion ROM at df080000 [disabled] [size=256K]

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
00: 86 80 d3 10 00 00 10 00 00 00 00 02 00 00 00 00
10: 00 00 0c df 00 00 00 df 01 e0 00 00 00 00 0e df
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 08 df c8 00 00 00 00 00 00 00 0b 01 00 00

The expansion ROM size appears to be getting set by this portion of
stubdom/pciutils.patch:
= = = =
          u32 u = pci_read_long(d, reg);
          if (u != 0xffffffff)
-           d->rom_base_addr = u;
+            {
+              d->rom_base_addr = u;
+              if (flags & PCI_FILL_SIZES)
+                {
+                  u32 size;
+                  pci_write_long(d, reg, ~0);
+                  d->rom_size = pci_read_long(d, reg);
+                  pci_write_long(d, reg, u);
+                }
+            }
= = = =

It looks like there are a few issues going on with this:
(1) The expansion ROM BAR at 0x30 appears to be read only, so the
write of ~0 to determine its size is not working.  As a result,
d->rom_size is getting set to the base address for the expansion ROM.
I assume 0x30 being read only is a pciback issue, but I don't know if
changes after 4.4.14 have affected this - I see there have been
changes to rom_write() and rom_init() in conf_space_header.c
(2) Even if that write issue wasn't happening, the above patch does
not look like the right way to determine the size of the expansion ROM
anyway.  For example, with the example device above having a 256K
expansion ROM, I believe a write of 0xffffffff to 0x30 would result in
a value of 0xfffc0001 (the lowest bit is an address decode
enable/disable), which we would not want to store in d->rom_size.
Instead, something like:
  d->rom_size = pci_size(u, pci_read_long(d, reg), PCI_ROM_ADDRESS_MASK);
probably should be done instead, in the same way the other BARs are being sized.
(3) (minor) When QEMU errors out, it takes a while for xl to time out
on it.  Perhaps it would make sense for QEMU to set something in
xenstore on its way out to let xl know it has errored out.

Can someone provide comments on items 1 and 2 above?  In particular, I
am not sure what the right answer is for 1.

Thanks,
Eric

PS: While looking into this, it looks like the patch discussed at
https://lists.xen.org/archives/html/xen-devel/2014-05/msg01465.html
(Prevent QEMU from mapping PCI option ROM at address 0) for
qemu-traditional never ended up getting resolved.  It seems like
keeping anything - particularly a reflashable option ROM - from
getting mapped to the NULL page would be a good idea.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel