[Xen-users] PCI Passthrough Problems/Questions

["Nick Couchman" <Nick.Couchman@xxxxxxxxx>]

I've asked a couple of these questions, already, but haven't really had
any response, so I thought I'd try, again, and try to provide a little
more detail.  I need to pass through a PCIe 4-port Serial card.  I've
done this successfully multiple times with different serial cards, but
the current one, a PCIe Perle quad-port RJ45, is causing some problems.
I believe the problem stems from the fact that the card is a PCI card
and that Perle just built a PCIe-PCI bridge so that they could reuse an
existing PCI design.  The following three PCI devices are all on the
same card:

# lspci
...
01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
Express-to-PCI Bridge (rev aa)
02:00.0 Serial controller: Perle Systems Ltd Device 0331
02:00.1 Bridge: Perle Systems Ltd Device 0331

I've tried various combinations of hiding 01:00.0, 02:00.0, and/or
02:00.1 from dom0 and passing those through.  First, if I try all three,
I put the following in my dom0 kernel arguments:

pciback.hide=(01:00.0)(02:00.0)(02:00.1)

Before someone suggests that maybe I have my kernel arguments wrong,
this is a SuSE kernel, so it is not pvops, and that is the correct
syntax - I can add other PCI(e) devices on the same exact host (like a
graphics card) and pass those through without any problem at all.  So
syntax is correct.  After I boot with these arguments, I check the dmesg
output for pciback messages:

# dmesg|grep pciback
[    0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap
pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts
[    0.000000] Kernel command line: root=/dev/local/dom0
resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1)
splash=silent quiet showopts
[    3.646529] pciback 0000:01:00.0: seizing device
[    3.646536] pciback 0000:02:00.0: seizing device
[    3.646540] pciback 0000:02:00.1: seizing device
[    3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[    3.646599] pciback 0000:02:00.1: PCI INT A disabled
[    3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[    3.646642] pciback 0000:02:00.0: PCI INT A disabled

So, everything points to the devices having been removed from dom0's
visibility and placed under the control of pciback.  A check with "xm"
confirms this:

# xm pci-list-assignable-devices
0000:02:00.0
0000:02:00.1
0000:01:00.0

Next, I set up the pci pass through in my HVM domU configuration.  By
the way, VTd is enabled and working fine - xm dmesg shows it as enabled,
and, as already mentioned, I can forward through other PCI(e) devices
without problem.  I set the following in my domU configuration:

pci=['01:00.0','02:00.0','02:00.1']

and then I try to start the domU:

# xm start XP
Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.0
Usage: xm start <DomainName>

???  Ummm, actually, the device is owned by pciback, so why is Xen
telling me it's not??  I can confirm via lspci -v:

# lspci -v
...
01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at d0000000 (64-bit, prefetchable) [size=64K]
        Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: fe500000-fe5fffff
        Capabilities: [40] Power Management version 2
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00
        Capabilities: [100] Power Budgeting <?>
        Kernel driver in use: pciback

Any hints on this particular part of the problem would be appreciated.
My next step was to leave this one out and just try forwarding through
02:00.0 and 02:00.1.  However, when I modify the domU configuration and
remove the 01:00.0 device and try to start the domU:

# xm start XP
Error: Failed to assign device to IOMMU
(0000:02:00.0@100,msitranslate=1,power_mgmt=0)
Usage: xm start <DomainName>

I've been able to find very, very little useful information on this
particular error - most of the issues are with trying to forward through
one or two of the devices on a card to one domU and the others to
another or not at all - splitting the devices on a card between domains.
I'm not really trying to do this, but I suspect that there's some sort
of tie between the PCIe-PCI bridge device (01:00.0) and the actual
serial card (02:00.0) that is causing Xen to need to have both those
devices present in the same domain.  Of course, because of the problems
I'm having forwarding the bridge through, that's not really working for
me.

Environment:
Xen: 4.0.0 (4.0.0_21091_06-0.1)
Kernel: 2.6.34.7 (2.6.34.7-0.4-xen, not pvops)
Hardware: Dell Optiplex 780

Any help, hints, information, would be greatly appreciated!

-Nick




--------
This e-mail may contain confidential and privileged material for the sole use 
of the intended recipient.  If this email is not intended for you, or you are 
not responsible for the delivery of this message to the intended recipient, 
please note that this message may contain SEAKR Engineering (SEAKR) 
Privileged/Proprietary Information.  In such a case, you are strictly 
prohibited from downloading, photocopying, distributing or otherwise using this 
message, its contents or attachments in any way.  If you have received this 
message in error, please notify us immediately by replying to this e-mail and 
delete the message from your mailbox.  Information contained in this message 
that does not relate to the business of SEAKR is neither endorsed by nor 
attributable to SEAKR.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users