Re: [Xen-devel] PCI Passthrough Problems/Questions

["Nick Couchman" <Nick.Couchman@xxxxxxxxx>]

On Mon, 2010-10-25 at 15:07 -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Oct 25, 2010 at 12:50:41PM -0600, Nick Couchman wrote:
> > On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote:
> > > On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote:
> > > > On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote:
> > > > 
> > > > > 
> > > > > What do you see on your Xen serial output? I presume you cranked up 
> > > > > logging:
> > > > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line.
> > > > > 
> > > > > Is there anything that shows up when you get the 'Failed to assign.." 
> > > > > ?
> > > > > 
> > > > 
> > > > The only messages I get on the serial console after setting those
> > > > parameters on the xen.gz kernel line in grub (and rebooting, of course)
> > > > are the following:
> > > > 
> > > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0
> > > > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0
> > > > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0)
> > > > failed
> > > > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22
> > > 
> > > So, -EINVAL. How comfortable are you sticking a bunch of
> > > dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? 
> > > Basically
> > > you need to figure which of the functions that are past line 1364
> > > are being called and return -EINVAL. 
> > 
> > I'm happy to give it a shot...it'll take a while to get the devel
> > environment configured, as I'm using packages right now and I don't even
> > think I have a compiler on this system.  I'll report back once I get
> > that done and give that a try.
> 
> Excellent. You might also want to CC Weidong (weidong.han@xxxxxxxxx) in the 
> future
> who is right now on travel and he might have better suggestions. CC-ing him 
> on this e-mail.

Okay, I'm fairly certain that the root cause is actually the issue Xen
has trying to forward through the actual PCIe-to-PCI bridge.  In my
original e-mail, I stated that if I try to add that bridge to the domU,
I get an error about it not being owned by PCI Backend or pci-stub.  In
the function domain_context_mapping, bus is the bus of the device you're
actually trying to forward through to the new domain, and secbus is set
with the function find_upstream_bridge, which appears to find the bus to
which your device is attached.  Here's the section of code in question
from iommu.c:

        if ( pdev_type(bus, devfn) == DEV_TYPE_PCIe2PCI_BRIDGE )
        {
            ret = domain_context_mapping_one(domain, drhd->iommu, bus,
devfn);
            dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one
ret: %d\n", ret);
            if ( ret )
                return ret;

            /*
             * Devices behind PCIe-to-PCI/PCIx bridge may generate
             * different requester-id. It may originate from devfn=0
             * on the secondary bus behind the bridge. Map that id
             * as well.
             */
            ret = domain_context_mapping_one(domain, drhd->iommu,
secbus, 0);
            dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one
ret: %d\n", ret);
        }
        else /* Legacy PCI bridge */ {
            ret = domain_context_mapping_one(domain, drhd->iommu, bus,
devfn);
            dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one
ret: %d\n", ret);
        }

If I understand this code, if the device is a PCIe2PCI bridge, the first
thing that's done is that the device itself is mapped, then the upstream
bridge device is mapped (secbus with devfn=0).  This is where the
command is failing for the device that I'm trying to forward.  So, the
question is, why can't Xen seem to forward through the bridge itself?
It doesn't seem to matter whether I have that particular bridge listed
on the kernel line in the pciback.hide section - Xen still thinks that
dom0 has a hold of that device and won't forward it on.

Where do I go from here?

-Nick



--------
This e-mail may contain confidential and privileged material for the sole use 
of the intended recipient.  If this email is not intended for you, or you are 
not responsible for the delivery of this message to the intended recipient, 
please note that this message may contain SEAKR Engineering (SEAKR) 
Privileged/Proprietary Information.  In such a case, you are strictly 
prohibited from downloading, photocopying, distributing or otherwise using this 
message, its contents or attachments in any way.  If you have received this 
message in error, please notify us immediately by replying to this e-mail and 
delete the message from your mailbox.  Information contained in this message 
that does not relate to the business of SEAKR is neither endorsed by nor 
attributable to SEAKR.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel