Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Mar 15, 2017 at 08:42:04AM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Mar 15, 2017 at 12:07:28PM +0000, Roger Pau Monné wrote:
> > On Fri, Mar 10, 2017 at 10:28:43AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Mar 10, 2017 at 12:23:18PM +0900, Roger Pau Monné wrote:
> > > > On Thu, Mar 09, 2017 at 07:29:34PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > > On Thu, Mar 09, 2017 at 01:26:45PM +0000, Julien Grall wrote:
> > > > > > Hi Konrad,
> > > > > > 
> > > > > > On 09/03/17 11:17, Konrad Rzeszutek Wilk wrote:
> > > > > > > On Thu, Mar 09, 2017 at 11:59:51AM +0900, Roger Pau Monné wrote:
> > > > > > > > On Wed, Mar 08, 2017 at 02:12:09PM -0500, Konrad Rzeszutek Wilk 
> > > > > > > > wrote:
> > > > > > > > > On Wed, Mar 08, 2017 at 07:06:23PM +0000, Julien Grall wrote:
> > > > > > > > > .. this as for SR-IOV devices you need the drivers to kick 
> > > > > > > > > the hardware
> > > > > > > > > to generate the new bus addresses. And those (along with the 
> > > > > > > > > BAR regions) are
> > > > > > > > > not visible in ACPI (they are constructued dynamically).
> > > > > > > > 
> > > > > > > > There's already code in Xen [0] to find out the size of the 
> > > > > > > > BARs of SR-IOV
> > > > > > > > devices, but I'm not sure what's the intended usage of that, 
> > > > > > > > does it need to
> > > > > > > > happen _after_ the driver in Dom0 has done whatever magic for 
> > > > > > > > this to work?
> > > > > > > 
> > > > > > > Yes. This is called via the PHYSDEVOP_pci_device_add hypercall 
> > > > > > > when
> > > > > > > the device driver in dom0 has finished "creating" the VF. See 
> > > > > > > drivers/xen/pci.c
> > > > > > 
> > > > > > We are thinking to not use PHYSDEVOP_pci_device_add hypercall for 
> > > > > > ARM and do
> > > > > > the PCI scanning in Xen.
> > > > > > 
> > > > > > If I understand correctly what you said, only the PCI driver will 
> > > > > > be able to
> > > > > > kick SR-IOV device and Xen would not be able to detect the device 
> > > > > > until it
> > > > > > has been fully configured. So it would mean that we have to keep
> > > > > > PHYSDEVOP_pci_device_add around to know when Xen can use the device.
> > > > > > 
> > > > > > Am I correct?
> > > > > 
> > > > > Yes. Unless the PCI drivers come up with some other way to tell the
> > > > > OS that oh, hey, there is this new PCI device with this BDF.
> > > > > 
> > > > > Or the underlaying bus on ARM can send some 'new device' information?
> > > > 
> > > > Hm, is this something standard between all the SR-IOV implementations, 
> > > > or each
> > > > vendors have their own sauce?
> > > 
> > > Gosh, all of them have their own sauce. The only thing that is the same
> > > is that suddenly behind the PF device there are PCI devies that are 
> > > responding
> > > to 0xcfc requests. MAgic!
> > 
> > I'm reading the PCI SR-IOV 1.1 spec, and I think we don't need to wait for 
> > the
> > device driver in Dom0 in order to get the information of the VF devices, 
> > what
> > Xen cares about is the position of the BARs (so that they can be mapped into
> > Dom0 at boot), and the PCI SBDF of each PF/VF, so that Xen can trap 
> > accesses to
> > it.
> > 
> > AFAICT both of this can be obtained without any driver-specific code, since
> > it's all contained in the PCI SR-IOV spec (but maybe I'm missing something).
> 
> CC-ing Venu,
> 
> Roger, could you point out which of the chapters has this?

This would be chapter 2 ("Initialization and Resource Allocation"), and then
there's a "IMPLEMENTATION NOTE" that shows how the PF/VF are matched to
function numbers in page 45 (I have the following copy, which is the latest
revision: "Single Root I/O Virtualization and Sharing Specification Revision
1.1" from January 20 2010 [0]).

The document is quite complex, but it is a standard that all SR-IOV devices
should follow so AFAICT Xen should be able to get all the information that it
needs from the PCI config space in order to detect the PF/VF BARs and the BDF
device addresses.

Roger.

[0] https://members.pcisig.com/wg/PCI-SIG/document/download/8238

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel