Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Roger Pau Monn? <roger.pau@xxxxxxxxxx>]

On Wed, Mar 15, 2017 at 11:54:07AM -0500, Venu Busireddy wrote:
> On Wed, Mar 15, 2017 at 04:38:39PM +0000, Roger Pau Monn? wrote:
> > On Wed, Mar 15, 2017 at 10:11:35AM -0500, Venu Busireddy wrote:
> > > On Wed, Mar 15, 2017 at 12:56:50PM +0000, Roger Pau Monn? wrote:
> > > > On Wed, Mar 15, 2017 at 08:42:04AM -0400, Konrad Rzeszutek Wilk wrote:
> > > > > On Wed, Mar 15, 2017 at 12:07:28PM +0000, Roger Pau Monn? wrote:
> > > > > > On Fri, Mar 10, 2017 at 10:28:43AM -0500, Konrad Rzeszutek Wilk 
> > > > > > wrote:
> > > > > > > On Fri, Mar 10, 2017 at 12:23:18PM +0900, Roger Pau Monn? wrote:
> > > > > > > > On Thu, Mar 09, 2017 at 07:29:34PM -0500, Konrad Rzeszutek Wilk 
> > > > > > > > wrote:
> > > > > > > > > On Thu, Mar 09, 2017 at 01:26:45PM +0000, Julien Grall wrote:
> > > > > > > > > > Hi Konrad,
> > > > > > > > > > 
> > > > > > > > > > On 09/03/17 11:17, Konrad Rzeszutek Wilk wrote:
> > > > > > > > > > > On Thu, Mar 09, 2017 at 11:59:51AM +0900, Roger Pau Monn? 
> > > > > > > > > > > wrote:
> > > > > > > > > > > > On Wed, Mar 08, 2017 at 02:12:09PM -0500, Konrad 
> > > > > > > > > > > > Rzeszutek Wilk wrote:
> > > > > > > > > > > > > On Wed, Mar 08, 2017 at 07:06:23PM +0000, Julien 
> > > > > > > > > > > > > Grall wrote:
> > > > > > > > > > > > > .. this as for SR-IOV devices you need the drivers to 
> > > > > > > > > > > > > kick the hardware
> > > > > > > > > > > > > to generate the new bus addresses. And those (along 
> > > > > > > > > > > > > with the BAR regions) are
> > > > > > > > > > > > > not visible in ACPI (they are constructued 
> > > > > > > > > > > > > dynamically).
> > > > > > > > > > > > 
> > > > > > > > > > > > There's already code in Xen [0] to find out the size of 
> > > > > > > > > > > > the BARs of SR-IOV
> > > > > > > > > > > > devices, but I'm not sure what's the intended usage of 
> > > > > > > > > > > > that, does it need to
> > > > > > > > > > > > happen _after_ the driver in Dom0 has done whatever 
> > > > > > > > > > > > magic for this to work?
> > > > > > > > > > > 
> > > > > > > > > > > Yes. This is called via the PHYSDEVOP_pci_device_add 
> > > > > > > > > > > hypercall when
> > > > > > > > > > > the device driver in dom0 has finished "creating" the VF. 
> > > > > > > > > > > See drivers/xen/pci.c
> > > > > > > > > > 
> > > > > > > > > > We are thinking to not use PHYSDEVOP_pci_device_add 
> > > > > > > > > > hypercall for ARM and do
> > > > > > > > > > the PCI scanning in Xen.
> > > > > > > > > > 
> > > > > > > > > > If I understand correctly what you said, only the PCI 
> > > > > > > > > > driver will be able to
> > > > > > > > > > kick SR-IOV device and Xen would not be able to detect the 
> > > > > > > > > > device until it
> > > > > > > > > > has been fully configured. So it would mean that we have to 
> > > > > > > > > > keep
> > > > > > > > > > PHYSDEVOP_pci_device_add around to know when Xen can use 
> > > > > > > > > > the device.
> > > > > > > > > > 
> > > > > > > > > > Am I correct?
> > > > > > > > > 
> > > > > > > > > Yes. Unless the PCI drivers come up with some other way to 
> > > > > > > > > tell the
> > > > > > > > > OS that oh, hey, there is this new PCI device with this BDF.
> > > > > > > > > 
> > > > > > > > > Or the underlaying bus on ARM can send some 'new device' 
> > > > > > > > > information?
> > > > > > > > 
> > > > > > > > Hm, is this something standard between all the SR-IOV 
> > > > > > > > implementations, or each
> > > > > > > > vendors have their own sauce?
> > > > > > > 
> > > > > > > Gosh, all of them have their own sauce. The only thing that is 
> > > > > > > the same
> > > > > > > is that suddenly behind the PF device there are PCI devies that 
> > > > > > > are responding
> > > > > > > to 0xcfc requests. MAgic!
> > > > > > 
> > > > > > I'm reading the PCI SR-IOV 1.1 spec, and I think we don't need to 
> > > > > > wait for the
> > > > > > device driver in Dom0 in order to get the information of the VF 
> > > > > > devices, what
> > > > > > Xen cares about is the position of the BARs (so that they can be 
> > > > > > mapped into
> > > > > > Dom0 at boot), and the PCI SBDF of each PF/VF, so that Xen can trap 
> > > > > > accesses to
> > > > > > it.
> > > > > > 
> > > > > > AFAICT both of this can be obtained without any driver-specific 
> > > > > > code, since
> > > > > > it's all contained in the PCI SR-IOV spec (but maybe I'm missing 
> > > > > > something).
> > > > > 
> > > > > CC-ing Venu,
> > > > > 
> > > > > Roger, could you point out which of the chapters has this?
> > > > 
> > > > This would be chapter 2 ("Initialization and Resource Allocation"), and 
> > > > then
> > > > there's a "IMPLEMENTATION NOTE" that shows how the PF/VF are matched to
> > > > function numbers in page 45 (I have the following copy, which is the 
> > > > latest
> > > > revision: "Single Root I/O Virtualization and Sharing Specification 
> > > > Revision
> > > > 1.1" from January 20 2010 [0]).
> > > > 
> > > > The document is quite complex, but it is a standard that all SR-IOV 
> > > > devices
> > > > should follow so AFAICT Xen should be able to get all the information 
> > > > that it
> > > > needs from the PCI config space in order to detect the PF/VF BARs and 
> > > > the BDF
> > > > device addresses.
> > > > 
> > > > Roger.
> > > > 
> > > > [0] https://members.pcisig.com/wg/PCI-SIG/document/download/8238
> > > 
> > > I do not have access to this document, so I have to rely on Rev 1.0
> > > document, but I don't think this aspect of the spec changed much.
> > > 
> > > In any case, I am afraid I am not seeing the overall picture, but I
> > > would like to comment on the last part of this discussion. Indeed, the
> > > configuration space (including the SR-IOV extended capability) contains
> > > all the information, but only the information necessary for the OS to
> > > "enumerate" the device (PF as well as VFs). The bus and device number
> > > (SBDF) assignment, and programming of the BARs, are all done during that
> > > enumeration. In this discussion, which entity is doing the enumeration?
> > > Xen, or Dom0?
> > 
> > Xen needs to let Dom0 manage the device, but at the same time it needs to
> > correctly map the device BARs into Dom0 physmap. I think the easiest 
> > solution
> > is to let Dom0 manage the device, and Xen should setup a trap to detect Dom0
> > setting the VF Enable bit (bit 0 in SR-IOV Control (08h)), at which point 
> > Xen
> > will size the VF BARs (and map them into Dom0) and also enumerate the VF
> > devices.
> 
> There was a second part in my earlier email. Copied below:
> 
> "If Xen waits until Dom0 enumerated the device, then the BAR positions
> are already within the Dom0's memory space! No further mapping is needed,
> right?"
> 
> As I asked, if Dom0 enumerates the device and programs the BARs, the BAR
> regions are already in Dom0's physical memory! What further mapping is
> needed? What am I missing?

No, that's not true. Xen is the one that performs the mapping into Dom0 physmap
on PVH (and ARM), so unless Xen has mapped those BARs into Dom0, Dom0 doesn't
have access to the BARs at all. Hence I think Xen should detect Dom0 setting
the VF Enable bit, properly size the VF BARs and map them into Dom0.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel