Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Venu Busireddy <venu.busireddy@xxxxxxxxxx>]

On Wed, Mar 15, 2017 at 04:38:39PM +0000, Roger Pau Monn? wrote:
> On Wed, Mar 15, 2017 at 10:11:35AM -0500, Venu Busireddy wrote:
> > On Wed, Mar 15, 2017 at 12:56:50PM +0000, Roger Pau Monn? wrote:
> > > On Wed, Mar 15, 2017 at 08:42:04AM -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Wed, Mar 15, 2017 at 12:07:28PM +0000, Roger Pau Monn? wrote:
> > > > > On Fri, Mar 10, 2017 at 10:28:43AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > > > On Fri, Mar 10, 2017 at 12:23:18PM +0900, Roger Pau Monn? wrote:
> > > > > > > On Thu, Mar 09, 2017 at 07:29:34PM -0500, Konrad Rzeszutek Wilk 
> > > > > > > wrote:
> > > > > > > > On Thu, Mar 09, 2017 at 01:26:45PM +0000, Julien Grall wrote:
> > > > > > > > > Hi Konrad,
> > > > > > > > > 
> > > > > > > > > On 09/03/17 11:17, Konrad Rzeszutek Wilk wrote:
> > > > > > > > > > On Thu, Mar 09, 2017 at 11:59:51AM +0900, Roger Pau Monn? 
> > > > > > > > > > wrote:
> > > > > > > > > > > On Wed, Mar 08, 2017 at 02:12:09PM -0500, Konrad 
> > > > > > > > > > > Rzeszutek Wilk wrote:
> > > > > > > > > > > > On Wed, Mar 08, 2017 at 07:06:23PM +0000, Julien Grall 
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > .. this as for SR-IOV devices you need the drivers to 
> > > > > > > > > > > > kick the hardware
> > > > > > > > > > > > to generate the new bus addresses. And those (along 
> > > > > > > > > > > > with the BAR regions) are
> > > > > > > > > > > > not visible in ACPI (they are constructued dynamically).
> > > > > > > > > > > 
> > > > > > > > > > > There's already code in Xen [0] to find out the size of 
> > > > > > > > > > > the BARs of SR-IOV
> > > > > > > > > > > devices, but I'm not sure what's the intended usage of 
> > > > > > > > > > > that, does it need to
> > > > > > > > > > > happen _after_ the driver in Dom0 has done whatever magic 
> > > > > > > > > > > for this to work?
> > > > > > > > > > 
> > > > > > > > > > Yes. This is called via the PHYSDEVOP_pci_device_add 
> > > > > > > > > > hypercall when
> > > > > > > > > > the device driver in dom0 has finished "creating" the VF. 
> > > > > > > > > > See drivers/xen/pci.c
> > > > > > > > > 
> > > > > > > > > We are thinking to not use PHYSDEVOP_pci_device_add hypercall 
> > > > > > > > > for ARM and do
> > > > > > > > > the PCI scanning in Xen.
> > > > > > > > > 
> > > > > > > > > If I understand correctly what you said, only the PCI driver 
> > > > > > > > > will be able to
> > > > > > > > > kick SR-IOV device and Xen would not be able to detect the 
> > > > > > > > > device until it
> > > > > > > > > has been fully configured. So it would mean that we have to 
> > > > > > > > > keep
> > > > > > > > > PHYSDEVOP_pci_device_add around to know when Xen can use the 
> > > > > > > > > device.
> > > > > > > > > 
> > > > > > > > > Am I correct?
> > > > > > > > 
> > > > > > > > Yes. Unless the PCI drivers come up with some other way to tell 
> > > > > > > > the
> > > > > > > > OS that oh, hey, there is this new PCI device with this BDF.
> > > > > > > > 
> > > > > > > > Or the underlaying bus on ARM can send some 'new device' 
> > > > > > > > information?
> > > > > > > 
> > > > > > > Hm, is this something standard between all the SR-IOV 
> > > > > > > implementations, or each
> > > > > > > vendors have their own sauce?
> > > > > > 
> > > > > > Gosh, all of them have their own sauce. The only thing that is the 
> > > > > > same
> > > > > > is that suddenly behind the PF device there are PCI devies that are 
> > > > > > responding
> > > > > > to 0xcfc requests. MAgic!
> > > > > 
> > > > > I'm reading the PCI SR-IOV 1.1 spec, and I think we don't need to 
> > > > > wait for the
> > > > > device driver in Dom0 in order to get the information of the VF 
> > > > > devices, what
> > > > > Xen cares about is the position of the BARs (so that they can be 
> > > > > mapped into
> > > > > Dom0 at boot), and the PCI SBDF of each PF/VF, so that Xen can trap 
> > > > > accesses to
> > > > > it.
> > > > > 
> > > > > AFAICT both of this can be obtained without any driver-specific code, 
> > > > > since
> > > > > it's all contained in the PCI SR-IOV spec (but maybe I'm missing 
> > > > > something).
> > > > 
> > > > CC-ing Venu,
> > > > 
> > > > Roger, could you point out which of the chapters has this?
> > > 
> > > This would be chapter 2 ("Initialization and Resource Allocation"), and 
> > > then
> > > there's a "IMPLEMENTATION NOTE" that shows how the PF/VF are matched to
> > > function numbers in page 45 (I have the following copy, which is the 
> > > latest
> > > revision: "Single Root I/O Virtualization and Sharing Specification 
> > > Revision
> > > 1.1" from January 20 2010 [0]).
> > > 
> > > The document is quite complex, but it is a standard that all SR-IOV 
> > > devices
> > > should follow so AFAICT Xen should be able to get all the information 
> > > that it
> > > needs from the PCI config space in order to detect the PF/VF BARs and the 
> > > BDF
> > > device addresses.
> > > 
> > > Roger.
> > > 
> > > [0] https://members.pcisig.com/wg/PCI-SIG/document/download/8238
> > 
> > I do not have access to this document, so I have to rely on Rev 1.0
> > document, but I don't think this aspect of the spec changed much.
> > 
> > In any case, I am afraid I am not seeing the overall picture, but I
> > would like to comment on the last part of this discussion. Indeed, the
> > configuration space (including the SR-IOV extended capability) contains
> > all the information, but only the information necessary for the OS to
> > "enumerate" the device (PF as well as VFs). The bus and device number
> > (SBDF) assignment, and programming of the BARs, are all done during that
> > enumeration. In this discussion, which entity is doing the enumeration?
> > Xen, or Dom0?
> 
> Xen needs to let Dom0 manage the device, but at the same time it needs to
> correctly map the device BARs into Dom0 physmap. I think the easiest solution
> is to let Dom0 manage the device, and Xen should setup a trap to detect Dom0
> setting the VF Enable bit (bit 0 in SR-IOV Control (08h)), at which point Xen
> will size the VF BARs (and map them into Dom0) and also enumerate the VF
> devices.

There was a second part in my earlier email. Copied below:

"If Xen waits until Dom0 enumerated the device, then the BAR positions
are already within the Dom0's memory space! No further mapping is needed,
right?"

As I asked, if Dom0 enumerates the device and programs the BARs, the BAR
regions are already in Dom0's physical memory! What further mapping is
needed? What am I missing?

Venu



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel