Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

["Edgar E. Iglesias" <edgar.iglesias@xxxxxxxxxx>]

On Wed, Jan 25, 2017 at 06:53:20PM +0000, Julien Grall wrote:
> Hi Stefano,
> 
> On 24/01/17 20:07, Stefano Stabellini wrote:
> >On Tue, 24 Jan 2017, Julien Grall wrote:
> >>>>## Discovering and register hostbridge
> >>>>
> >>>>Both ACPI and Device Tree do not provide enough information to fully
> >>>>instantiate an host bridge driver. In the case of ACPI, some data may come
> >>>>from ASL,
> >>>
> >>>The data available from ASL is just to initialize quirks and non-ECAM
> >>>controllers, right? Given that SBSA mandates ECAM, and we assume that
> >>>ACPI is mostly (if not only) for servers, then I think it is safe to say
> >>>that in the case of ACPI we should have all the info to fully
> >>>instantiate an host bridge driver.
> >>
> >>From the spec, the MCFG will only describe host bridge available at boot 
> >>(see
> >>4.2 in "PCI firmware specification, rev 3.2"). All the other host bridges 
> >>will
> >>be described in ASL.
> >>
> >>So we need DOM0 to feed Xen about the latter host bridges.
> >
> >Unfortunately PCI specs are only accessible by PCI SIG members
> >organizations. In other words, I cannot read the doc.
> >
> >Could you please explain what kind of host bridges are not expected to
> >be available at boot? Do you know of any examples?
> 
> Roger answered to this answer in on a reply to this e-mail. So I will skip
> it. Let me know if you need for details.
> 
> >
> >
> >>>>whilst for Device Tree the segment number is not available.
> >>>>
> >>>>So Xen needs to rely on DOM0 to discover the host bridges and notify Xen
> >>>>with all the relevant informations. This will be done via a new hypercall
> >>>>PHYSDEVOP_pci_host_bridge_add. The layout of the structure will be:
> >>>
> >>>I understand that the main purpose of this hypercall is to get Xen and Dom0
> >>>to
> >>>agree on the segment numbers, but why is it necessary? If Dom0 has an
> >>>emulated contoller like any other guest, do we care what segment numbers
> >>>Dom0 will use?
> >>
> >>I was not planning to have a emulated controller for DOM0. The physical one 
> >>is
> >>not necessarily ECAM compliant so we would have to either emulate the 
> >>physical
> >>one (meaning multiple different emulation) or an ECAM compliant.
> >>
> >>The latter is not possible because you don't know if there is enough free 
> >>MMIO
> >>space for the emulation.
> >>
> >>In the case on ARM, I don't see much the point to emulate the host bridge 
> >>for
> >>DOM0. The only thing we need in Xen is to access the configuration space, we
> >>don't have about driving the host bridge. So I would let DOM0 dealing with
> >>that.
> >>
> >>Also, I don't see any reason for ARM to trap DOM0 configuration space 
> >>access.
> >>The MSI will be configured using the interrupt controller and it is a 
> >>trusted
> >>Domain.
> >
> >These last you sentences raise a lot of questions. Maybe I am missing
> >something. You might want to clarify the strategy for Dom0 and DomUs,
> >and how they differ, in the next version of the doc.
> >
> >At some point you wrote "Instantiation of a specific driver for the host
> >controller can be easily done if Xen has the information to detect it.
> >However, those drivers may require resources described in ASL." Does it
> >mean you plan to drive the physical host bridge from Xen and Dom0
> >simultaneously?
> 
> I may miss some bits, so feel free to correct me if I am wrong.
> 
> My understanding is host bridge can be divided in 2 parts:
>       - Initialization of the host bridge
>       - Access the configuration space
> 
> For generic host bridge, the initialization is inexistent. However some host
> bridge (e.g xgene, xilinx) may require some specific setup and also
> configuring clocks. Given that Xen only requires to access the configuration
> space, I was thinking to let DOM0 initialization the host bridge. This would
> avoid to import a lot of code in Xen, however this means that we need to
> know when the host bridge has been initialized before accessing the
> configuration space.


Yes, that's correct.
There's a sequence on the ZynqMP that involves assiging Gigabit Transceivers
to PCI (GTs are shared among PCIe, USB, SATA and the Display Port),
enabling clocks and configuring a few registers to enable ECAM and MSI.

I'm not sure if this could be done prior to starting Xen. Perhaps.
If so, bootloaders would have to know a head of time what devices
the GTs are supposed to be configured for.



> 
> Now regarding the configuration space, I think we can divide in 2 category:
>       - indirect access, the configuration space are multiplexed. An example
> would be the legacy method on x86 (e.g 0xcf8 and 0xcfc). A similar method is
> used for x-gene PCI driver ([1]).
>       - ECAM like access, where each PCI configuration space will have it is 
> own
> address space. I said "ECAM like" because some host bridge will require some
> bits fiddling when accessing register (see thunder-ecam [2])
> 
> There are also host bridges that mix both indirect access and ECAM like
> access depending on the device configuration space accessed (see thunder-pem
> [3]).
> 
> When using ECAM like host bridge, I don't think it will be an issue to have
> both DOM0 and Xen accessing configuration space at the same time. Although,
> we need to define who is doing what. In general case, DOM0 should not
> touched an assigned PCI device. The only possible interaction would be
> resetting a device (see my answer below).
> 
> When using indirect access, we cannot let DOM0 and Xen accessing any PCI
> configuration space at the same time. So I think we would have to emulate
> the physical host controller.
> 
> Unless we have a big requirement to trap DOM0 access to the configuration
> space, I would only keep the emulation to the strict minimum (e.g for
> indirect access) to avoid ending-up handling all the quirks for ECAM like
> host bridge.
> 
> If we need to trap the configuration space, I would suggest the following
> for ECAM like host bridge:
>       - For physical host bridge that does not require initialization and is
> nearly ECAM compatible (e.g require register fiddling) => replace by a
> generic host bridge emulation for DOM0
>       - For physical host bridge that require initialization but is ECAM
> compatible (e.g AFAICT xilinx [4]) => trap the ECAM access but let DOM0
> handling the host bridge initialization

Sounds good to me.


>       - For all other host bridges => I don't know if there are host bridges
> falling under this category. I also don't have any idea how to handle this.
> 
> >
> >Otherwise, if Dom0 is the only one to drive the physical host bridge,
> >and Xen is the one to provide the emulated host bridge, how are DomU PCI
> >config reads and writes supposed to work in details?
> 
> I think I have answered to this question with my explanation above. Let me
> know if it is not the case.
> 
> >  How is MSI configuration supposed to work?
> 
> For GICv3 ITS, the MSI will be configured with the eventID (it is uniq
> per-device) and the address of the doorbell. The linkage between the LPI and
> "MSI" will be done through the ITS.
> 
> For GICv2m, the MSI will be configured with an SPIs (or offset on some
> GICv2m) and the address of the doorbell. Note that for DOM0 SPIs are mapped
> 1:1.
> 
> So in both case, I don't think it is necessary to trap MSI configuration for
> DOM0. This may not be true if we want to handle other MSI controller.
> 
> I have in mind the xilinx MSI controller (embedded in the host bridge? [4])
> and xgene MSI controller ([5]). But I have no idea how they work and if we
> need to support them. Maybe Edgar could share details on the Xilinx one?


The Xilinx controller has 2 dedicated SPIs and pages for MSIs. AFAIK, there's no
way to protect the MSI doorbells from mal-configured end-points raising 
malicious EventIDs.
So perhaps trapped config accesses from domUs can help by adding this protection
as drivers configure the device.

On Linux, Once MSI's hit, the kernel takes the SPI interrupts, reads
out the EventID from a FIFO in the controller and injects a new IRQ into
the kernel.

I hope that helps!
Best regards,
Edgar


> 
> >
> >
> >>>>XXX: Shall we limit DOM0 the access to the configuration space from that
> >>>>moment?
> >>>
> >>>If we can, we should
> >>
> >>Why would be the benefits? For now, I see a big drawback: resetting a PCI
> >>devices would need to be done in Xen rather than DOM0. As you may now there
> >>are a lot of quirks for reset.
> >>
> >>So for me, it looks more sensible to handle this in DOM0 and let DOM0 a full
> >>access to the configuration space. Overall he is a trusted domain.
> >
> >PCI reset is worth of its own chapter in the doc :-)
> >
> >Dom0 is a trusted domain, but when possible, I consider an improvement
> >to limit the amount of trust we put in it. Also, as I wrote above, I
> >don't understand what is the plan to deal with concurrent accesses to
> >the host bridge from Dom0 and Xen.
> 
> I believe I gave more details now :). If it sounds sensible, I will add it
> in the next version of the design doc.
> 
> >
> >In any case, regarding PCI reset, we should dig out past discussions on
> >the merits of doing reset in the hypervisor vs. dom0. I agree
> >introducing PCI reset quirks in Xen is not nice but I recall that
> >XenClient did it to avoid possible misbehaviors of the device. We need
> >to be careful about ordering PCI reset against domain destruction. I
> >couldn't find any email discussions to reference, maybe it is worth
> >contacting the OpenXT guys about it.
> 
> I've got a vague recall of this code back when I was working at XenClient.
> 
> I gave a brief look to the Xen patchqueue [6] of openxt and was not able to
> find a patch to reset PCI in Xen. However, the have a patch to fix the one
> in Linux [7].
> 
> There is some ex-XenClient working on OpenXt. I will ask them to see if they
> remember anything. Also CC Andrew, just in case he knows the story.
> 
> Cheers,
> 
> [1] drivers/pci/host/pci-xgene.c
> [2] drivers/pci/host/pci-thunder-ecam.c
> [3] drivers/pci/host/pci-thunder-pem.c
> [4] drivers/pci/host/pcie-xilinx-nwl.c
> [5] drivers/pci/host/pcie-xgene-msi.c
> [6] https://github.com/OpenXT-Extras/xen-common-pq
> [7]
> https://github.com/OpenXT-Extras/linux-3.11-pq/blob/master/master/pci-pt-flr
> 
> -- 
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel