Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Julien Grall <julien.grall@xxxxxxxxxx>]

On 02/02/17 15:33, Edgar E. Iglesias wrote:
> On Wed, Feb 01, 2017 at 07:04:43PM +0000, Julien Grall wrote:
> > On 31/01/2017 19:06, Edgar E. Iglesias wrote:
> > > On Tue, Jan 31, 2017 at 05:09:53PM +0000, Julien Grall wrote:
> > Thank you for the documentation. I am trying to understand if we could move
> > initialization in Xen as suggested by Stefano. I looked at the driver in
> > Linux and the code looks simple not many dependencies. However, I was not
> > able to find where the Gigabit Transceivers are configured. Do you have any
> > link to the code for that?
> Hi Julien,
Hi Edgar,

> I suspect that this setup has previously been done by the initial bootloader
> auto-generated from design configuration tools.
>
> Now, this is moving into Linux.
Do you know why they decide to move the code in Linux? What would be the 
problem to let the bootloader configuring the GT?
> There's a specific driver that does that but AFAICS, it has not been upstreamed 
> yet.
> You can see it here:
> https://github.com/Xilinx/linux-xlnx/blob/master/drivers/phy/phy-zynqmp.c
>
> DTS nodes that need a PHY can then just refer to it, here's an example from 
> SATA:
> &sata {
>         phy-names = "sata-phy";
>         phys = <&lane3 PHY_TYPE_SATA 1 3 150000000>;
> };
>
> I'll see if I can find working examples for PCIe on the ZCU102. Then I'll share
> DTS, Kernel etc.
I've found a device tree on the github from the ZCU102: 
zynqmp-zcu102.dts, it looks like there is no use of PHY for the pcie so 
far.
Lets imagine in the future, pcie will use the PHY. If we decide to 
initialize the hostbridge in Xen, we would also have to pull the PHY 
code in the hypervisor. Leaving aside the problem to pull more code in 
Xen, this is not nice because the PHY is used by different components 
(e.g SATA, USB). So Xen and DOM0 would have to share the PHY.
For Xen POV, the best solution would be the bootloader initializing the 
PHY because starting Xen. So we can keep all the hostbridge 
(initialization + access) in Xen.
If it is not possible, then I would prefer to see the hostbridge 
initialization in DOM0.
> If you are looking for a platform to get started, an option could be if I get 
> you a build of
> our QEMU that includes models for the PCIe controller, MSI and SMMU connections.
> These models are friendly wrt. PHY configs and initialization sequences, it will
> accept pretty much any sequence and still work. This would allow you to focus on
> architectural issues rather than exact details of init sequences (which we can
> deal with later).
From my understanding the problem is where the hostbridge should be 
initialized. In an ideal world, I think this is the goal of the 
bootloader. If it is not possible then depending on the complexity, the 
initialization would have to be done either in Xen or DOM0.
I guess this could be decided on case by case basis. I will suggest 
different possibility in the design document.
[...]

> > From a design point of view, it would make more sense to have the MSI
> > controller driver in Xen as the hostbridge emulation for guest will also
> > live there.
> >
> > So if we receive MSI in Xen, we need to figure out a way for DOM0 and guest
> > to receive MSI. The same way would be the best, and I guess non-PV if
> > possible. I know you are looking to boot unmodified OS in a VM. This would
> > mean we need to emulate the MSI controller and potentially xilinx PCI
> > controller. How much are you willing to modify the OS?
> Today, we have not yet implemented PCIe drivers for our baremetal SDK. So
> things are very open and we could design with pretty much anything in mind.
>
> Yes, we could perhaps include a very small model with most registers dummied.
> Implementing the MSI read FIFO would allow us to:
>
> 1. Inject the MSI doorbell SPI into guests. The guest will then see the same
>    IRQ as on real HW.
>
> 2. Guest reads host-controller registers (MSI FIFO) to get the signaled MSI.
The Xilinx PCIe hostbridge is not the only hostbridge having MSI 
controller embedded. So I would like to see a generic solution if 
possible. This would avoid to increase the code required for emulation 
in Xen.
My concern with a FIFO is it will require an upper bound to avoid using 
to much memory in Xen. What if the FIFO is full? Will you drop MSI?
> > Regarding the MSI doorbell, I have seen it is configured by the software
> > using a physical address of a page allocated in the RAM. When the PCI
> > devices is writing into the doorbell does the access go through the SMMU?
> That's a good question. On our QEMU model it does, but I'll have to dig a 
> little to see if that is the case on real HW aswell.
>
> > Regardless the answer, I think we would need to map the MSI doorbell page in
> > the guest. Meaning that even if we trap MSI configuration access, a guess
> > could DMA in the page. So if I am not mistaken, MSI would be insecure in
> > this case :/.
> >
> > Or maybe we could avoid mapping the doorbell in the guest and let Xen
> > receive an SMMU abort. When receiving the SMMU abort, Xen could sanitize the
> > value and write into the real MSI doorbell. Not sure if it would works
> > thought.
> Yeah, this is a problem.
> I'm not sure if SMMU aborts would work because I don't think we know the value 
> of the data written when we take the abort.
> Without the data, I'm not sure how we would distinguish between different MSI's 
> from the same device.
You are right, you don't get the data written and therefore it is not 
possible to distinguish MSIs. I got confused with the data abort trap.
> Also, even if the MSI doorbell would be protected by the SMMU, all PCI devices 
> are presented with the same AXI Master ID.
> BTW, this master-ID SMMU limitation is a showstopper for domU guests isn't it?
That's limitation is only for your current version of the hardware correct?

> Or do you have ideas around that? Perhaps some PV way to request mappings for 
> DMA?
Guest memory would have to be direct mapped as we do for DOM0. However, 
it means the guest should be able to parse the firmware table (DT, ACPI) 
in order to know where the RAM banks has been positioned.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel