[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

On 02/02/17 15:33, Edgar E. Iglesias wrote:
On Wed, Feb 01, 2017 at 07:04:43PM +0000, Julien Grall wrote:
On 31/01/2017 19:06, Edgar E. Iglesias wrote:
On Tue, Jan 31, 2017 at 05:09:53PM +0000, Julien Grall wrote:
Thank you for the documentation. I am trying to understand if we could move
initialization in Xen as suggested by Stefano. I looked at the driver in
Linux and the code looks simple not many dependencies. However, I was not
able to find where the Gigabit Transceivers are configured. Do you have any
link to the code for that?

Hi Julien,

Hi Edgar,

I suspect that this setup has previously been done by the initial bootloader
auto-generated from design configuration tools.

Now, this is moving into Linux.

Do you know why they decide to move the code in Linux? What would be the problem to let the bootloader configuring the GT?

There's a specific driver that does that but AFAICS, it has not been upstreamed 
You can see it here:

DTS nodes that need a PHY can then just refer to it, here's an example from 
&sata {
        phy-names = "sata-phy";
        phys = <&lane3 PHY_TYPE_SATA 1 3 150000000>;

I'll see if I can find working examples for PCIe on the ZCU102. Then I'll share
DTS, Kernel etc.

I've found a device tree on the github from the ZCU102: zynqmp-zcu102.dts, it looks like there is no use of PHY for the pcie so far.

Lets imagine in the future, pcie will use the PHY. If we decide to initialize the hostbridge in Xen, we would also have to pull the PHY code in the hypervisor. Leaving aside the problem to pull more code in Xen, this is not nice because the PHY is used by different components (e.g SATA, USB). So Xen and DOM0 would have to share the PHY.

For Xen POV, the best solution would be the bootloader initializing the PHY because starting Xen. So we can keep all the hostbridge (initialization + access) in Xen.

If it is not possible, then I would prefer to see the hostbridge initialization in DOM0.

If you are looking for a platform to get started, an option could be if I get 
you a build of
our QEMU that includes models for the PCIe controller, MSI and SMMU connections.
These models are friendly wrt. PHY configs and initialization sequences, it will
accept pretty much any sequence and still work. This would allow you to focus on
architectural issues rather than exact details of init sequences (which we can
deal with later).

From my understanding the problem is where the hostbridge should be initialized. In an ideal world, I think this is the goal of the bootloader. If it is not possible then depending on the complexity, the initialization would have to be done either in Xen or DOM0.

I guess this could be decided on case by case basis. I will suggest different possibility in the design document.


From a design point of view, it would make more sense to have the MSI
controller driver in Xen as the hostbridge emulation for guest will also
live there.

So if we receive MSI in Xen, we need to figure out a way for DOM0 and guest
to receive MSI. The same way would be the best, and I guess non-PV if
possible. I know you are looking to boot unmodified OS in a VM. This would
mean we need to emulate the MSI controller and potentially xilinx PCI
controller. How much are you willing to modify the OS?

Today, we have not yet implemented PCIe drivers for our baremetal SDK. So
things are very open and we could design with pretty much anything in mind.

Yes, we could perhaps include a very small model with most registers dummied.
Implementing the MSI read FIFO would allow us to:

1. Inject the MSI doorbell SPI into guests. The guest will then see the same
   IRQ as on real HW.

2. Guest reads host-controller registers (MSI FIFO) to get the signaled MSI.

The Xilinx PCIe hostbridge is not the only hostbridge having MSI controller embedded. So I would like to see a generic solution if possible. This would avoid to increase the code required for emulation in Xen.

My concern with a FIFO is it will require an upper bound to avoid using to much memory in Xen. What if the FIFO is full? Will you drop MSI?

Regarding the MSI doorbell, I have seen it is configured by the software
using a physical address of a page allocated in the RAM. When the PCI
devices is writing into the doorbell does the access go through the SMMU?

That's a good question. On our QEMU model it does, but I'll have to dig a 
little to see if that is the case on real HW aswell.

Regardless the answer, I think we would need to map the MSI doorbell page in
the guest. Meaning that even if we trap MSI configuration access, a guess
could DMA in the page. So if I am not mistaken, MSI would be insecure in
this case :/.

Or maybe we could avoid mapping the doorbell in the guest and let Xen
receive an SMMU abort. When receiving the SMMU abort, Xen could sanitize the
value and write into the real MSI doorbell. Not sure if it would works

Yeah, this is a problem.
I'm not sure if SMMU aborts would work because I don't think we know the value 
of the data written when we take the abort.
Without the data, I'm not sure how we would distinguish between different MSI's 
from the same device.

You are right, you don't get the data written and therefore it is not possible to distinguish MSIs. I got confused with the data abort trap.

Also, even if the MSI doorbell would be protected by the SMMU, all PCI devices 
are presented with the same AXI Master ID.
BTW, this master-ID SMMU limitation is a showstopper for domU guests isn't it?

That's limitation is only for your current version of the hardware correct?

Or do you have ideas around that? Perhaps some PV way to request mappings for 

Guest memory would have to be direct mapped as we do for DOM0. However, it means the guest should be able to parse the firmware table (DT, ACPI) in order to know where the RAM banks has been positioned.


Julien Grall

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.