Re: Keystone Issue

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 24 Jun 2020, Bertrand Marquis wrote:
> > On 23 Jun 2020, at 21:50, CodeWiz2280 <codewiz2280@xxxxxxxxx> wrote:
> > 
> > Is it possible to passthrough PCI devices to domU on the 32-bit arm
> > keystone?  Any info is appreciated.
> > 
> > I found some old information online that "gic-v2m" is required.  I'm
> > not sure if the GIC-400 on the K2E supports "gic_v2m".  Based on the
> > fact that there is no "gic-v2m-frame" tag in the K2E device tree i'm
> > guessing that it does not.
> > 
> > If it is possible, is there a good example for arm that I can follow?
> 
> There is no PCI passthrough support on Arm for now in Xen.
> 
> This is something I am working on and I will present something on this 
> subject at the Xen summit.
> But we are not targeting arm32 for now.
> 
> The only thing possible for now is to have PCI devices handled by Dom0 and 
> using xen virtual drivers to pass the functionalities (ethernet, block or 
> others).

It should also possible to pass the entire PCI controller, together with
the whole aperture and all interrupts to a domU. The domU would get all
PCI devices this way, not just one.


 
> > On Wed, Jun 17, 2020 at 7:52 PM CodeWiz2280 <codewiz2280@xxxxxxxxx> wrote:
> >> 
> >> On Wed, Jun 17, 2020 at 2:46 PM Stefano Stabellini
> >> <sstabellini@xxxxxxxxxx> wrote:
> >>> 
> >>> On Wed, 17 Jun 2020, CodeWiz2280 wrote:
> >>>> On Tue, Jun 16, 2020 at 2:23 PM Marc Zyngier <maz@xxxxxxxxxx> wrote:
> >>>>> 
> >>>>> On 2020-06-16 19:13, CodeWiz2280 wrote:
> >>>>>> On Tue, Jun 16, 2020 at 4:11 AM Marc Zyngier <maz@xxxxxxxxxx> wrote:
> >>>>>>> 
> >>>>>>> On 2020-06-15 20:14, CodeWiz2280 wrote:
> >>>>>>> 
> >>>>>>> [...]
> >>>>>>> 
> >>>>>>>> Also, the latest linux kernel still has the X-Gene storm distributor
> >>>>>>>> address as "0x78010000" in the device tree, which is what the Xen 
> >>>>>>>> code
> >>>>>>>> considers a match with the old firmware.  What were the addresses for
> >>>>>>>> the device tree supposed to be changed to?
> >>>>>>> 
> >>>>>>> We usually don't care, as the GIC address is provided by the
> >>>>>>> bootloader,
> >>>>>>> whether via DT or ACPI (this is certainly what happens on Mustang).
> >>>>>>> Whatever is still in the kernel tree is just as dead as the platform
> >>>>>>> it
> >>>>>>> describes.
> >>>>>>> 
> >>>>>>>> Is my understanding
> >>>>>>>> correct that there is a different base address required to access the
> >>>>>>>> "non-secure" region instead of the "secure" 0x78010000 region?  I'm
> >>>>>>>> trying to see if there are corresponding different addresses for the
> >>>>>>>> keystone K2E, but haven't found them yet in the manuals.
> >>>>>>> 
> >>>>>>> There is no such address. Think of the NS bit as an *address space*
> >>>>>>> identifier.
> >>>>>>> 
> >>>>>>> The only reason XGene presents the NS part of the GIC at a different
> >>>>>>> address is because XGene is broken enough not to have EL3, hence no
> >>>>>>> secure mode. To wire the GIC (and other standard ARM IPs) to the core,
> >>>>>>> the designers simply used the CPU NS signal as an address bit.
> >>>>>>> 
> >>>>>>> On your platform, the NS bit does exist. I strongly suppose that it
> >>>>>>> isn't wired to the GIC. Please talk to your SoC vendor for whether iot
> >>>>>>> is possible to work around this.
> >>>>>>> 
> >>>>>> I do have a question about this out to TI, but at least this method
> >>>>>> gives me something to work with in the meantime.  I was just looking
> >>>>>> to confirm that there wouldn't be any other undesirable side effects
> >>>>>> with Dom0 or DomU when using it.  Was there an actual FPGA for the
> >>>>>> X-Gene that needed to be updated which controlled the GIC access?  Or
> >>>>>> by firmware do you mean the boot loader (e.g. uboot).  Thanks for the
> >>>>>> support so far to all.
> >>>>> 
> >>>>> As I said, the specific case of XGene was just a matter of picking the
> >>>>> right address, as the NS bit is used as an address bit on this platform.
> >>>>> This was possible because this machine doesn't have any form of
> >>>>> security. So no HW was changed, no FPGA reprogrammed. Only a firmware
> >>>>> table was fixed to point to the right spot. Not even u-boot or EFI was
> >>>>> changed.
> >>>> Ok, thank you for clarifying.  I have one more question if you don't
> >>>> mind.  I'm aware that dom0 can share physical memory with dom1 via
> >>>> grant tables.
> >>>> However, is it possible to reserve a chunk of contiguous physical
> >>>> memory and directly allocate it only to dom1?
> >>>> For example, if I wanted dom1 to have access to 8MB of contiguous
> >>>> memory at 0x8200_0000 (in addition to whatever virtual memory Xen
> >>>> gives it).
> >>>> How would one go about doing this on ARM?  Is there something in the
> >>>> guest config or device tree that can be set?  Thanks for you help.
> >>> 
> >>> There isn't a "proper" way to do it, only a workaround.
> >>> 
> >>> It is possible to do it by using the iomem option, which is meant for
> >>> device MMIO regions.
> >>> 
> >>> We have patches in the Xilinx Xen tree (not upstream) to allow for
> >>> specifying the cacheability that you want for the iomem mapping so that
> >>> you can map it as normal memory. This is the latest branch:
> >>> 
> >>> https://github.com/Xilinx/xen.git xilinx/release-2020.1
> >>> 
> >>> The relevant commits are the ones from:
> >>> https://github.com/Xilinx/xen/commit/a5c76ac1c5dc14d3e9ccedc5c1e7dd2ddc1397b6
> >>> to:
> >>> https://github.com/Xilinx/xen/commit/b4b7e91c1524f9cf530b81b7ba95df2bf50c78e4
> >>> 
> >>> You might want to make sure that the page is not used by the normal
> >>> memory allocator. This document explains how to something along those
> >>> lines:
> >>> 
> >>> https://github.com/Xilinx/xen/commit/35f72d9130448272e07466bd73cc30406f33135e
> >> 
> >> Thank you.  I appreciate it.
>