Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 16 Dec 2019, Julien Grall wrote:
> On 16/12/2019 18:02, Andrei Cherechesu wrote:
> > Hello,
> 
> Hello,
> 
> > My name is Andrei Cherechesu and I'm a Software Engineer at NXP
> > 
> > Semiconductors in the Automotive department, Linux BSP Team.
> > 
> > I would like to tell you have done a great job so far with Xen.
> 
> Thank you for your interest in Xen on Arm.

Yes, thank you!!


> > Thus, we have ported and integrated Xen ARM in the Linux BSP for our
> > 
> > boards.
> > 
> > Currently, we are trying to passthrough devices in order to statically
> > 
> > assign them to different DomUs at boot time (Dom0less configuration).
> > 
> > Our boards do not have an IOMMU and so our work is only possible
> > 
> > with the help of the patches submitted by Stefano Stabellini that
> > 
> > implemented "xen,force-assign-without-iommu" property in the
> > 
> > partial device trees used for passthrough.
> > 
> > I made a configuration in order to setup the dtb /chosen
> > 
> > node to boot a Dom0less VM in parallel with Dom0, with a
> > 
> > peripherical device statically assigned to it.
> > 
> > However, only if I advertise the passthrough dtb node in u-boot,
> > 
> > when I boot Xen, I am given the following error:
> > 
> > (XEN) *** LOADING DOMAIN 0 ***
> > 
> > (XEN) Loading d0 kernel from boot module @ 00000000be800000
> > 
> > (XEN) Loading ramdisk from boot module @ 00000000b1000000
> > 
> > (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> > 
> > (XEN) BANK[0] 0x000000d0000000-0x000000f0000000 (512MB)
> > 
> > (XEN) Grant table range: 0x00000080200000-0x00000080240000
> > 
> > (XEN) Allocating PPI 16 for event channel interrupt
> > 
> > (XEN) Loading zImage from 00000000be800000 to
> > 00000000d0080000-00000000d0930008
> > 
> > (XEN) Loading dom0 initrd from 00000000b1000000 to
> > 0x00000000d8200000-0x00000000e01fffff
> > 
> > (XEN) Loading dom0 DTB to 0x00000000d8000000-0x00000000d800596a
> > 
> > (XEN) Initial low memory virq threshold set at 0x4000 pages.
> > 
> > (XEN) Scrubbing Free RAM in background
> > 
> > (XEN) Std. Loglevel: All
> > 
> > (XEN) Guest Loglevel: All
> > 
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > 
> > (XEN) domain.c:322: IOMMU is not enabled
> > 
> > (XEN)
> > 
> > (XEN) ****************************************
> > 
> > (XEN) Panic on CPU 0:
> > 
> > (XEN) Error creating domain domU1
> > 
> > (XEN) ****************************************
> > 
> > My Xen is taken from Stefano’s tree
> > (http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git)
> > 
> > that he specified in this
> > <https://lists.xenproject.org/archives/html/xen-devel/2019-10/msg00414.html>
> > thread index, and only has the additional commits
> > 
> > that add support for our boards applied by myself.
> 
> This series as been merged in the upcoming release of Xen. I would therefore
> recommend to use the upstream staging/master in case bugs may have been fixed
> there.
> 
> > 
> >  From what I've read on the mailing lists, the Dom0less Device
> > 
> > Passthrough Without IOMMU feature is supported, at least,
> > 
> > on boards that do have a physical IOMMU but have some devices that
> > 
> > are not behind it.
> > 
> > My question is: In the way that it is currently implemented,
> > 
> > is the Dom0less Device Passthrough feature currently supported
> > 
> > on boards that do not have an IOMMU at all?
> 
> Looking at the code, there are a few issues in current Xen that would prevent
> passthrough without IOMMU to work.
> 
> For a first, Xen is requesting to create instiate an IOMMU context for the
> domain if a partial DT is present (see create_domUs()). The following path
> should get rid of the panic above (/!\ it is a gross hack and not for
> upstream/production):
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index dd9c3b73ba..5700b992df 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -2474,9 +2474,6 @@ void __init create_domUs(void)
>              panic("Missing property 'cpus' for domain %s\n",
>                    dt_node_name(node));
> 
> -        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
> -            d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> -
>          if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
>          {
>              d_cfg.arch.nr_spis = gic_number_lines() - 32;
> 
> But even with this patch, RAM in DomU is not direct mapped (i.e Guest Physical
> Address == Host Physical Address). This means that DMA-capable device would
> not work properly in DomU.
> 
> We could theoritically map DomU direct mapped, but this would break the
> isolation provided by the hypervisor.

Yes, being able to map the DomU memory 1:1 can be pretty useful for some
very embedded dom0less configurations, in fact I was surprised that a
couple of Xilinx users asked me for that recently. Typically, the users
are aware of the consequences but they still find them better than the
alternative (i.e. the lack of isolation is bad but is tolerable in their
configuration.)

>From an implementation perspective, it should be a matter of calling
allocate_memory_11 instead of allocate_memory from construct_domU. I
wanted to experiment with it myself but I haven't had the time. If
nothing else, it would be useful to have a patch around to do it if
needed.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel