RE: Aligning Xen to physical memory maps on embedded systems

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

On Tue, 2 Mar 2021, Levenglick Dov wrote:
> Thank you.
> A few final comments below + one last question regarding the Xilinx forks:
> Xen 4.13 is first available on the 2020.1 branch. Is it required that the 
> 2020.1 branch of linux-xlnx be used as well, or can I keep the 2019.1 branch 
> that I am currently using?

Xilinx recommends to always use the same version everywhere, so 2020.1
for Xen, Linux, firwmare, etc.

That said, it should be no problem to use Xen 2020.1 with everything
else from 2019.1. Given that you are using dom0less, you just need to
rebuild the Xen hypervisor alone, you don't even need to update the dom0
rootfs.


> > -----Original Message-----
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Sent: Tuesday, March 2, 2021 3:42 AM
> > To: Levenglick Dov <Dov.Levenglick@xxxxxxxxxxxxxxxx>
> > Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; Stefano Stabellini
> > <sstabellini@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Xen-
> > users@xxxxxxxxxxxxxxxxxxxx; Wei Chen <Wei.Chen@xxxxxxx>; Penny Zheng
> > <Penny.Zheng@xxxxxxx>; Luca Fancellu <Luca.Fancellu@xxxxxxx>
> > Subject: RE: Aligning Xen to physical memory maps on embedded systems
> > 
> > On Mon, 1 Mar 2021, Levenglick Dov wrote:
> > > > (+ Penny, Wei and Luca)
> > > >
> > > > > On 23 Feb 2021, at 01:52, Stefano Stabellini <sstabellini@xxxxxxxxxx>
> > wrote:
> > > > >
> > > > > On Mon, 22 Feb 2021, Levenglick Dov wrote:
> > > > >>>> The system has 2GB of RAM (0x00000000 - 0x80000000) of which
> > > > >>>> Xen and the DomU have an allocation of 1.25GB, per this memory
> > map:
> > > > >>>> 1. DomU1: 0x60000000 - 0x80000000 2. DomU2: 0x40000000 -
> > > > >>>> 0x60000000 3. Xen: 0x30000000 - 0x40000000
> > > > >>>
> > > > >>> How did you tell Xen which regions is assigned to which guests?
> > > > >>> Are your domain mapped 1:1 (i.e guest physical address == host
> > > > >>> physical
> > > > address)?
> > > > >>
> > > > >> I am working on a solution where if the "xen,domain" memory has
> > > > >> #size-cell cells the content is backward compatible. But if it
> > > > >> contains (#address-cells + #size-cells), the address cells should
> > > > >> be
> > > > considered the physical start address.
> > > > >> During the mapping of the entire address space insetup_mm(), the
> > > > >> carved out addresses would be added to the  reserved memory
> > > > >> address space. When the DomU is to be created, this physical
> > > > >> space would be mapped to it. The virtual addresses are less of an
> > > > >> issue and needn't be
> > > > mapped 1x1 (although they could be).
> > > > >
> > > > > As of today neither upstream Xen nor the Xilinx Xen tree come with
> > > > > the feature of allowing the specification of an address range for
> > > > > dom0less guests.
> > > > >
> > > > > The only thing that Xilinx Xen allows, which is not upstream yet,
> > > > > is the ability of creating dom0less guests 1:1 mapped using the 
> > > > > "direct-
> > map"
> > > > > property. But the memory allocation is still done by Xen (you
> > > > > can't select the addresses).
> > > > >
> > > > > Some time ago I worked on a hacky prototype to allow the
> > > > > specification of address ranges, see:
> > > > >
> > > > > http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unst
> > > > > able .git direct-map-2 from
> > > > > 7372466b21c3b6c96bb7a52754e432bac883a1e3
> > > > onward.
> > > > >
> > > > > In particular, have a look at "xen/arm: introduce 1:1 mapping for
> > > > > domUs". The work is not complete: it might not work depending on
> > > > > the memory ranges you select for your domUs. In particular, you
> > > > > can't select top-of-RAM addresses for your domUs. However, it
> > > > > might help you getting started.
> > > > >
> > > > >
> > > > >>>> I am able to support True Dom0-less by means of the patch/hack
> > > > >>>> demonstrated By Stefano Stabellini at
> > > > >>> https://youtu.be/UfiP9eAV0WA?t=1746.
> > > > >>>>
> > > > >>>> I was able to forcefully put the Xen binary at the address
> > > > >>>> range immediately below 0x40000000 by means of modifying
> > > > get_xen_paddr()
> > > > >>>> -
> > > > >>> in itself an ugly hack.
> > > > >>>>
> > > > >>>> My questions are:
> > > > >>>> 1. Since Xen performs runtime allocations from its heap, it is
> > allocating
> > > > >>>>    downwards from 0x80000000 - thereby "stealing" memory from
> > > > DomU1.
> > > > >>>
> > > > >>> In theory, any memory reserved for domains should have been
> > > > >>> carved out from the heap allocator. This would be sufficient to
> > > > >>> prevent Xen allocating memory from the ranges you described
> > above.
> > > > >>>
> > > > >>> Therefore, to me this looks like a bug in the tree you are using.
> > > > >>
> > > > >> This would be a better approach, but because Xen perform
> > > > >> allocations from its heap prior to allocating memory to DomU -
> > > > >> and since it allocates from the top of the heap - it is basically
> > > > >> taking memory that I
> > > > wanted to set aside for the DomU.
> > > > >
> > > > > Yeah, this is the main problem that my prototype above couldn't solve.
> > >
> > > Stephano: Is the approach that I previously described a feasible one?
> > >   1. Mark the addresses that I want to set aside as reserved
> > >   2. When reaching the proper DomU, map them and then use the mapping
> > > This approach would solve the heap issue
> > 
> > My first suggestion would be actually to let the hypervisor pick the address
> > ranges. If you don't change setup, you'll see that they are actually stable
> > across reboot. WARNING: Xen doesn't promise that they are stable;
> > however, in practice, they are stable unless you change device tree or
> > configuration or software versions.
> > 
> > That said, yes, I think your approach might work with some limitations (e.g.
> > Xen reclaiming memory on domU destruction but you probably don't care
> > about that). It could be a decent stopgap until we get a better solution.
> 
> Is DomU destruction an option on true Dom0-less? Who would be doing the 
> destruction?

Destruction, yes. You should be able to use "xl destroy" in Dom0 already
today to destroy a dom0less domU. Pass a domid instead of domain name
(they don't have a domain name). Of course you need the xl tools in the
Xen rootfs for that, so if you are going to update Xen, then you also
need to update the Xen tools, hence the Dom0 rootfs. The Xen tools and
Xen actually need to be of the same version.

If you intend to create again a dom0less domain after destroying it
(reboot), then you need to have a config file in dom0 with the same
configuration so that you can call xl create.