RE: Proposal for Porting Xen to Armv8-R64 - DraftA

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 3 Mar 2022, Wei Chen wrote:
> > On Thu, 3 Mar 2022, Wei Chen wrote:
> > > > On Wed, 2 Mar 2022, Wei Chen wrote:
> > > > > > > > > > If not, and considering that we have to generate
> > > > > > > > > > ARM_MPU_*_MEMORY_START/END anyway at build time, would it
> > make
> > > > > > sense
> > > > > > > > to
> > > > > > > > > > also generate mpu,guest-memory-section, xen,static-mem,
> > etc.
> > > > at
> > > > > > build
> > > > > > > > > > time rather than passing it via device tree to Xen at
> > runtime?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Did you mean we still add these information in device tree,
> > but
> > > > for
> > > > > > > > build
> > > > > > > > > time only. In runtime we don't parse them?
> > > > > > > >
> > > > > > > > Yes, something like that, but see below.
> > > > > > > >
> > > > > > > >
> > > > > > > > > > What's the value of doing ARM_MPU_*_MEMORY_START/END at
> > build
> > > > time
> > > > > > and
> > > > > > > > > > everything else at runtime?
> > > > > > > > >
> > > > > > > > > ARM_MPU_*_MEMORY_START/END is defined by platform. But other
> > > > things
> > > > > > are
> > > > > > > > > users customized. They can change their usage without
> > rebuild
> > > > the
> > > > > > image.
> > > > > > > >
> > > > > > > > Good point.
> > > > > > > >
> > > > > > > > We don't want to have to rebuild Xen if the user updated a
> > guest
> > > > > > kernel,
> > > > > > > > resulting in a larger boot-module-section.
> > > > > > > >
> > > > > > > > So I think it makes sense that "mpu,boot-module-section" is
> > > > generated
> > > > > > by
> > > > > > > > the scripts (e.g. ImageBuilder) at build time, and Xen reads
> > the
> > > > > > > > property at boot from the runtime device tree.
> > > > > > > >
> > > > > > > > I think we need to divide the information into two groups:
> > > > > > > >
> > > > > > > >
> > > > > > > > # Group1: board info
> > > > > > > >
> > > > > > > > This information is platform specific and it is not meant to
> > > > change
> > > > > > > > depending on the VM configuration. Ideally, we build Xen for a
> > > > > > platform
> > > > > > > > once, then we can use the same Xen binary together with any
> > > > > > combination
> > > > > > > > of dom0/domU kernels and ramdisks.
> > > > > > > >
> > > > > > > > This kind of information doesn't need to be exposed to the
> > runtime
> > > > > > > > device tree. But we can still use a build-time device tree to
> > > > generate
> > > > > > > > the addresses if it is convenient.
> > > > > > > >
> > > > > > > > XEN_START_ADDRESS, ARM_MPU_DEVICE_MEMORY_*, and
> > > > > > ARM_MPU_NORMAL_MEMORY_*
> > > > > > > > seem to be part of this group.
> > > > > > > >
> > > > > > >
> > > > > > > Yes.
> > > > > > >
> > > > > > > >
> > > > > > > > # Group2: boot configuration
> > > > > > > >
> > > > > > > > This information is about the specific set of binaries and VMs
> > > > that we
> > > > > > > > need to boot. It is conceptually similar to the dom0less
> > device
> > > > tree
> > > > > > > > nodes that we already have. If we change one of the VM
> > binaries,
> > > > we
> > > > > > > > likely have to refresh the information here.
> > > > > > > >
> > > > > > > > "mpu,boot-module-section" probably belongs to this group
> > (unless
> > > > we
> > > > > > find
> > > > > > > > a way to define "mpu,boot-module-section" generically so that
> > we
> > > > don't
> > > > > > > > need to change it any time the set of boot modules change.)
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > I agree.
> > > > > > >
> > > > > > > > > > It looks like we are forced to have the sections
> > definitions
> > > > at
> > > > > > build
> > > > > > > > > > time because we need them before we can parse device tree.
> > In
> > > > that
> > > > > > > > case,
> > > > > > > > > > we might as well define all the sections at build time.
> > > > > > > > > >
> > > > > > > > > > But I think it would be even better if Xen could
> > automatically
> > > > > > choose
> > > > > > > > > > xen,static-mem, mpu,guest-memory-section, etc. on its own
> > > > based on
> > > > > > the
> > > > > > > > > > regular device tree information (/memory, /amba, etc.),
> > > > without
> > > > > > any
> > > > > > > > need
> > > > > > > > > > for explicitly describing each range with these new
> > properties.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > for mpu,guest-memory-section, with the limitations: no other
> > > > usage
> > > > > > > > between
> > > > > > > > > different guest' memory nodes, this is OK. But for
> > xen,static-
> > > > mem
> > > > > > (heap),
> > > > > > > > > we just want everything on a MPU system is dertermistic. But,
> > of
> > > > > > course
> > > > > > > > Xen
> > > > > > > > > can select left memory for heap without static-mem.
> > > > > > > >
> > > > > > > > It is good that you think they can be chosen by Xen.
> > > > > > > >
> > > > > > > > Differently from "boot-module-section", which has to do with
> > the
> > > > boot
> > > > > > > > modules selected by the user for a specific execution,
> > > > > > > > guest-memory-section and static-mem are Xen specific memory
> > > > > > > > policies/allocations.
> > > > > > > >
> > > > > > > > A user wouldn't know how to fill them in. And I worry that
> > even a
> > > > > > script
> > > > > > >
> > > > > > > But users should know it, because static-mem for guest must be
> > > > allocated
> > > > > > > in this range. And users take the responsibility to set the
> > DomU's
> > > > > > > static allocate memory ranges.
> > > > > >
> > > > > > Let me premise that my goal is to avoid having many users
> > reporting
> > > > > > errors to xen-devel and xen-users when actually it is just a wrong
> > > > > > choice of addresses.
> > > > > >
> > > > > > I think we need to make a distinction between addresses for the
> > boot
> > > > > > modules, e.g. addresses where to load xen, the dom0/U kernel,
> > dom0/U
> > > > > > ramdisk in memory at boot time, and VM static memory addresses.
> > > > > >
> > > > > > The boot modules addresses are particularly difficult to fill in
> > > > because
> > > > > > they are many and a small update in one of the modules could
> > > > invalidate
> > > > > > all the other addresses. This is why I ended up writing
> > ImageBuilder.
> > > > > > Since them, I received several emails from users thanking me for
> > > > > > ImageBuilder :-)
> > > > > >
> > > > >
> > > > > Thanks +999 😊
> > > > >
> > > > >
> > > > > > The static VM memory addresses (xen,static-mem) should be a bit
> > easier
> > > > > > to fill in correctly. They are meant to be chosen once, and it
> > > > shouldn't
> > > > > > happen that an update on a kernel forces the user to change all
> > the VM
> > > > > > static memory addresses. Also, I know that some users actually
> > want to
> > > > > > be able to choose the domU addresses by hand because they have
> > > > specific
> > > > > > needs. So it is good that we can let the user choose the addresses
> > if
> > > > > > they want to.
> > > > > >
> > > > >
> > > > > Yes.
> > > > >
> > > > > > With all of that said, I do think that many users won't have an
> > > > opinion
> > > > > > on the VM static memory addresses and won't know how to choose
> > them.
> > > > > > It would be error prone to let them try to fill them in by hand.
> > So I
> > > > > > was already planning on adding support to ImageBuilder to
> > > > automatically
> > > > > > generate xen,static-mem for dom0less domains.
> > > > > >
> > > > >
> > > > > Let me make sure that's what you said: Users give an VM memory size
> > to
> > > > > ImageBuilder, and ImageBuilder will generate xen,static-mem = <start,
> > > > size>.
> > > > > For specific VM, ImageBuilder also can accept start and size as
> > inputs?
> > > > >
> > > > > Do I understand this correctly?
> > > >
> > > > Yes, exactly
> > > >
> > > >
> > > > > > Going back to this specific discussion about boot-module-section:
> > I
> > > > can
> > > > > > see now that, given xen,static-mem is chosen by ImageBuilder (or
> > > > >
> > > > > By hand : )
> > > > >
> > > > > > similar) and not Xen, then it makes sense to have ImageBuilder (or
> > > > > > similar) also generate boot-module-section.
> > > > > >
> > > > >
> > > > > If my above understanding is right, then yes.
> > > >
> > > > Yes, I think we are on the same page
> > > >
> > > >
> > > > > > > > like ImageBuilder wouldn't be the best place to pick these
> > values
> > > > --
> > > > > > > > they seem too "important" to leave to a script.
> > > > > > > >
> > > > > > > > But it seems possible to choose the values in Xen:
> > > > > > > > - Xen knows ARM_MPU_NORMAL_MEMORY_* because it was defined at
> > > > build
> > > > > > time
> > > > > > > > - Xen reads boot-module-section from device tree
> > > > > > > >
> > > > > > > > It should be possible at this point for Xen to pick the best
> > > > values
> > > > > > for
> > > > > > > > guest-memory-section and static-mem based on the memory
> > available.
> > > > > > > >
> > > > > > >
> > > > > > > How Xen to pick? Does it mean in static allocation DomU DT node,
> > we
> > > > just
> > > > > > > need a size, but don't require a start address for static-mem?
> > > > > >
> > > > > > Yes the idea was that the user would only provide the size (e.g.
> > > > > > DOMU_STATIC_MEM[1]=1024) and the addresses would be automatically
> > > > > > calculated. But I didn't mean to change the existing xen,static-
> > mem
> > > > > > device tree bindings. So it is best if the xen,static-mem
> > addresses
> > > > > > generation is done by ImageBuilder (or similar tool) instead of
> > Xen.
> > > > > >
> > > > >
> > > > > If we still keep the option for user to specify the start and size
> > > > > parameters for VM memory, because it maybe very important for a
> > > > > deterministic system (fully static system), I agree with you.
> > > > >
> > > > > And in current static-allocation, I think Xen doesn't generate
> > > > > xen,static-mem addresses, all by hands...
> > > >
> > > > Yeah
> > > >
> > >
> > > I will update my proposal to cover our above discussion, but I forgot
> > one
> > > thing. As the platform header files will be generated from DTS, does it
> > > mean we have to maintain platform dts files in Xen like what Zephyr has
> > > done?
> > 
> > I would prefer not to have to maintain platform dts files in Xen like
> > Zephyr is doing. Ideally, the user should be able to take any
> > spec-compliant device tree file and use it. I would say: let's start
> > without adding the dts files to Xen (we might have one under docs/ but
> > just as an example.) We can add them later if the need arise.
> > 
> 
> But without any default dts, that means we can't start to build Xen for
> v8R? It seems in this case, we need Makefile to print some message to
> tell users to specific his/her dts path.
 
Yes, exactly.



> > > And do you have some idea to integrate the "ImageBuilder"? Make it
> > > as a submodule of Xen or integrate to xen-tools?
> > 
> > I think it would be best if ImageBuilder was kept as a separate
> > repository because there should be no strong ties between ImageBuilder
> > versions and Xen versions. It is more convenient to handle it in a
> > separate repository, especially as Yocto and other build systems might
> > clone ImageBuilder during the build to generate boot.scr (it is already
> > the case).
> > 
> > That said, it might be good to make it more "official" but moving it to
> > Xen Project. I can talk to George about creating
> > http://xenbits.xen.org/git-http/imagebuilder.git or
> > https://gitlab.com/xen-project/imagebuilder.
> 
> That's good : )