RE: Proposal for Porting Xen to Armv8-R64 - DraftA

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Stefano,

> -----Original Message-----
> From: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> Sent: 2022年3月3日 10:04
> To: Wei Chen <Wei.Chen@xxxxxxx>
> Cc: Stefano Stabellini <sstabellini@xxxxxxxxxx>; xen-
> devel@xxxxxxxxxxxxxxxxxxxx; julien@xxxxxxx; Bertrand Marquis
> <Bertrand.Marquis@xxxxxxx>; Penny Zheng <Penny.Zheng@xxxxxxx>; Henry Wang
> <Henry.Wang@xxxxxxx>; nd <nd@xxxxxxx>; George.Dunlap@xxxxxxxxxx
> Subject: RE: Proposal for Porting Xen to Armv8-R64 - DraftA
> 
> On Thu, 3 Mar 2022, Wei Chen wrote:
> > > On Wed, 2 Mar 2022, Wei Chen wrote:
> > > > > > > > > If not, and considering that we have to generate
> > > > > > > > > ARM_MPU_*_MEMORY_START/END anyway at build time, would it
> make
> > > > > sense
> > > > > > > to
> > > > > > > > > also generate mpu,guest-memory-section, xen,static-mem,
> etc.
> > > at
> > > > > build
> > > > > > > > > time rather than passing it via device tree to Xen at
> runtime?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Did you mean we still add these information in device tree,
> but
> > > for
> > > > > > > build
> > > > > > > > time only. In runtime we don't parse them?
> > > > > > >
> > > > > > > Yes, something like that, but see below.
> > > > > > >
> > > > > > >
> > > > > > > > > What's the value of doing ARM_MPU_*_MEMORY_START/END at
> build
> > > time
> > > > > and
> > > > > > > > > everything else at runtime?
> > > > > > > >
> > > > > > > > ARM_MPU_*_MEMORY_START/END is defined by platform. But other
> > > things
> > > > > are
> > > > > > > > users customized. They can change their usage without
> rebuild
> > > the
> > > > > image.
> > > > > > >
> > > > > > > Good point.
> > > > > > >
> > > > > > > We don't want to have to rebuild Xen if the user updated a
> guest
> > > > > kernel,
> > > > > > > resulting in a larger boot-module-section.
> > > > > > >
> > > > > > > So I think it makes sense that "mpu,boot-module-section" is
> > > generated
> > > > > by
> > > > > > > the scripts (e.g. ImageBuilder) at build time, and Xen reads
> the
> > > > > > > property at boot from the runtime device tree.
> > > > > > >
> > > > > > > I think we need to divide the information into two groups:
> > > > > > >
> > > > > > >
> > > > > > > # Group1: board info
> > > > > > >
> > > > > > > This information is platform specific and it is not meant to
> > > change
> > > > > > > depending on the VM configuration. Ideally, we build Xen for a
> > > > > platform
> > > > > > > once, then we can use the same Xen binary together with any
> > > > > combination
> > > > > > > of dom0/domU kernels and ramdisks.
> > > > > > >
> > > > > > > This kind of information doesn't need to be exposed to the
> runtime
> > > > > > > device tree. But we can still use a build-time device tree to
> > > generate
> > > > > > > the addresses if it is convenient.
> > > > > > >
> > > > > > > XEN_START_ADDRESS, ARM_MPU_DEVICE_MEMORY_*, and
> > > > > ARM_MPU_NORMAL_MEMORY_*
> > > > > > > seem to be part of this group.
> > > > > > >
> > > > > >
> > > > > > Yes.
> > > > > >
> > > > > > >
> > > > > > > # Group2: boot configuration
> > > > > > >
> > > > > > > This information is about the specific set of binaries and VMs
> > > that we
> > > > > > > need to boot. It is conceptually similar to the dom0less
> device
> > > tree
> > > > > > > nodes that we already have. If we change one of the VM
> binaries,
> > > we
> > > > > > > likely have to refresh the information here.
> > > > > > >
> > > > > > > "mpu,boot-module-section" probably belongs to this group
> (unless
> > > we
> > > > > find
> > > > > > > a way to define "mpu,boot-module-section" generically so that
> we
> > > don't
> > > > > > > need to change it any time the set of boot modules change.)
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > I agree.
> > > > > >
> > > > > > > > > It looks like we are forced to have the sections
> definitions
> > > at
> > > > > build
> > > > > > > > > time because we need them before we can parse device tree.
> In
> > > that
> > > > > > > case,
> > > > > > > > > we might as well define all the sections at build time.
> > > > > > > > >
> > > > > > > > > But I think it would be even better if Xen could
> automatically
> > > > > choose
> > > > > > > > > xen,static-mem, mpu,guest-memory-section, etc. on its own
> > > based on
> > > > > the
> > > > > > > > > regular device tree information (/memory, /amba, etc.),
> > > without
> > > > > any
> > > > > > > need
> > > > > > > > > for explicitly describing each range with these new
> properties.
> > > > > > > > >
> > > > > > > >
> > > > > > > > for mpu,guest-memory-section, with the limitations: no other
> > > usage
> > > > > > > between
> > > > > > > > different guest' memory nodes, this is OK. But for
> xen,static-
> > > mem
> > > > > (heap),
> > > > > > > > we just want everything on a MPU system is dertermistic. But,
> of
> > > > > course
> > > > > > > Xen
> > > > > > > > can select left memory for heap without static-mem.
> > > > > > >
> > > > > > > It is good that you think they can be chosen by Xen.
> > > > > > >
> > > > > > > Differently from "boot-module-section", which has to do with
> the
> > > boot
> > > > > > > modules selected by the user for a specific execution,
> > > > > > > guest-memory-section and static-mem are Xen specific memory
> > > > > > > policies/allocations.
> > > > > > >
> > > > > > > A user wouldn't know how to fill them in. And I worry that
> even a
> > > > > script
> > > > > >
> > > > > > But users should know it, because static-mem for guest must be
> > > allocated
> > > > > > in this range. And users take the responsibility to set the
> DomU's
> > > > > > static allocate memory ranges.
> > > > >
> > > > > Let me premise that my goal is to avoid having many users
> reporting
> > > > > errors to xen-devel and xen-users when actually it is just a wrong
> > > > > choice of addresses.
> > > > >
> > > > > I think we need to make a distinction between addresses for the
> boot
> > > > > modules, e.g. addresses where to load xen, the dom0/U kernel,
> dom0/U
> > > > > ramdisk in memory at boot time, and VM static memory addresses.
> > > > >
> > > > > The boot modules addresses are particularly difficult to fill in
> > > because
> > > > > they are many and a small update in one of the modules could
> > > invalidate
> > > > > all the other addresses. This is why I ended up writing
> ImageBuilder.
> > > > > Since them, I received several emails from users thanking me for
> > > > > ImageBuilder :-)
> > > > >
> > > >
> > > > Thanks +999 😊
> > > >
> > > >
> > > > > The static VM memory addresses (xen,static-mem) should be a bit
> easier
> > > > > to fill in correctly. They are meant to be chosen once, and it
> > > shouldn't
> > > > > happen that an update on a kernel forces the user to change all
> the VM
> > > > > static memory addresses. Also, I know that some users actually
> want to
> > > > > be able to choose the domU addresses by hand because they have
> > > specific
> > > > > needs. So it is good that we can let the user choose the addresses
> if
> > > > > they want to.
> > > > >
> > > >
> > > > Yes.
> > > >
> > > > > With all of that said, I do think that many users won't have an
> > > opinion
> > > > > on the VM static memory addresses and won't know how to choose
> them.
> > > > > It would be error prone to let them try to fill them in by hand.
> So I
> > > > > was already planning on adding support to ImageBuilder to
> > > automatically
> > > > > generate xen,static-mem for dom0less domains.
> > > > >
> > > >
> > > > Let me make sure that's what you said: Users give an VM memory size
> to
> > > > ImageBuilder, and ImageBuilder will generate xen,static-mem = <start,
> > > size>.
> > > > For specific VM, ImageBuilder also can accept start and size as
> inputs?
> > > >
> > > > Do I understand this correctly?
> > >
> > > Yes, exactly
> > >
> > >
> > > > > Going back to this specific discussion about boot-module-section:
> I
> > > can
> > > > > see now that, given xen,static-mem is chosen by ImageBuilder (or
> > > >
> > > > By hand : )
> > > >
> > > > > similar) and not Xen, then it makes sense to have ImageBuilder (or
> > > > > similar) also generate boot-module-section.
> > > > >
> > > >
> > > > If my above understanding is right, then yes.
> > >
> > > Yes, I think we are on the same page
> > >
> > >
> > > > > > > like ImageBuilder wouldn't be the best place to pick these
> values
> > > --
> > > > > > > they seem too "important" to leave to a script.
> > > > > > >
> > > > > > > But it seems possible to choose the values in Xen:
> > > > > > > - Xen knows ARM_MPU_NORMAL_MEMORY_* because it was defined at
> > > build
> > > > > time
> > > > > > > - Xen reads boot-module-section from device tree
> > > > > > >
> > > > > > > It should be possible at this point for Xen to pick the best
> > > values
> > > > > for
> > > > > > > guest-memory-section and static-mem based on the memory
> available.
> > > > > > >
> > > > > >
> > > > > > How Xen to pick? Does it mean in static allocation DomU DT node,
> we
> > > just
> > > > > > need a size, but don't require a start address for static-mem?
> > > > >
> > > > > Yes the idea was that the user would only provide the size (e.g.
> > > > > DOMU_STATIC_MEM[1]=1024) and the addresses would be automatically
> > > > > calculated. But I didn't mean to change the existing xen,static-
> mem
> > > > > device tree bindings. So it is best if the xen,static-mem
> addresses
> > > > > generation is done by ImageBuilder (or similar tool) instead of
> Xen.
> > > > >
> > > >
> > > > If we still keep the option for user to specify the start and size
> > > > parameters for VM memory, because it maybe very important for a
> > > > deterministic system (fully static system), I agree with you.
> > > >
> > > > And in current static-allocation, I think Xen doesn't generate
> > > > xen,static-mem addresses, all by hands...
> > >
> > > Yeah
> > >
> >
> > I will update my proposal to cover our above discussion, but I forgot
> one
> > thing. As the platform header files will be generated from DTS, does it
> > mean we have to maintain platform dts files in Xen like what Zephyr has
> > done?
> 
> I would prefer not to have to maintain platform dts files in Xen like
> Zephyr is doing. Ideally, the user should be able to take any
> spec-compliant device tree file and use it. I would say: let's start
> without adding the dts files to Xen (we might have one under docs/ but
> just as an example.) We can add them later if the need arise.
> 

But without any default dts, that means we can't start to build Xen for
v8R? It seems in this case, we need Makefile to print some message to
tell users to specific his/her dts path.

> 
> > And do you have some idea to integrate the "ImageBuilder"? Make it
> > as a submodule of Xen or integrate to xen-tools?
> 
> I think it would be best if ImageBuilder was kept as a separate
> repository because there should be no strong ties between ImageBuilder
> versions and Xen versions. It is more convenient to handle it in a
> separate repository, especially as Yocto and other build systems might
> clone ImageBuilder during the build to generate boot.scr (it is already
> the case).
> 
> That said, it might be good to make it more "official" but moving it to
> Xen Project. I can talk to George about creating
> http://xenbits.xen.org/git-http/imagebuilder.git or
> https://gitlab.com/xen-project/imagebuilder.

That's good : )