Re: Discussion of Xenheap problems on AArch64

[Julien Grall <julien@xxxxxxx>]

On 21/04/2021 10:32, Henry Wang wrote:
> Hi Julien,
Hi Henry,

> > -----Original Message-----
> > From: Julien Grall <julien@xxxxxxx>
> > Sent: Wednesday, April 21, 2021 5:04 PM
> > To: Henry Wang <Henry.Wang@xxxxxxx>; sstabellini@xxxxxxxxxx; xen-
> > devel@xxxxxxxxxxxxxxxxxxxx
> > Cc: Wei Chen <Wei.Chen@xxxxxxx>; Penny Zheng
> > <Penny.Zheng@xxxxxxx>; Bertrand Marquis <Bertrand.Marquis@xxxxxxx>
> > Subject: Re: Discussion of Xenheap problems on AArch64
> >
> >
> >
> > On 21/04/2021 07:28, Henry Wang wrote:
> > > Hi,
> > Hi Henry,
> >
> > > We are trying to implement the static memory allocation on AArch64. Part
> > of
> > > this feature is the reserved heap memory allocation, where a specific range
> > of
> > > memory is reserved only for heap. In the development process, we found a
> > > pitfall in current AArch64 setup_xenheap_mappings() function.
> > >
> > > According to a previous discussion in community
> > > https://lore.kernel.org/xen-devel/20190216134456.10681-1-
> > peng.fan@xxxxxxx/,
> > > on AArch64, bootmem is initialized after setup_xenheap_mappings(),
> > > setup_xenheap_mappings() may try to allocate memory before memory
> > has been
> > > handed over to the boot allocator. If the reserved heap memory allocation
> > is
> > > introduced, either of below 2 cases will trigger a crash:
> > >
> > > 1. If the reserved heap memory is at the end of the memory block list and
> > the
> > > gap between reserved and unreserved memory is bigger than 512GB, when
> > we setup
> > > mappings from the beginning of the memory block list, we will get OOM
> > caused
> > > by lack of pages in boot allocator. This is because the memory that is
> > reserved
> > > for heap has not been mapped and added to the boot allocator.
> > >
> > > 2. If we add the memory that is reserved for heap to boot allocator first,
> > and
> > > then setup mappings for banks in the memory block list, we may get a page
> > which
> > > has not been setup mapping, causing a data abort.
> > There are a few issues with setup_xenheap_mappings(). I have been
> > reworking the code on my spare time and started to upstream bits of it.
> > A PoC can be found here:
> >
> > https://xenbits.xen.org/gitweb/?p=people/julieng/xen-
> > unstable.git;a=shortlog;h=refs/heads/pt/dev
> Really great news! Thanks you very much for the information and your hard
> work on the PoC :) I will start to go through your PoC code then.
I spent sometimes today to clean-up the PoC and sent a series on the ML 
(see [1]). This has been lightly tested so far.
Would you be able to give a try and let me know if it helps your problem?

For convenience, I have pushed a branch with the series applied here:

https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/pt/rfc-v2

Cheers,

[1] https://lore.kernel.org/xen-devel/20210425201318.15447-1-julien@xxxxxxx/

--
Julien Grall