RE: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Penny Zheng <Penny.Zheng@xxxxxxx>]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xxxxxxx>
> Sent: Tuesday, May 18, 2021 7:48 PM
> To: Penny Zheng <Penny.Zheng@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx;
> sstabellini@xxxxxxxxxx
> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; Wei Chen
> <Wei.Chen@xxxxxxx>; nd <nd@xxxxxxx>
> Subject: Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
> 
> Hi Penny,
> 
> On 18/05/2021 06:07, Penny Zheng wrote:
> > Create one design doc for 1:1 direct-map and static allocation.
> > It is the first draft and aims to describe why and how we allocate
> > 1:1 direct-map(guest physical == physical) domains, and why and how we
> > let domains on static allocation.
> >
> > Signed-off-by: Penny Zheng <penny.zheng@xxxxxxx>
> > ---
> >   docs/designs/static_alloc_and_direct_map.md | 239
> ++++++++++++++++++++
> >   1 file changed, 239 insertions(+)
> >   create mode 100644 docs/designs/static_alloc_and_direct_map.md
> >
> > diff --git a/docs/designs/static_alloc_and_direct_map.md
> > b/docs/designs/static_alloc_and_direct_map.md
> > new file mode 100644
> > index 0000000000..fdda162188
> > --- /dev/null
> > +++ b/docs/designs/static_alloc_and_direct_map.md
> > @@ -0,0 +1,239 @@
> > +# Preface
> > +
> > +The document is an early draft for 1:1 direct-map memory map (`guest
> > +physical == physical`) of domUs and Static Allocation.
> > +Since the implementation of these two features shares a lot, we would
> > +like to introduce both in one design.
> > +
> > +Right now, these two features are limited to ARM architecture.
> > +
> > +This design aims to describe why and how the guest would be created
> > +as 1:1 direct-map domain, and why and what the static allocation is.
> > +
> > +This document is partly based on Stefano Stabellini's patch serie v1:
> > +[direct-map DomUs](
> > +https://lists.xenproject.org/archives/html/xen-devel/2020-
> 04/msg00707.html).
> 
> While for the reviewer this is a useful information to have, I am not sure a
> future reader needs to know all the history. So I would move this to the
> commit message.
> 
> > +
> > +This is a first draft and some questions are still unanswered. When
> > +this is the case, it will be included under chapter `DISCUSSION`.
> > +
> > +# Introduction on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +## Background
> > +
> > +Cases where needs static allocation:
> > +
> > +  * Static allocation needed whenever a system has a pre-defined
> > +non-changing behaviour. This is usually the case in safety world
> > +where system must behave the same upon reboot, so memory resource
> for
> > +both XEN and domains should be static and pre-defined.
> > +
> > +  * Static allocation needed whenever a guest wants to allocate
> > +memory from refined memory ranges. For example, a system has one
> > +high-speed RAM region, and would like to assign it to one specific domain.
> > +
> > +  * Static allocation needed whenever a system needs a guest
> > +restricted to some known memory area due to hardware limitations
> > +reason. For example, some device can only do DMA to a specific part of
> the memory.
> > +
> > +Limitations:
> > +  * There is no consideration for PV devices at the moment.
> > +
> > +## Design on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +These pre-defined memory, -- Static Momery, as parts of RAM reserved
> > +in the
> 
> s/Momery/Memory/
> 

Thx.

> > +beginning, shall never go to heap allocator or boot allocator for any use.
> 
> I think you mean "buddy" rather than "heap". Looking at your code, you are
> treating static memory region as domheap pages.
> 
> > +
> > +### Static Allocation for Domains
> > +
> > +### New Deivce Tree Node: `xen,static_mem`
> 
> S/Deivce/
> 

Thx.

> > +
> > +Here introduces new `xen,static_mem` node to define static memory
> > +nodes for one specific domain.
> > +
> > +For domains on static allocation, users need to pre-define guest RAM
> > +regions in configuration, through `xen,static_mem` node under approriate
> `domUx` node.
> > +
> > +Here is one example:
> > +
> > +
> > +        domU1 {
> > +            compatible = "xen,domain";
> > +            #address-cells = <0x2>;
> > +            #size-cells = <0x2>;
> > +            cpus = <2>;
> > +            xen,static-mem = <0x0 0xa0000000 0x0 0x20000000>;
> > +            ...
> > +        };
> > +
> > +RAM at 0xa0000000 of 512 MB are static memory reserved for domU1 as
> its RAM.
> > +
> > +### New Page Flag: `PGC_reserved`
> > +
> > +In order to differentiate and manage pages reserved as static memory
> > +with those which are allocated from heap allocator for normal
> > +domains, we shall introduce a new page flag `PGC_reserved` to tell.
> > +
> > +Grant pages `PGC_reserved` when initializing static memory.
> > +
> > +### New linked page list: `reserved_page_list` in  `struct domain`
> > +
> > +Right now, for normal domains, on assigning pages to domain, pages
> > +allocated from heap allocator as guest RAM shall be inserted to one
> > +linked page list `page_list` for later managing and storing.
> > +
> > +So in order to tell, pages allocated from static memory, shall be
> > +inserted to a different linked page list `reserved_page_list`.
> 
> You already have the flag ``PGC_reserved`` to indicate whether the memory
> is reserved or not. So why do you also need to link list it?
> 

Yes, I introduce this link list to try to fix issues of rebooting domain on 
static allocation.

And after taking your suggestion on getting static memory regions info again 
from
device tree configuration, I think this link list is not necessary. And I will 
delete here
and in codes too.

> > +
> > +Later, when domain get destroyed and memory relinquished, only pages
> > +in `page_list` go back to heap, and pages in `reserved_page_list` shall 
> > not.
> 
> While going through the series, I could not find any code implementing this.
> However, this is not enough to prevent a page to go to the heap allocator
> because a domain can release memory at runtime using hypercalls like
> XENMEM_remove_from_physmap.
> 
> One of the use case is when the guest decides to balloon out some memory.
> This will call free_domheap_pages().
> 
> Effectively, you are treating static memory as domheap pages. So I think it
> would be better if you hook in free_domheap_pages() to decide which
> allocator is used.
> 
> Now, if a guest can balloon out memory, it can also balloon in memory.
> There are two cases:
>     1) The region used to be RAM region statically allocated
>     2) The region used to be unallocated.
> 
> I think for 1), we need to be able to re-use the page previously. For 2), it 
> is
> not clear to me whether a guest with memory statically allocated should be
> allowed to allocate "dynamic" pages.
> 

Yeah, I share the same with you of hooking in free_domheap_pages(). I'm thinking
that if pages of PGC_reserved, we may create a new func free_staticmem_pages to
free them.

For issues on ballooning out or in, it is not supported here.
Domain on Static Allocation and 1:1 direct-map are all based on dom0-less right
now, so no PV, grant table, event channel, etc, considered.

Right now, it only supports device got passthrough into the guest.

> > +### Memory Allocation for Domains on Static Allocation
> > +
> > +RAM regions pre-defined as static memory for one specifc domain shall
> > +be parsed and reserved from the beginning. And they shall never go to
> > +any memory allocator for any use.
> 
> Technically, you are introducing a new allocator. So do you mean they should
> not be given to neither the buddy allocator nor the bot allocator?
> 

Yes. These pre-defined RAM regions will not be given to any current
memory allocator. If be given there, there is no guarantee that it will
not be allocated for other use.

And right now, in my current design, these pre-defined RAM regions are either 
for
one specific domain as guest RAM or as XEN heap.
  
> > +
> > +Later when allocating static memory for this specific domain, after
> > +acquiring those reserved regions, users need to a do set of
> > +verification before assigning.
> > +For each page there, it at least includes the following steps:
> > +1. Check if it is in free state and has zero reference count.
> > +2. Check if the page is reserved(`PGC_reserved`).
> > +
> > +Then, assigning these pages to this specific domain, and all pages go
> > +to one new linked page list `reserved_page_list`.
> > +
> > +At last, set up guest P2M mapping. By default, it shall be mapped to
> > +the fixed guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`,
> > +just like normal domains. But later in 1:1 direct-map design, if
> > +`direct-map` is set, the guest physical address will equal to physical
> address.
> > +
> > +### Static Allocation for Xen itself
> > +
> > +### New Deivce Tree Node: `xen,reserved_heap`
> 
> s/Deivce/Device/
> 

Thx.

> > +
> > +Static memory for Xen heap refers to parts of RAM reserved in the
> > +beginning for Xen heap only. The memory is pre-defined through XEN
> > +configuration using physical address ranges.
> > +
> > +The reserved memory for Xen heap is an optional feature and can be
> > +enabled by adding a device tree property in the `chosen` node.
> > +Currently, this feature is only supported on AArch64.
> > +
> > +Here is one example:
> > +
> > +
> > +        chosen {
> > +            xen,reserved-heap = <0x0 0x30000000 0x0 0x40000000>;
> > +            ...
> > +        };
> > +
> > +RAM at 0x30000000 of 1G size will be reserved as heap memory. Later,
> > +heap allocator will allocate memory only from this specific region.
> 
> This section is quite confusing. I think we need to clearly differentiate 
> heap vs
> allocator.
> 
> In Xen we have two heaps:
>     1) Xen heap: It is always mapped in Xen virtual address space. This is
> mainly used for xen internal allocation.
>     2) Domain heap: It may not always be mapped in Xen virtual address space.
> This is mainly used for domain memory and mapped on-demand.
> 
> For Arm64 (and x86), two heaps are allocated from the same region. But on
> Arm32, they are different.
> 
> We also have two allocator:
>     1) Boot allocator: This is used during boot only. There is no concept of
> heap at this time.
>     2) Buddy allocator: This is the current runtime allocator. This can either
> allocator from either heap.
> 
> AFAICT, this design is introducing a 3rd allocator that will return domain 
> heap
> pages.
> 
> Now, back to this section, are you saying you will separate the two heaps and
> force the buddy allocator to allocate xen heap pages from a specific region?
> 
> [...]

I will try to explain clearly here. 
The intention behind this reserved heap is that for supporting total static 
system, we
not only want to pre-define memory resource for guests, but also for xen runtime
allocation. Any runtime behavior are more predictable.

Right now, on AArch64, all RAM, except reserved memory, will be given to buddy
allocator as heap,  like you said, guest RAM for normal domain will be allocated
from there, xmalloc eventually is get memory from there, etc. So we want to 
refine
the heap here, not iterating through bootinfo.mem to set up XEN heap, but like
iterating bootinfo. reserved_heap to set up XEN heap.

True, on ARM32, xen heap and domain heap are separately mapped, which is more
complicated here. That's why I only talking about implementing these features on
AArch64 as first step.

> 
> > +### Memory Allocation for 1:1 direct-map Domain
> > +
> > +Implementing memory allocation for 1:1 direct-map domain includes two
> parts:
> > +Static Allocation for Domain and 1:1 direct-map.
> > +
> > +The first part has been elaborated before in chapter `Memory
> > +Allocation for Domains on Static Allocation`. Then, to ensure 1:1
> > +direct-map, when setting up guest P2M mapping, it needs to make sure
> > +that guest physical address equal to physical address(`gfn == mfn`).
> > +
> > +*DISCUSSION:
> > +
> > +  * Here only supports booting up one domain on static allocation or
> > +on 1:1 direct-map through device tree, is `xl` also needed?
> 
> I think they can be separated for now.
> 

Agree~~

> Cheers,
> 
> --
> Julien Grall