RE: [PATCH 08/10] xen/arm: introduce reserved_page_list

[Penny Zheng <Penny.Zheng@xxxxxxx>]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xxxxxxx>
> Sent: Tuesday, May 18, 2021 7:02 PM
> To: Penny Zheng <Penny.Zheng@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx;
> sstabellini@xxxxxxxxxx
> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; Wei Chen
> <Wei.Chen@xxxxxxx>; nd <nd@xxxxxxx>
> Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list
> 
> Hi Penny,
> 
> On 18/05/2021 06:21, Penny Zheng wrote:
> > Since page_list under struct domain refers to linked pages as gueast
> > RAM
> 
> s/gueast/guest/
> 

Thx~

> > allocated from heap, it should not include reserved pages of static memory.
> 
> You already have PGC_reserved to indicate they are "static memory". So why
> do you need yet another list?
> 
> >
> > The number of PGC_reserved pages assigned to a domain is tracked in a
> > new 'reserved_pages' counter. Also introduce a new reserved_page_list
> > to link pages of static memory. Let page_to_list return
> > reserved_page_list, when flag is PGC_reserved.
> >
> > Later, when domain get destroyed or restarted, those new values will
> > help relinquish memory to proper place, not been given back to heap.
> >
> > Signed-off-by: Penny Zheng <penny.zheng@xxxxxxx>
> > ---
> >   xen/common/domain.c     | 1 +
> >   xen/common/page_alloc.c | 7 +++++--
> >   xen/include/xen/sched.h | 5 +++++
> >   3 files changed, 11 insertions(+), 2 deletions(-)
> >
> > diff --git a/xen/common/domain.c b/xen/common/domain.c index
> > 6b71c6d6a9..c38afd2969 100644
> > --- a/xen/common/domain.c
> > +++ b/xen/common/domain.c
> > @@ -578,6 +578,7 @@ struct domain *domain_create(domid_t domid,
> >       INIT_PAGE_LIST_HEAD(&d->page_list);
> >       INIT_PAGE_LIST_HEAD(&d->extra_page_list);
> >       INIT_PAGE_LIST_HEAD(&d->xenpage_list);
> > +    INIT_PAGE_LIST_HEAD(&d->reserved_page_list);
> >
> >       spin_lock_init(&d->node_affinity_lock);
> >       d->node_affinity = NODE_MASK_ALL; diff --git
> > a/xen/common/page_alloc.c b/xen/common/page_alloc.c index
> > f1f1296a61..e3f07ec6c5 100644
> > --- a/xen/common/page_alloc.c
> > +++ b/xen/common/page_alloc.c
> > @@ -2410,7 +2410,7 @@ int assign_pages(
> >
> >           for ( i = 0; i < nr_pfns; i++ )
> >           {
> > -            ASSERT(!(pg[i].count_info & ~PGC_extra));
> > +            ASSERT(!(pg[i].count_info & ~(PGC_extra |
> > + PGC_reserved)));
> I think this change belongs to the previous patch.
> 

Ok. It will be re-organized into previous commit of
"xen/arm: intruduce alloc_domstatic_pages"

> >               if ( pg[i].count_info & PGC_extra )
> >                   extra_pages++;
> >           }
> > @@ -2439,6 +2439,9 @@ int assign_pages(
> >           }
> >       }
> >
> > +    if ( pg[0].count_info & PGC_reserved )
> > +        d->reserved_pages += nr_pfns;
> 
> reserved_pages doesn't seem to be ever read or decremented. So why do
> you need it?
>

Yeah, I may delete it from this Patch Serie.

Like I addressed in before commits:

"when implementing rebooting domain on static allocation, memory will be 
relinquished
and right now, all shall be freed back to heap, which is not suitable for 
static memory here.
` relinquish_memory(d, &d->page_list)  --> put_page -->  free_domheap_page`

For pages in PGC_reserved, now, I am considering that, other than giving it 
back to heap,
maybe creating a new global `struct page_info*[DOMID]` value to hold.

So it is better to have a new field in struct page_info, as follows, to hold 
such info.

/* Page is reserved. */
struct {
    /*
     * Reserved Owner of this page,
     * if this page is reserved to a specific domain.
     */
    domid_t reserved_owner;
} reserved;
" 

So I will delete here, and re-import when implementing rebooting domain on 
static allocation.

> > +
> >       if ( !(memflags & MEMF_no_refcount) &&
> >            unlikely(domain_adjust_tot_pages(d, nr_pfns) == nr_pfns) )
> >           get_knownalive_domain(d);
> > @@ -2452,7 +2455,7 @@ int assign_pages(
> >               page_set_reserved_owner(&pg[i], d);
> >           smp_wmb(); /* Domain pointer must be visible before updating
> refcnt. */
> >           pg[i].count_info =
> > -            (pg[i].count_info & PGC_extra) | PGC_allocated | 1;
> > +            (pg[i].count_info & (PGC_extra | PGC_reserved)) |
> > + PGC_allocated | 1;
> 
> Same here.

I'll re-organize it to the previous commit.

> 
> >           page_list_add_tail(&pg[i], page_to_list(d, &pg[i]));
> >       }
> >
> > diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index
> > 3982167144..b6333ed8bb 100644
> > --- a/xen/include/xen/sched.h
> > +++ b/xen/include/xen/sched.h
> > @@ -368,6 +368,7 @@ struct domain
> >       struct page_list_head page_list;  /* linked list */
> >       struct page_list_head extra_page_list; /* linked list (size 
> > extra_pages) */
> >       struct page_list_head xenpage_list; /* linked list (size
> > xenheap_pages) */
> > +    struct page_list_head reserved_page_list; /* linked list (size
> > + reserved pages) */
> >
> >       /*
> >        * This field should only be directly accessed by
> > domain_adjust_tot_pages() @@ -379,6 +380,7 @@ struct domain
> >       unsigned int     outstanding_pages; /* pages claimed but not possessed
> */
> >       unsigned int     max_pages;         /* maximum value for
> domain_tot_pages() */
> >       unsigned int     extra_pages;       /* pages not included in
> domain_tot_pages() */
> > +    unsigned int     reserved_pages;    /* pages of static memory */
> >       atomic_t         shr_pages;         /* shared pages */
> >       atomic_t         paged_pages;       /* paged-out pages */
> >
> > @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list(
> >       if ( pg->count_info & PGC_extra )
> >           return &d->extra_page_list;
> >
> > +    if ( pg->count_info & PGC_reserved )
> > +        return &d->reserved_page_list;
> > +
> >       return &d->page_list;
> >   }
> >
> >
> 
> Cheers,
> 
> --
> Julien Grall

Cheers

Penny Zheng