Re: [PATCH v1 2/6] xen/riscv: introduce things necessary for p2m initialization

[Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>]

On 5/22/25 6:09 PM, Jan Beulich wrote:

> On 22.05.2025 17:53, Oleksii Kurochko wrote:
> > On 5/20/25 3:37 PM, Jan Beulich wrote:
> > > On 09.05.2025 17:57, Oleksii Kurochko wrote:
> > > > +static struct page_info *p2m_get_clean_page(struct domain *d)
> > > > +{
> > > > +    struct page_info *page;
> > > > +
> > > > +    /*
> > > > +     * As mentioned in the Priviliged Architecture Spec (version 20240411)
> > > > +     * As explained in Section 18.5.1, for the paged virtual-memory schemes
> > > > +     * (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB
> > > > +     * and must be aligned to a 16-KiB boundary.
> > > > +     */
> > > > +    page = alloc_domheap_pages(NULL, 2, 0);
> > > >           
> > > Shouldn't this allocation come from the domain's P2M pool (which is yet
> > > to be introduced)?
> > >         
> > First, I will drop p2m_get_clean_page() as it will be used only for p2m root page
> > table allocation.
> >
> > p2m_init() is called by domain_create() [->arch_domain_create()->p2m_init()] from create_domUs():
> > [https://gitlab.com/xen-project/xen/-/blob/staging/xen/common/device-tree/dom0less-build.c?ref_type=heads#L984].
> >
> > When p2m_init() is called, p2m pool isn't ready and domain isn't created yet. Last one
> > is also crucial for usage of p2m pool as p2m pool belongs to domain and thereby it is
> > using alloc_domheap_page(d, ...) (Not NULL as for allocation of p2m root table above),
> > so domain should be created first.
> >       
> Yet that is part of my point: This allocation should be against the domain,
> so it is properly accounted. What's the problem with allocating the root
> table when the pools is being created / filled?
>     
I can't use pages from pool for root table as they aren't properly aligned.

At the moment, creation of p2m pool looks like:
 int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted)
 {
     struct page_info *pg;

     ASSERT(spin_is_locked(&d->arch.paging.lock));

     for ( ; ; )
     {
         if ( d->arch.paging.p2m_total_pages < pages )
         {
             /* Need to allocate more memory from domheap */
             pg = alloc_domheap_page(d, MEMF_no_owner);
             if ( pg == NULL )
             {
                 printk(XENLOG_ERR "Failed to allocate P2M pages.\n");
                 return -ENOMEM;
             }
             ACCESS_ONCE(d->arch.paging.p2m_total_pages) =
                 d->arch.paging.p2m_total_pages + 1;
             page_list_add_tail(pg, &d->arch.paging.p2m_freelist);
         }
         ...
     }

     return 0;
 }
alloc_domheap_page(d, MEMF_no_owner) allocates page table with order 0, so 4k-aligned page table.
But if I needed 16k for root table and it should be 16k-aligned then I still have to use
alloc_domheap_pages(NULL, 2, 0);

Or do you mean that I have to something like:
 int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted)
 {
     struct page_info *pg;
 
     ASSERT(spin_is_locked(&d->arch.paging.lock));
 
+    if ( !d->arch.p2m.root )
+    {
+        unsigned int order = get_order_from_bytes(KB(16));
+        unsigned int nr_pages = _AC(1,U) << order;
+        /*
+        * As mentioned in the Priviliged Architecture Spec (version 20240411)
+        * As explained in Section 18.5.1, for the paged virtual-memory schemes
+        * (Sv32x4, Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB
+        * and must be aligned to a 16-KiB boundary.
+        */
+        d->arch.p2m.root = alloc_domheap_pages(d, order, MEMF_no_owner);
+        if ( d->arch.p2m.root == NULL )
+            panic("root page table hasn't been allocated\n");
+
+        clear_and_clean_page(d->arch.p2m.root);
+
+        /* TODO: do I need TLB flush here? */
+
+        ACCESS_ONCE(d->arch.paging.p2m_total_pages) =
+            d->arch.paging.p2m_total_pages + nr_pages;
+    }
+
...
}
(I will the current version of p2m_alloc_table() instead of open-coding.)


> > > > +{
> > > > +    unsigned long ppn;
> > > > +    unsigned long hgatp_mode;
> > > > +
> > > > +    ppn = PFN_DOWN(page_to_maddr(page_info)) & HGATP_PPN;
> > > > +
> > > > +    /* ASID (VMID) not supported yet */
> > > > +
> > > > +#if RV_STAGE1_MODE == SATP_MODE_SV39
> > > > +    hgatp_mode = HGATP_MODE_SV39X4;
> > > > +#elif RV_STAGE1_MODE == SATP_MODE_SV48
> > > > +    hgatp_mode = HGATP_MODE_SV48X4;
> > > > +#else
> > > > +    #error "add HGATP_MODE"
> > > >           
> > > As before, please have the # of pre-processor directives in the first column.
> > >
> > > > +#endif
> > > > +
> > > > +    return ppn | (hgatp_mode << HGATP_MODE_SHIFT);
> > > >           
> > > Use MASK_INSR()?
> > >         
> > Do you mean MASK_INSR(hgatp_mode, HGATP_MODE_MASK)?
> > If yes, then I didn't get what is the point then?
> >       
> The point is that generally ..._SHIFT is redundant when you also have
> ..._MASK; that's what MASK_EXTR() and MASK_INSR() leverage.
>     
At the moment, there is no mask for HGATP_MODE so if to use *_MASK then I
have to introduce it if it better to have *_MASK instead of *_SHIFT.

> > > > +static int p2m_alloc_table(struct domain *d)
> > > > +{
> > > > +    struct p2m_domain *p2m = p2m_get_hostp2m(d);
> > > > +
> > > > +    p2m->root = p2m_allocate_root(d);
> > > > +    if ( !p2m->root )
> > > > +        return -ENOMEM;
> > > > +
> > > > +    p2m->hgatp = hgatp_from_page_info(p2m->root);
> > > > +
> > > > +    /*
> > > > +     * Make sure that all TLBs corresponding to the new VMID are flushed
> > > > +     * before using it.
> > > > +     */
> > > > +    p2m_write_lock(p2m);
> > > > +    p2m_force_tlb_flush_sync(p2m);
> > > > +    p2m_write_unlock(p2m);
> > > >           
> > > While Andrew directed you towards a better model in general, it won't be
> > > usable here then, as the guest didn't run on any pCPU(s) yet. Imo you
> > > want to do a single global flush e.g. when VMIDs wrap around. That'll be
> > > fewer global flushes than one per VM creation.
> > >         
> > I am not sure that I get a phrase 'VMIDs wrap around'.
> >       
> You have to allocate them somehow. Typically you'll use the next one available.
> At some point you will need to start over, searching from the beginning. Prior
> to that now allocation of a new one will require any flush, as none of them
> had be in use before (after boot or the last such flush).
>     
Thanks. Now I get your point.

Won't be better to do TLB flushing during destroying of a domain so then we will
be sure that TLBs connected to freed VMID aren't present in TLB anymore?

IIUC, it will work only if VMID is used, right?
In case if VMID isn't used, probably we can drop flushing here and do a flush
during booting, right?
Won't be enough to flushing of guest TLB only during context switch?

> > I am going to implement, p2m_force_tlb_flush_sync() as:
> >   static void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
> >   {
> >     ...
> >       sbi_remote_hfence_gvma(d->dirty_cpumask, 0, 0);
> >     ...
> >   }
> >
> > With such implementation if the guest didn't run on any pCPU(s) yet
> > then d->dirty_cpumask is empty, then sbi_remote_hfence_gvma() will do nothing
> > as hmask will be NULL (https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/riscv/sbi.c?ref_type=heads#L238).
> > I am not sure that it is a good idea as I can't find a guarantee in the spec
> > that TLB will be empty during boot time.
> >       
> If in doubt, do one global flush while booting.
>     
By booting you mean somewhere in continue_new_vcpu()?

~ Oleksii