RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Julien, Jan

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of Wei
> Chen
> Sent: 2021年8月28日 11:09
> To: Julien Grall <julien@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx;
> sstabellini@xxxxxxxxxx; Jan Beulich <jbeulich@xxxxxxxx>
> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>
> Subject: RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do
> NUMA init failed fallback
> 
> Hi Julien,
> 
> > -----Original Message-----
> > From: Julien Grall <julien@xxxxxxx>
> > Sent: 2021年8月27日 22:30
> > To: Wei Chen <Wei.Chen@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx;
> > sstabellini@xxxxxxxxxx; jbeulich@xxxxxxxx
> > Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>
> > Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do
> > NUMA init failed fallback
> >
> > Hi,
> >
> > On 11/08/2021 11:24, Wei Chen wrote:
> > > When Xen initialize NUMA failed, some architectures may need to
> > > do fallback actions. For example, in device tree based NUMA, Arm
> > > need to reset the distance between any two nodes.
> >
> >  From the description here, I don't understand why we need to reset the
> > distance for Arm but not x86. In fact...
> >
> > >
> > > Signed-off-by: Wei Chen <wei.chen@xxxxxxx>
> > > ---
> > >   xen/arch/arm/numa.c        | 13 +++++++++++++
> > >   xen/common/numa.c          |  3 +++
> > >   xen/include/asm-arm/numa.h |  1 +
> > >   xen/include/asm-x86/numa.h |  6 ++++++
> > >   4 files changed, 23 insertions(+)
> > >
> > > diff --git a/xen/arch/arm/numa.c b/xen/arch/arm/numa.c
> > > index 6eebf8e8bc..2a18c97470 100644
> > > --- a/xen/arch/arm/numa.c
> > > +++ b/xen/arch/arm/numa.c
> > > @@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int
> bank,
> > >
> > >           return 0;
> > >   }
> > > +
> > > +void __init arch_numa_init_failed_fallback(void)
> > > +{
> > > +    int i, j;
> > > +
> > > +    /* Reset all node distance to remote_distance */
> > > +    for ( i = 0; i < MAX_NUMNODES; i++ ) {
> > > +        for ( j = 0; j < MAX_NUMNODES; j++ ) {
> > > +            numa_set_distance(i, j,
> > > +                (i == j) ? NUMA_LOCAL_DISTANCE :
> NUMA_REMOTE_DISTANCE);
> > > +        }
> > > +    }
> > > +}
> >
> > ... this implementation looks fairly generic. So can you explain why we
> > need it on Arm but not x86?
> >
> 
> This implementation is DT only, for x86, it's using acpi_slit.
> For now, I am not quit sure ACPI need to do fallback or not.
> Or say in another way, I don't know how to implement the fallback
> for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left
> an empty helper for x86.
> 
> @Jan Beulich Could you give me some suggestion about x86 fallback?
> 
> 

I have a quick look into Linux. When Arch do numa init failed,
the numa_free_distance will be invoked to revert numa_distance.


> > > diff --git a/xen/common/numa.c b/xen/common/numa.c
> > > index d15c2fc311..88f1594127 100644
> > > --- a/xen/common/numa.c
> > > +++ b/xen/common/numa.c
> > > @@ -405,4 +405,7 @@ void __init numa_initmem_init(unsigned long
> > start_pfn, unsigned long end_pfn)
> > >       cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
> > >       setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
> > >                       (u64)end_pfn << PAGE_SHIFT);
> > > +
> > > +    /* architecture specified fallback operations */
> > > +    arch_numa_init_failed_fallback();
> > >   }
> > > diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> > > index dd31324b0b..a3982a94b6 100644
> > > --- a/xen/include/asm-arm/numa.h
> > > +++ b/xen/include/asm-arm/numa.h
> > > @@ -28,6 +28,7 @@ extern s8 device_tree_numa;
> > >   extern void numa_init(bool acpi_off);
> > >   extern int numa_device_tree_init(const void *fdt);
> > >   extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t
> > distance);
> > > +extern void arch_numa_init_failed_fallback(void);
> > >
> > >   /*
> > >    * Temporary for fake NUMA node, when CPU, memory and distance
> > > diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> > > index e63869135c..26280b0f3a 100644
> > > --- a/xen/include/asm-x86/numa.h
> > > +++ b/xen/include/asm-x86/numa.h
> > > @@ -22,4 +22,10 @@ extern void init_cpu_to_node(void);
> > >   void srat_parse_regions(u64 addr);
> > >   unsigned int arch_get_dma_bitsize(void);
> > >
> > > +/* Dummy function for numa init failed in numa_initmem_init */
> > > +static inline void arch_numa_init_failed_fallback(void)
> > > +{
> > > +    return;
> >
> > NIT: The return is pointless.
> >
> 
> OK
> 
> > > +}
> > > +
> > >   #endif
> > >
> >
> > Cheers,
> >
> > --
> > Julien Grall