Re: [Xen-devel] Question about memory allocation on NUMA node.

[Yuji Shimada <shimada-yxb@xxxxxxxxxxxxxxx>]

Thanks for your reply.

On Fri, 17 Oct 2008 14:10:49 +0200
Andre Przywara <andre.przywara@xxxxxxx> wrote:

> Yuji,
> 
> > I think that the memory relating guest domain should be allocated from
> > the NUMA node on which the guest run.
> > Because the latency of the same NUMA node is better than that of a
> > different one.
> > 
> > According to this idea, most of the codes are good in xen-unstable.
> > But some memory relating guest domain are allocated from the NUMA node
> > on which CPU #0 run.
> No, it will default to the node on which the domain runs, see below.
> > 
> > For example,
> >     - xen/arch/x86/domain.c
> >         setup_compat_l4(struct vcpu *v)
> >           struct page_info *pg = alloc_domheap_page(NULL, 0);
> 0 for memflags means default behaviour, if you look at the
> implementation of alloc_domheap_pages
> (http://lxr.xensource.com/lxr/source/xen/common/page_alloc.c?a=x86_64#L774)
> a value of zero will evalutate to node=NUMA_NO_NODE, which in turn
> will be replaced by 'node = domain_to_node(d);', which is what you
> wanted to insert below. So I see no problem here.

If first domain parameter is 'NULL' and second memflag parameter is
'0' in alloc_domheap_page, the 'node' value is NOT replaced by
'domain_to_node(d)'.
So the 'node' value is 'NUMA_NO_NODE'.

Please see below.

xen/common/page_alloc.c
774 struct page_info *alloc_domheap_pages(
775     struct domain *d, unsigned int order, unsigned int memflags)
776 {
777     struct page_info *pg = NULL;
778     unsigned int bits = memflags >> _MEMF_bits, zone_hi = NR_ZONES - 1;
779     unsigned int node = (uint8_t)((memflags >> _MEMF_node) - 1);
780 
781     ASSERT(!in_irq());
782 
783     if ( (node == NUMA_NO_NODE) && (d != NULL) )
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
784         node = domain_to_node(d);
…
810 }


So it is better to use NUMA node number of memflag parameter to allocate
memory from guest's NUMA node.

For example,
    '*pg = alloc_domheap_page(NULL, 0);'
    is better to be replaced by the following examples.

    '*pg = alloc_domheap_page(NULL, MEMF_node(domain_to_node(d)));'

What do you think?

Thanks,

--
Yuji Shimada

> 
> Regards,
> Andre.
> 
> > 
> > I think this memory should be allocated from the NUMA node on which
> > the guest run.
> > 
> > For example, 
> >     - xen/arch/x86/domain.c at
> >         setup_compat_l4(struct vcpu *v)
> >           struct page_info *pg = alloc_domheap_page(NULL,
> >                                  MEMF_node(domain_to_node(v->domain)));
> > 
> > As a result, machine performance becomes better.
> > 
> > What do you think about this idea? 
> > I'd like some comments.
> > 
> > If the developers agree with me, I would like to list them and submit
> > patch.
> > 
> --
> Andre Przywara
> AMD-Operating System Research Center (OSRC), Dresden, Germany
> Tel: +49 351 277-84917
> ----to satisfy European Law for business letters:
> AMD Saxony Limited Liability Company & Co. KG,
> Wilschdorfer Landstr. 101, 01109 Dresden, Germany
> Register Court Dresden: HRA 4896, General Partner authorized
> to represent: AMD Saxony LLC (Wilmington, Delaware, US)
> General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel