Re: [Xen-devel] [PATCH] Question about memory allocation on NUMA node.

[Andre Przywara <andre.przywara@xxxxxxx>]

Yuji,

sorry for the late reply.
>>> I think that the memory relating guest domain should be allocated
>>> from the NUMA node on which the guest run.
>>> Because the latency of the same NUMA node is better than that of a
>>> different one.
>>>
>>> According to this idea, most of the codes are good in xen-unstable.
>>> But some memory relating guest domain are allocated from the NUMA
>>> node on which CPU #0 run.
>> No, it will default to the node on which the domain runs, see below.
>>> For example,
>>>     - xen/arch/x86/domain.c
>>>         setup_compat_l4(struct vcpu *v)
>>>           struct page_info *pg = alloc_domheap_page(NULL, 0);
>> 0 for memflags means default behaviour, if you look at the
>> implementation of alloc_domheap_pages
>> (http://lxr.xensource.com/lxr/source/xen/common/page_alloc.c?a=x86_64#L774)
>> a value of zero will evalutate to node=NUMA_NO_NODE, which in turn
>> will be replaced by 'node = domain_to_node(d);', which is what you
>> wanted to insert below. So I see no problem here.
> 
> If first domain parameter is 'NULL' and second memflag parameter is
> '0' in alloc_domheap_page, the 'node' value is NOT replaced by
> 'domain_to_node(d)'.
> So the 'node' value is 'NUMA_NO_NODE'.
Oh, you are right, I missed that.
> 
> Please see below.
> 
> xen/common/page_alloc.c
> 774 struct page_info *alloc_domheap_pages(
> 775     struct domain *d, unsigned int order, unsigned int memflags)
> 776 {
> 777     struct page_info *pg = NULL;
> 778     unsigned int bits = memflags >> _MEMF_bits, zone_hi =
> NR_ZONES - 1; 779     unsigned int node = (uint8_t)((memflags >>
> _MEMF_node) - 1); 780 
> 781     ASSERT(!in_irq());
> 782 
> 783     if ( (node == NUMA_NO_NODE) && (d != NULL) )
>         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 784         node = domain_to_node(d);
> …
> 810 }
> 
> 
> So it is better to use NUMA node number of memflag parameter to
> allocate memory from guest's NUMA node.
> 
> For example,
>     '*pg = alloc_domheap_page(NULL, 0);'
>     is better to be replaced by the following examples.
> 
>     '*pg = alloc_domheap_page(NULL, MEMF_node(domain_to_node(d)));'
> 
> What do you think?
I would revert to giving a valid domain pointer as the first parameter
instead of duplicating the code in alloc_domheap_pages().
Patch attached.

Signed-off-by: Andre Przywara <andre.przywara@xxxxxxx>

Regards,
Andre.

> 
> Thanks,
> 
> --
> Yuji Shimada
> 
>> Regards,
>> Andre.
>>
>>> I think this memory should be allocated from the NUMA node on which
>>> the guest run.
>>>
>>> For example, 
>>>     - xen/arch/x86/domain.c at
>>>         setup_compat_l4(struct vcpu *v)
>>>           struct page_info *pg = alloc_domheap_page(NULL,
>>>                                  MEMF_node(domain_to_node(v->domain)));
>>>
>>> As a result, machine performance becomes better.
>>>
>>> What do you think about this idea? 
>>> I'd like some comments.
>>>
>>> If the developers agree with me, I would like to list them and
>>> submit patch.
>>>
>> --
>> Andre Przywara
>> AMD-Operating System Research Center (OSRC), Dresden, Germany
>> Tel: +49 351 277-84917
>> ----to satisfy European Law for business letters:
>> AMD Saxony Limited Liability Company & Co. KG,
>> Wilschdorfer Landstr. 101, 01109 Dresden, Germany
>> Register Court Dresden: HRA 4896, General Partner authorized
>> to represent: AMD Saxony LLC (Wilmington, Delaware, US)
>> General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
> 

--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG,
Wilschdorfer Landstr. 101, 01109 Dresden, Germany
Register Court Dresden: HRA 4896, General Partner authorized
to represent: AMD Saxony LLC (Wilmington, Delaware, US)
General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy

Attachment: numa_alloc_compat_l4.patch
Description: Text Data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel