Re: [Xen-devel] arm64: Approach for DT based NUMA and issues

[Vijay Kilari <vijay.kilari@xxxxxxxxx>]

On Mon, Nov 28, 2016 at 2:21 AM, Dario Faggioli
<dario.faggioli@xxxxxxxxxx> wrote:
> On Sun, 2016-11-27 at 12:23 +0000, Julien Grall wrote:
>> Hi Dario,
>>
> Hi,
>
>> On 27/11/2016 01:01, Dario Faggioli wrote:
>> > On Sat, 2016-11-26 at 12:29 +0530, Vijay Kilari wrote:
>> > I agree that we need to support vNUMA for Dom0 sooner rather than
>> > later, and I agree that Dom0 is a bit special, so some tricks may
>> > be
>> > necessary. But until we don't implement vNUMA for Dom0, Dom0 is
>> > just a
>> > non-NUMA virtual machine, and the kernel running inside that should
>> > just behave like it behaves on a non-NUMA box.
>> >
>> > Again, I don't know much about ARM, but I think that, until we
>> > don't
>> > have vNUMA for Dom0, that devm_zalloc() thing will just see 1 and
>> > only
>> > 1 NUMA node from which to allocate memory.
>>
>> I would rather divide the NUMA work for ARM in 2 distinct tasks:
>>       - Make Xen NUMA-aware
>>       - Make DOM0 NUMA-aware
>>
> That makes perfect sense to me, and FWIW, is also what I'd do. In fact,
> the whole point of what I was saying was not to confuse Xen NUMA
> support and Dom0 NUMA support; if we want to do both of them, the
> latter right after the former, fine, but they're separate things
> indeed.
  Yes, agreed. Whatever the existing Xen NUMA-Aware code is completely kept
under x86, which can be used for arm as well. So needs cleanup and make common
for both archs.

  Regarding Dom0 NUMA-aware, in arm Dom0 is completely not NUMA-aware, not even
to the extent supported in x86.

>
>> Vijay, if I understood correctly what Dario said, on x86 DOM0 is not
>> yet
>> NUMA-aware.
>>
> You did. It is not.
>
>> > > Ex: SMMU driver of device on node 1 tries to allocate memory
>> > > on node 1.
>> > >
>> > > ISSUE:
>> > >  - Dom0's memory should be split across all the available memory
>> > > nodes
>> > >    of the system and memory nodes should be generated
>> > > accordingly.
>> > >
>> > ...This is the default behavior, at least on x86.
>>
>> Are you speaking about the command line parameter dom0_nodes?
>>
> Not exactly. As said, Dom0 is not NUMA aware and does not have any
> virtual NUMA layout.
>
> This means that, by default, Dom0 memory is indeed spread among various
> existing nodes. Eg., on my NUMA test box here at home, here's how
> things are for Dom0:

This default behaviour of spreading memory across existing nodes is
better to some
extent compared to ARM.. On ARM, All the allocation is based on allocator.
All it assumes all the memory is on single node.

>
> (XEN) [  970.100116] NODE0 start->1720320 size->1572864 free->0
> (XEN) [  970.100122] NODE1 start->0 size->1720320 free->460155
> (XEN) [  970.100130] CPU0...7 -> NODE0
> (XEN) [  970.100136] CPU8...15 -> NODE1
> (XEN) [  970.100140] Memory location of each domain:
> (XEN) [  970.100149] Domain 0 (total: 258512):
> (XEN) [  970.102268]     Node 0: 159254
> (XEN) [  970.102273]     Node 1: 99258
>
> dom0_nodes=x is a way to tell Xen to (try as hard as it can) to only
> allocate the memory for dom0 only from NUMA node x but, even if more
> than one node is specified, that does not include giving to him a
> virtual NUMA topology, nor making it aware of the underline NUMA
> topology of the host in any way.
>

   AFAIK, dom0_nodes is implemented only in x86 not in arm.

>> > Generating DT nodes for Dom0 is exactly what I mean when I say
>> > "implementing / enabling vNUMA for Dom0" (in this case on ARM).
>> >
>> > So, yes, let's do it, but let's discuss how to do it properly
>> > (e.g., if
>> > there's anything that can be common between archs, such as some
>> > bits of
>> > the interface).
>>
>> I would expect vNUMA for Dom0 to be common between x86 and ARM.
>>
> As much as possible, indeed.
>
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel