RE: DMA restriction and NUMA node number

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Jan,

> -----Original Message-----
> From: Jan Beulich <jbeulich@xxxxxxxx>
> Sent: 2021年7月13日 18:21
> To: Julien Grall <julien@xxxxxxx>; Wei Chen <Wei.Chen@xxxxxxx>
> Cc: Penny Zheng <Penny.Zheng@xxxxxxx>; Bertrand Marquis
> <Bertrand.Marquis@xxxxxxx>; xen-devel@xxxxxxxxxxxxx; Roger Pau Monné
> <roger.pau@xxxxxxxxxx>; Stefano Stabellini <sstabellini@xxxxxxxxxx>;
> Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> Subject: Re: DMA restriction and NUMA node number
>
> On 13.07.2021 11:26, Julien Grall wrote:
> > On 13/07/2021 04:19, Wei Chen wrote:
> >> I am doing some NUMA testing on Xen. And I find the DMA restriction is
> >> based on NUMA node number [1].
> >>      if ( !dma_bitsize && (num_online_nodes() > 1) )
> >>          dma_bitsize = arch_get_dma_bitsize();
> >>
> >> On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
> >> reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
> >> will override to 32. This may be caused by the Arm64 version
> >> arch_get_dma_bitsize, it may be a simple implementation and not NUMA
> >> aware.
> >>
> >> But I still quite curious about why DMA restriction depends on NUMA
> >> node number.
>
> So really do you mean "node count", not "node number"?
>

I mean NUMA online nodes. If "node count" here means "count how many NUMA
nodes are online", my answer is yes for "node count".

> >> In Arm64, dma_bitsize does not change when the NUMA node
> >> changes. So we didn't expect arch_get_dma_bitsize to be called here.
> >>
> >> I copied Keir's commit message from 2008. It seems this code was
> considered
> >> only for x86, when he was working on it. But I'm not an x86 expert, so
> I
> >> hope Xen x86 folks can give some help. Understanding this will help us
> to
> >
> > It is best to CCed the relevant person so they know you have requested
> > there input. I have added the x86 maintainers in the thread.
> >
> >> do some adaptations to Arm in subsequent modifications : )
> >>
> >> commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
> >> Author: Keir Fraser <keir.fraser@xxxxxxxxxx>
> >> Date:   Mon Jul 28 16:40:30 2008 +0100
> >>
> >>      Simplify 'dma heap' logic.
> >>
> >>      1. Only useful for NUMA systems, so turn it off on non-NUMA
> systems by
> >>         default.
> >>      2. On NUMA systems, by default relate the DMA heap size to NUMA
> node 0
> >>         memory size (so that not all of node 0's memory ends up being
> 'DMA
> >>         heap').
> >>      3. Remove the 'dma emergency pool'. It's less useful now that
> running
> >>         out of low memory isn;t as fatal as it used to be (e.g., when
> we
> >>         needed to be able to allocate low-memory PAE page directories).
>
> So on x86 memory starts from 0, and we want to be cautious with giving
> out memory that may be needed for special purposes (first and foremost
> DMA). With the buddy allocator working from high addresses to lower ones,
> low addresses will be used last (unless specifically requested) without
> any further precautions when not taking NUMA into account. This in
> particular covers the case of just a single NUMA node.
>
> When taking NUMA into account we need to be more careful: If a single
> node contains the majority (or all) of the more precious memory, we
> want to prefer non-local allocations over exhausting the more precious
> memory ranges. Hence we need to set aside some largely arbitrary amount
> allocation of which would happen only after also exhausting all other
> nodes' memory.
>
> I hope I have suitably reconstructed the thinking back then. And yes,
> there are x86 implications in here.
>

Thanks for your explanation, we have a deeper understanding of the
thoughts of this DMA restriction. Although the memory layout of Arm
and x86 is not the same, we can try to keep this idea still in common,
and try resolve this difference in Arm's ARCH.

Thanks,
Wei Chen

> Jan

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended recipient, 
please notify the sender immediately and do not disclose the contents to any 
other person, use it for any purpose, or store or copy the information in any 
medium. Thank you.