[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

DMA restriction and NUMA node number

  • To: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>
  • From: Wei Chen <Wei.Chen@xxxxxxx>
  • Date: Tue, 13 Jul 2021 03:19:38 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HE4/j+9ppXS1MAB9BJaWimkipXECDi8DSsGDky5Uqow=; b=cSFjsPzfbVmldRmsYduElN+v8fWO9HuVlgx8YZ7YtBe3SjDDSgQLHuWOZtNWGv0pumP2hYkaG2SZqBeTXeKpJvx5wr2lHU7Qn4e+Ag7wsdNkxgRD47OxZeS5MkxZ5zmc3GvwugPI2FFvxwRsruHLyxl6mQzoNkSj/iLmD6zDhi0vZnPnNTBx9XQzhpeOdJzFCzsII027Ak/zqR6iVPiymfqRhn+/5v0UvlA5ttfFQnJNL5iXTRnK8r9qFvCr9bUo3HRubw21E0x8mLKvmEljf5uHtNSpr1R6HtDRJUgC+v1CYRE3lnXIZkLIS1nc2zDhxNVuZ7pYRgAdyWGUYzsJ5A==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LP9GIMl4+RB10vyGUDUEbhV98Cz4Y81/vJJhapeYQ4NcTT57v/bRhSRUkVKDBxUqBroi1lT3OgrtpLdjX//+3SxQCG9Z6c8lKCY2RA9arJNEuyViXXaBrFNjJVb5Z20SKjozh4ppKYywcNXHIjYPGU9VtDR2wGYxwwSrqpQdO0BppPA7mpOSS8e9ffxnaXKtSVq9nh21d1WVC3X1gLcLQDeqpd9+tEGbwUloFqHuYs1d/W0OODOskri6ayNPgmVkUXmHV3FoRR6m3wxM4q+sv87sLQxbdYzXkk0HeTs+gdrkWxe45lET/s1V3KH5/+ck51stybJ/iXHZ+SHr8zhsIA==
  • Authentication-results-original: lists.xen.org; dkim=none (message not signed) header.d=none;lists.xen.org; dmarc=none action=none header.from=arm.com;
  • Cc: Penny Zheng <Penny.Zheng@xxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>
  • Delivery-date: Tue, 13 Jul 2021 03:20:05 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: lists.xen.org; dkim=none (message not signed) header.d=none;lists.xen.org; dmarc=none action=none header.from=arm.com;
  • Thread-index: Add3jZlm0dMwxzV2RvOni5fmVkcvwg==
  • Thread-topic: DMA restriction and NUMA node number


I am doing some NUMA testing on Xen. And I find the DMA restriction is
based on NUMA node number [1].
    if ( !dma_bitsize && (num_online_nodes() > 1) )
        dma_bitsize = arch_get_dma_bitsize();

On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
will override to 32. This may be caused by the Arm64 version
arch_get_dma_bitsize, it may be a simple implementation and not NUMA

But I still quite curious about why DMA restriction depends on NUMA
node number. In Arm64, dma_bitsize does not change when the NUMA node
changes. So we didn't expect arch_get_dma_bitsize to be called here.

I copied Keir's commit message from 2008. It seems this code was considered
only for x86, when he was working on it. But I'm not an x86 expert, so I
hope Xen x86 folks can give some help. Understanding this will help us to
do some adaptations to Arm in subsequent modifications : )

commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
Author: Keir Fraser <keir.fraser@xxxxxxxxxx>
Date:   Mon Jul 28 16:40:30 2008 +0100

    Simplify 'dma heap' logic.

    1. Only useful for NUMA systems, so turn it off on non-NUMA systems by
    2. On NUMA systems, by default relate the DMA heap size to NUMA node 0
       memory size (so that not all of node 0's memory ends up being 'DMA
    3. Remove the 'dma emergency pool'. It's less useful now that running
       out of low memory isn;t as fatal as it used to be (e.g., when we
       needed to be able to allocate low-memory PAE page directories).


Wei Chen

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended recipient, 
please notify the sender immediately and do not disclose the contents to any 
other person, use it for any purpose, or store or copy the information in any 
medium. Thank you.



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.