Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Thanks for the big contribution!

I just wanted to let you know that the series passed all the gitlab-ci
build tests without issues.

The runtime tests originally failed due to unrelated problems (there was
a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying
issue and restarted the failed tests and now they passed.

This is the pipeline:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940

There are still two runtime x86 tests that fail but I don't think the
failures are related to your series.


On Wed, 11 Aug 2021, Wei Chen wrote:
> Xen memory allocation and scheduler modules are NUMA aware.
> But actually, on x86 has implemented the architecture APIs
> to support NUMA. Arm was providing a set of fake architecture
> APIs to make it compatible with NUMA awared memory allocation
> and scheduler.
> 
> Arm system was working well as a single node NUMA system with
> these fake APIs, because we didn't have multiple nodes NUMA
> system on Arm. But in recent years, more and more Arm devices
> support multiple nodes NUMA system. Like TX2, some Hisilicon
> chips and the Ampere Altra.
> 
> So now we have a new problem. When Xen is running on these Arm
> devices, Xen still treat them as single node SMP systems. The
> NUMA affinity capability of Xen memory allocation and scheduler
> becomes meaningless. Because they rely on input data that does
> not reflect real NUMA layout.
> 
> Xen still think the access time for all of the memory is the
> same for all CPUs. However, Xen may allocate memory to a VM
> from different NUMA nodes with different access speeds. This
> difference can be amplified in workloads inside VM, causing
> performance instability and timeouts. 
> 
> So in this patch series, we implement a set of NUMA API to use
> device tree to describe the NUMA layout. We reuse most of the
> code of x86 NUMA to create and maintain the mapping between
> memory and CPU, create the matrix between any two NUMA nodes.
> Except ACPI and some x86 specified code, we have moved other
> code to common. In next stage, when we implement ACPI based
> NUMA for Arm64, we may move the ACPI NUMA code to common too,
> but in current stage, we keep it as x86 only.
> 
> This patch serires has been tested and booted well on one
> Arm64 NUMA machine and one HPE x86 NUMA machine.
> 
> Hongda Deng (2):
>   xen/arm: return default DMA bit width when platform is not set
>   xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
> 
> Wei Chen (38):
>   tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
>   xen/arm: Print a 64-bit number in hex from early uart
>   xen/x86: Initialize memnodemapsize while faking NUMA node
>   xen: decouple NUMA from ACPI in Kconfig
>   xen/arm: use !CONFIG_NUMA to keep fake NUMA API
>   xen/x86: Move NUMA memory node map functions to common
>   xen/x86: Move numa_add_cpu_node to common
>   xen/x86: Move NR_NODE_MEMBLKS macro to common
>   xen/x86: Move NUMA nodes and memory block ranges to common
>   xen/x86: Move numa_initmem_init to common
>   xen/arm: introduce numa_set_node for Arm
>   xen/arm: set NUMA nodes max number to 64 by default
>   xen/x86: move NUMA API from x86 header to common header
>   xen/arm: Create a fake NUMA node to use common code
>   xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
>   xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
>   xen: fdt: Introduce a helper to check fdt node type
>   xen/arm: implement node distance helpers for Arm64
>   xen/arm: introduce device_tree_numa as a switch for device tree NUMA
>   xen/arm: introduce a helper to parse device tree processor node
>   xen/arm: introduce a helper to parse device tree memory node
>   xen/arm: introduce a helper to parse device tree NUMA distance map
>   xen/arm: unified entry to parse all NUMA data from device tree
>   xen/arm: Add boot and secondary CPU to NUMA system
>   xen/arm: build CPU NUMA node map while creating cpu_logical_map
>   xen/x86: decouple nodes_cover_memory with E820 map
>   xen/arm: implement Arm arch helpers Arm to get memory map info
>   xen: move NUMA memory and CPU parsed nodemasks to common
>   xen/x86: move nodes_cover_memory to common
>   xen/x86: make acpi_scan_nodes to be neutral
>   xen: export bad_srat and srat_disabled to extern
>   xen: move numa_scan_nodes from x86 to common
>   xen: enable numa_scan_nodes for device tree based NUMA
>   xen/arm: keep guest still be NUMA unware
>   xen: introduce an arch helper to do NUMA init failed fallback
>   xen/arm: enable device tree based NUMA in system init
>   xen/x86: move numa_setup to common to support NUMA switch in command
>     line
>   xen/x86: move dump_numa info hotkey to common
> 
>  tools/libs/util/libxlu_pci.c    |   3 +-
>  xen/arch/arm/Kconfig            |  10 +
>  xen/arch/arm/Makefile           |   2 +
>  xen/arch/arm/arm64/head.S       |   9 +-
>  xen/arch/arm/bootfdt.c          |   8 +-
>  xen/arch/arm/domain_build.c     |  17 +-
>  xen/arch/arm/efi/efi-boot.h     |  25 --
>  xen/arch/arm/numa.c             | 162 +++++++++
>  xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
>  xen/arch/arm/platform.c         |   4 +-
>  xen/arch/arm/setup.c            |  14 +
>  xen/arch/arm/smpboot.c          |  37 +-
>  xen/arch/x86/Kconfig            |   2 +-
>  xen/arch/x86/numa.c             | 421 +----------------------
>  xen/arch/x86/srat.c             | 147 +-------
>  xen/common/Kconfig              |   3 +
>  xen/common/Makefile             |   1 +
>  xen/common/libfdt/fdt_ro.c      |  15 +
>  xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
>  xen/common/page_alloc.c         |   2 +-
>  xen/drivers/acpi/Kconfig        |   3 +-
>  xen/drivers/acpi/Makefile       |   2 +-
>  xen/include/asm-arm/numa.h      |  33 ++
>  xen/include/asm-arm/setup.h     |   6 +
>  xen/include/asm-x86/acpi.h      |   4 -
>  xen/include/asm-x86/config.h    |   1 -
>  xen/include/asm-x86/numa.h      |  65 +---
>  xen/include/asm-x86/setup.h     |   1 -
>  xen/include/xen/libfdt/libfdt.h |  25 ++
>  xen/include/xen/nodemask.h      |   2 +
>  xen/include/xen/numa.h          |  80 +++++
>  31 files changed, 1325 insertions(+), 659 deletions(-)
>  create mode 100644 xen/arch/arm/numa.c
>  create mode 100644 xen/arch/arm/numa_device_tree.c
>  create mode 100644 xen/common/numa.c
> 
> -- 
> 2.25.1
>