RFC: arm64: Handling reserved memory nodes

[Leo Yan <leo.yan@xxxxxxxxxx>]

Hi all,

I'd like to discuss for how to handle the reserved memory nodes in DT
binding on Xen / Arm64.  Note, now I am using DTB when boot Xen but
not UEFI/ACPI (ACPI is disabled in this case).

## Failure

I ported Xen on a platform, after the kernel booting, the Xen hypervisor
reports error:

  (XEN) arch/arm/p2m.c:2202: d0v0: Failing to acquire the MFN 0x1a02dc

This error is caused by kernel using an invalid memory frame number
0x1a02dc, we can convert it to the address:

  0x1a02dc << PAGE_SHIFT = 0x1_a02d_c000

## Reason

Two important things we need to check.  One is what's the DT binding
passed from the bootloader to Xen hypervisor, and the second thing is
what's the DT binding passed from Xen hypervisor to kernel.

We can see the bootloader passes below memory nodes to Xen hypervisor:

  (XEN) RAM: 0000000020000000 - 00000000bfffffff
  (XEN) RAM: 00000001a0000000 - 00000002ffffffff
  (XEN)
  (XEN) MODULE[0]: 0000000020100000 - 0000000020265000 Xen
  (XEN) MODULE[1]: 0000000023000000 - 0000000023024000 Device Tree
  (XEN) MODULE[2]: 0000000024000000 - 0000000028000000 Kernel
  (XEN)  RESVD[0]: 0000000020000000 - 000000002000ffff
  (XEN)  RESVD[1]: 0000000040000000 - 000000005fffffff
  (XEN)  RESVD[2]: 00000001a0000000 - 00000001bfffffff
  (XEN)  RESVD[3]: 000000002e000000 - 000000002fffffff

We can see the second DDR section is:

  [0x0000_0001_a000_0000 .. 0x0000_0002_ffff_ffff]

And there have reserved memory section is:

  [0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff]

When register the boot memory sections, dt_unreserved_regions() will
remove all reserved memory sections, which means the section
[0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff] is not managed by Xen
hypervisor at all.  If later kernel uses any pages in this range, Xen
will report the error.

But Dom0 kernel's memory nodes are:

  [    0.000000] Early memory node ranges
  [    0.000000]   node   0: [mem 0x0000000020000000-0x000000002000ffff]
  [    0.000000]   node   0: [mem 0x000000002e000000-0x000000002fffffff]
  [    0.000000]   node   0: [mem 0x0000000040000000-0x000000005fffffff]
  [    0.000000]   node   0: [mem 0x0000000060000000-0x0000000077bfffff]
  [    0.000000]   node   0: [mem 0x00000001a0000000-0x00000001bfffffff]

Based on this log, we can know Xen hypervisor passes the reserved memory
regions to Dom0 kernel, and when Dom0 kernel tries to allocate pages
from these reserved memory regions, Xen hypervisor reports error.

For more specific, this issue is cause by the commit 248faa637d ("xen/arm:
add reserved-memory regions to the dom0 memory node").  When Xen
hypervisor synthesizes DT nodes, it calls below function to generate
_normal_ memory nodes for the reserved memory regions.

  make_memory_node()
  {
     /*
      * Create a second memory node to store the ranges covering
      * reserved-memory regions.
      */
     res = make_memory_node(d, kinfo->fdt, addrcells, sizecells,
                            &bootinfo.reserved_mem);
     if ( res )
         return res;

     ...
  }

## Fixes

I think it's wrong to add the reserved memory regions into the DT
binding as normal memory nodes for Dom0 kernel.  On the other hand, we
cannot simply remove these reserved memory regions and don't pass to
Dom0 kernel - we might reserve memory for specific purpose, for example,
ramoops [1] for kernel debugging.

The right thing to do is to keep these reserved memory nodes to Dom0
kernel.  So one task is to record properties for these reserved memory
node name and properties and pass to Dom0 kernel.

The difficulty is how we can avoid allocate these reserved memory
regions in Xen hypervisor.  We cannot register the reserved memory
into the boot pages, otherwise, the reserved memory might be allocated
in the early phase.  But we need to register these pages into the
frame management framework and reserve them in the first place, so
that we can allow Dom0 kernel to use them.  (I checked a bit the static
memory mechanism, seems to me we cannot use it to resolve this issue).

Before I proceed, I'd like to check if Xen community has related
discussion or not?  Or any suggestions or input will be appreciate!

Leo

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/hisilicon/hi6220-hikey.dts#n54