Removing Linux memory hotplug limitations

[Demi Marie Obenour <demi@xxxxxxxxxxxxxxxxxxxxxx>]

Qubes OS is trying to switch from relying on populate-on-demand to
memory hotplug for Linux guests.  However, this runs into a problem,
which is that only a limited amount of memory can be hotplugged.

My experiments with Qubes OS’s build of Linux 6.3.2 reveal:

- The more memory a guest starts off with, the more memory that can be
  added to it via hotplug.

- The memory that the guest is not able to use remains available on the
  host and can be assigned to dom0 (and, presumably, to oother guests).

- There is no sudden jump at 2GiB or 3GiB as far as I can tell.

- There are no kernel warning messages unless I try to add a huge amount
  of memory (far beyond what can be successfully hotplugged).  In
  particular, there are no warning messages from drivers/xen/balloon.c.

- There are several waits in the balloon driver that should probably
  hvae comments added.

- `cat /sys/devices/system/memory/memory*/state` reveals that all memory
  devices are online.

- `echo $((1 << 63)) | sudo tee 
/sys/devices/system/xen_memory/xen_memory0/target`
  causes a kernel crash (BUG_ON(ret != nr_pages) in drivers/xen/balloon.c).
  Patch coming.

- The initial amount of memory assigned to a guest is irrelevant.

- The maximum amount of memory assigned to a guest is highly relevant.
  Table below:

   Initial maximum memory:      Maximum after hotplug
             400M                       2733M 
             500M                       3067M 
             600M                       3535M
             700M                       3919M
             800M                       4315
             900m                       4711
             1000m                      5105

SciPy linear regression gives:

- Slope: 3.9943
- Y-Intercept: 1116.14
- R: 0.99965
- stderr: 0.047

In short, there is a very clear, nearly linear relationship between the
amount of cold-plugged memory and the amount of memory that can be
hotplugged later.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature