Xen DomU Bootloader Experiences

[Elliott Mitchell <ehem+xen@xxxxxxx>]

A few times there have been mentions of a need to choose between boot
methods for DomUs.  There is a need to decide on ones to recommend and
put effort into supportting.  I may not have tried that many nor done
particularly great amounts of experimentation, but I do have some
experience with multiple User Domain bootloaders.

PyGRUB
Xen's bootloader.  PyGRUB is quite functional within its limits.  In
particular it simulates the domain's environment in Domain 0.  This means
the security exposure is problematic.  Another big concern is that it
only does GRUB v1 syntax.  For a long while Debian had a package for
generating those files on a modern system, but that package was dropped.

Yet PyGRUB does avoid needing to use external tools to retrieve the
kernel.  If the kernel is updated inside the domain, this does get the
new kernel.  Further being architecture-independent this works on x86,
ARM*, RISC-V and PowerPC.

As it is the only GRUB-flavor loader available on ARM*, that is the only
place where I've used PyGRUB.


PvGRUB
I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
one disadvantage is needing to get patches into an external project for
changes in Xen.

Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
only way in which PvGRUB is inferior to PyGRUB.

As PvGRUB is only available for x86, that is the only place I've used
PvGRUB.


EDK2/Tianocore
Quite well-known for being the basis of most x86 firmwares, plus being
part of a typical Qemu setup.  Not nearly as well known for being a Xen
DomU bootloader.

When it was working you would build their ArmVirtXen.dsc file and get
XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
kernel.  If you looked at the console you saw something which looked and
acted pretty similar to a UEFI firmware on x86 machines.  This was
extremely functional for OSes which didn't particularly like GRUB.
Notably I've read of it being able to load a Redmond OS and it was quite
functional for booting an ARM64 port of FreeBSD.

Sometime after November 16th, 2022 or commit fff6d81270.  The built
images stopped functioning.  This is actually rather concerning since it
may also effects firmwares built for x86 HVM domains.  I don't presently
know whether there are multiple bugs, or a single one effecting all Xen
builds.

There is also an urgent need to get EDK2/Tianocore updated to match
Xen/ARM's disallowing mapping the shared information page multiple times.
As I did not wish to become deeply involved with EDK2/Tianocore I sent a
patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
is an urgent need for a liason.



Recommendations:
PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
running within Domain 0.  Given this I feel the Xen Project should be
heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
would default to neither building nor installing PyGRUB on x86.  For
other architectures PyGRUB is still useful.

The Xen Project should formally ask the GRUB Project to port PvGRUB to
ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
Without a proper bootloader VMs aren't too useful.


The Xen Project needs people to work with EDK2/Tianocore.  The oldest
report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
years later the bug is still present.

The ability to configure XEN_EFI.fd as a domain kernel is a feature
highly worthy of being ported to x86.  For OSes which don't particularly
like GRUB, but do have PV drivers this is an ideal boot method.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445