Re: Xen DomU Bootloader Experiences

[Juergen Gross <jgross@xxxxxxxx>]

On 12.11.25 22:13, Elliott Mitchell wrote:
> A few times there have been mentions of a need to choose between boot
> methods for DomUs.  There is a need to decide on ones to recommend and
> put effort into supportting.  I may not have tried that many nor done
> particularly great amounts of experimentation, but I do have some
> experience with multiple User Domain bootloaders.
>
> PyGRUB
> Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> particular it simulates the domain's environment in Domain 0.  This means
> the security exposure is problematic.  Another big concern is that it
> only does GRUB v1 syntax.  For a long while Debian had a package for
> generating those files on a modern system, but that package was dropped.
>
> Yet PyGRUB does avoid needing to use external tools to retrieve the
> kernel.  If the kernel is updated inside the domain, this does get the
> new kernel.  Further being architecture-independent this works on x86,
> ARM*, RISC-V and PowerPC.
>
> As it is the only GRUB-flavor loader available on ARM*, that is the only
> place where I've used PyGRUB.
There is one further advantage for PyGRUB: it can look into the kernel
_before_ the domU is being created, so it can tell Xen tools whether a
32- or 64-bit domU is needed based on the selected kernel.

This is the main reason why PyGRUB is still existing.

> PvGRUB
> I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> one disadvantage is needing to get patches into an external project for
> changes in Xen.
>
> Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> only way in which PvGRUB is inferior to PyGRUB.
>
> As PvGRUB is only available for x86, that is the only place I've used
> PvGRUB.
Naming is difficult. :-)

You are talking about grub-pv. pv-grub is the Xen-internal variant based
on Mini-OS and legacy grub 0.97, supporting grub for PV-domUs.

grub-pv comes basically in three flavors, all x86-only:

- for 32-bit PV-guests
- for 64-bit PV-guests
- for PVH-guests (32- or 64-bit)

Adding PVH support to upstream Grub for e.g. Arm should be rather easy.

> EDK2/Tianocore
> Quite well-known for being the basis of most x86 firmwares, plus being
> part of a typical Qemu setup.  Not nearly as well known for being a Xen
> DomU bootloader.
>
> When it was working you would build their ArmVirtXen.dsc file and get
> XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> kernel.  If you looked at the console you saw something which looked and
> acted pretty similar to a UEFI firmware on x86 machines.  This was
> extremely functional for OSes which didn't particularly like GRUB.
> Notably I've read of it being able to load a Redmond OS and it was quite
> functional for booting an ARM64 port of FreeBSD.
>
> Sometime after November 16th, 2022 or commit fff6d81270.  The built
> images stopped functioning.  This is actually rather concerning since it
> may also effects firmwares built for x86 HVM domains.  I don't presently
> know whether there are multiple bugs, or a single one effecting all Xen
> builds.
>
> There is also an urgent need to get EDK2/Tianocore updated to match
> Xen/ARM's disallowing mapping the shared information page multiple times.
> As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> is an urgent need for a liason.
>
>
>
> Recommendations:
> PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> running within Domain 0.  Given this I feel the Xen Project should be
> heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> would default to neither building nor installing PyGRUB on x86.  For
> other architectures PyGRUB is still useful.
>
> The Xen Project should formally ask the GRUB Project to port PvGRUB to
> ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> Without a proper bootloader VMs aren't too useful.
Well, I did the grub-pvh implementation.

Doing that for other architectures shouldn't be rocket science. :-)

> The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> years later the bug is still present.
>
> The ability to configure XEN_EFI.fd as a domain kernel is a feature
> highly worthy of being ported to x86.  For OSes which don't particularly
> like GRUB, but do have PV drivers this is an ideal boot method.

Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature