Re: Xen DomU Bootloader Experiences

["Teddy Astie" <teddy.astie@xxxxxxxxxx>]

Le 12/11/2025 à 22:16, Elliott Mitchell a écrit :
> A few times there have been mentions of a need to choose between boot
> methods for DomUs.  There is a need to decide on ones to recommend and
> put effort into supportting.  I may not have tried that many nor done
> particularly great amounts of experimentation, but I do have some
> experience with multiple User Domain bootloaders.
>
> PyGRUB
> Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> particular it simulates the domain's environment in Domain 0.  This means
> the security exposure is problematic.  Another big concern is that it
> only does GRUB v1 syntax.  For a long while Debian had a package for
> generating those files on a modern system, but that package was dropped.
>
> Yet PyGRUB does avoid needing to use external tools to retrieve the
> kernel.  If the kernel is updated inside the domain, this does get the
> new kernel.  Further being architecture-independent this works on x86,
> ARM*, RISC-V and PowerPC.
>
> As it is the only GRUB-flavor loader available on ARM*, that is the only
> place where I've used PyGRUB.
>

To be fair, I don't really like PyGRUB, it hasn't really worked well for
me (perhaps I'm trying too modern distros ?). It's still what's being
used for XAPI, which doesn't make things practical and even sometimes
brittle.

I'm more tending to the other alternatives.

>
> PvGRUB
> I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> one disadvantage is needing to get patches into an external project for
> changes in Xen.
>
> Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> whether PvGRUB unmaps its mapping of vcpu_info data.  The second is

I don't think it maps vcpu_info, otherwise, most (all ?) Linux kernels
will immediately break.

> needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> only way in which PvGRUB is inferior to PyGRUB.
>

Perhaps it could be a good idea; although I'm more tending to EDK2
instead which works well with it and cover more cases (unless EDK2 stops
working for some obscure reason). For a more long-term solution, I would
prefer something different to GRUB but basing on "Boot Loader
Specification", maybe Edera's Sprout [1] or rust-hypervisor-firmware [2]
as a PVH firmware.

> As PvGRUB is only available for x86, that is the only place I've used
> PvGRUB.
>

>
> EDK2/Tianocore
> Quite well-known for being the basis of most x86 firmwares, plus being
> part of a typical Qemu setup.  Not nearly as well known for being a Xen
> DomU bootloader.
>
> When it was working you would build their ArmVirtXen.dsc file and get
> XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> kernel.  If you looked at the console you saw something which looked and
> acted pretty similar to a UEFI firmware on x86 machines.  This was
> extremely functional for OSes which didn't particularly like GRUB.
> Notably I've read of it being able to load a Redmond OS and it was quite
> functional for booting an ARM64 port of FreeBSD.
>
> Sometime after November 16th, 2022 or commit fff6d81270.  The built
> images stopped functioning.  This is actually rather concerning since it
> may also effects firmwares built for x86 HVM domains.  I don't presently
> know whether there are multiple bugs, or a single one effecting all Xen
> builds.
>

Actually, you can use it for x86 like you do for ARM, with PVH (using
the same binary as for HVM). But it broke at some point, and I'm mostly
living with a older but working custom built version of it.

We definitely want to sort-out this, as it is going to be a blocker for
providing better foundations for PVH guests on XAPI/XCP-ng.

> There is also an urgent need to get EDK2/Tianocore updated to match
> Xen/ARM's disallowing mapping the shared information page multiple times.
> As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> is an urgent need for a liason.
>
>
>
> Recommendations:
> PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> running within Domain 0.  Given this I feel the Xen Project should be
> heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> would default to neither building nor installing PyGRUB on x86.  For
> other architectures PyGRUB is still useful.
>

+1. At least making it deprecated, and at the same time improving the
ergonomics of alternatives (i.e PVH+EDK2, legacy grub-pv and/or "PV-GRUB").

> The Xen Project should formally ask the GRUB Project to port PvGRUB to
> ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> Without a proper bootloader VMs aren't too useful.
>

EDK2 / U-Boot should be able to fill this niche at least for ARM; not
sure about RISC-V though, but I guess it is going to be a similar story.

Unsure about PowerPC through.

>
> The Xen Project needs people to work with EDK2/Tianocore.  The oldest
> report I've seen of the EDK2/Tianocore issue dates to mid-2023.  Now two
> years later the bug is still present.
>

> The ability to configure XEN_EFI.fd as a domain kernel is a feature
> highly worthy of being ported to x86.  For OSes which don't particularly
> like GRUB, but do have PV drivers this is an ideal boot method.
>

AFAICT it is already there :)

>

[1] https://github.com/edera-dev/sprout
[2] https://github.com/cloud-hypervisor/rust-hypervisor-firmware


--
Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech