Re: Xen DomU Bootloader Experiences

[Elliott Mitchell <ehem+xen@xxxxxxx>]

On Thu, Nov 13, 2025 at 07:46:25AM +0100, Juergen Gross wrote:
> On 12.11.25 22:13, Elliott Mitchell wrote:
> > A few times there have been mentions of a need to choose between boot
> > methods for DomUs.  There is a need to decide on ones to recommend and
> > put effort into supportting.  I may not have tried that many nor done
> > particularly great amounts of experimentation, but I do have some
> > experience with multiple User Domain bootloaders.
> > 
> > PyGRUB
> > Xen's bootloader.  PyGRUB is quite functional within its limits.  In
> > particular it simulates the domain's environment in Domain 0.  This means
> > the security exposure is problematic.  Another big concern is that it
> > only does GRUB v1 syntax.  For a long while Debian had a package for
> > generating those files on a modern system, but that package was dropped.
> > 
> > Yet PyGRUB does avoid needing to use external tools to retrieve the
> > kernel.  If the kernel is updated inside the domain, this does get the
> > new kernel.  Further being architecture-independent this works on x86,
> > ARM*, RISC-V and PowerPC.
> > 
> > As it is the only GRUB-flavor loader available on ARM*, that is the only
> > place where I've used PyGRUB.
> 
> There is one further advantage for PyGRUB: it can look into the kernel
> _before_ the domU is being created, so it can tell Xen tools whether a
> 32- or 64-bit domU is needed based on the selected kernel.
> 
> This is the main reason why PyGRUB is still existing.

Okay.

While I can believe this is a big advantage in some environments, I
suspect this is rarely used.  Whereas the security profile of PyGRUB is
troublesome,  GRUBv1 syntax nearly overwhelms all advantages by itself.
Do you think changing the default to not build/install on x86 is
inappropriate?

I suspect Tianocore/EDK2's 32/64-bit support might be a better choice for
an environment where this matters.


> > PvGRUB
> > I'm sure nearly everyone knows about PvGRUB.  By being a proper port of
> > GRUB to run directly on Xen, it overcomes PyGRUB's disadvantages.  The
> > one disadvantage is needing to get patches into an external project for
> > changes in Xen.
> > 
> > Two changes to Xen urgently need propogation to PvGRUB.  I'm unsure
> > whether PvGRUB unmaps its mapping of vcpu_info data.  The second is
> > needing to work on ARM*, RISC-V and PowerPC.  The latter is the one and
> > only way in which PvGRUB is inferior to PyGRUB.
> > 
> > As PvGRUB is only available for x86, that is the only place I've used
> > PvGRUB.
> 
> Naming is difficult. :-)
> 
> You are talking about grub-pv. pv-grub is the Xen-internal variant based
> on Mini-OS and legacy grub 0.97, supporting grub for PV-domUs.
> 
> grub-pv comes basically in three flavors, all x86-only:
> 
> - for 32-bit PV-guests
> - for 64-bit PV-guests
> - for PVH-guests (32- or 64-bit)
> 
> Adding PVH support to upstream Grub for e.g. Arm should be rather easy.

I figured as much, yet this has not yet been done.  The other two
implementations using GRUBv1 syntax is a severe weakness.  I rate this as
high priority since GRUBv2 syntax is rather valuable.  I would try to
lay groundwork for RISC-V and PowerPC too.


> > EDK2/Tianocore
> > Quite well-known for being the basis of most x86 firmwares, plus being
> > part of a typical Qemu setup.  Not nearly as well known for being a Xen
> > DomU bootloader.
> > 
> > When it was working you would build their ArmVirtXen.dsc file and get
> > XEN_EFI.fd as output.  You would then use XEN_EFI.fd for the domain's
> > kernel.  If you looked at the console you saw something which looked and
> > acted pretty similar to a UEFI firmware on x86 machines.  This was
> > extremely functional for OSes which didn't particularly like GRUB.
> > Notably I've read of it being able to load a Redmond OS and it was quite
> > functional for booting an ARM64 port of FreeBSD.
> > 
> > Sometime after November 16th, 2022 or commit fff6d81270.  The built
> > images stopped functioning.  This is actually rather concerning since it
> > may also effects firmwares built for x86 HVM domains.  I don't presently
> > know whether there are multiple bugs, or a single one effecting all Xen
> > builds.
> > 
> > There is also an urgent need to get EDK2/Tianocore updated to match
> > Xen/ARM's disallowing mapping the shared information page multiple times.
> > As I did not wish to become deeply involved with EDK2/Tianocore I sent a
> > patch to xen-devel close to 1.5 years ago.  Lack of action suggests there
> > is an urgent need for a liason.
> > 
> > 
> > 
> > Recommendations:
> > PyGRUB is functional within its limits.  Problems are GRUBv1 syntax and
> > running within Domain 0.  Given this I feel the Xen Project should be
> > heading towards deprecating PyGRUB.  Since PvGRUB works for x86 now, I
> > would default to neither building nor installing PyGRUB on x86.  For
> > other architectures PyGRUB is still useful.
> > 
> > The Xen Project should formally ask the GRUB Project to port PvGRUB to
> > ARM, RISC-V and PowerPC.  The need for PvGRUB on ARM seems rather urgent.
> > Without a proper bootloader VMs aren't too useful.
> 
> Well, I did the grub-pvh implementation.
> 
> Doing that for other architectures shouldn't be rocket science. :-)

Okay.  I've seen some it written about a number of times and no action.


My reading had been the people most knowledgeable about adapting the
tools were unsure of how to direct effort.  Proper GRUBv2 is high-value
for Linux.  Tianocore/EDK2 is valuable for everyone, high value for non-
Linux.  Worst case Tianocore/EDK2 can boot GRUB-UEFI, I doubt GRUB can
boot Tianocore/EDK2.

Is there any scenario not adaquately addressed by trying for that pair?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445