AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0

[Klaus Darilion <klaus.darilion@xxxxxx>]

Just for the records: meanwhile I can also boot 22.04 domUs with 22.04 dom0.
The problem was a build-bug in the Ubuntu Xen package. You can find the details 
here:
https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1956166

regards
Klaus

> -----Ursprüngliche Nachricht-----
> Von: Klaus Darilion
> Gesendet: Montag, 19. September 2022 22:45
> An: xen-users@xxxxxxxxxxxxxxxxxxxx
> Betreff: AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> dom0
> 
> Sorry for the late repsonse. I just found some time for testing.
> 
> Thanks for the detailed answer - that made things much clearer.
> 
> As suggested by you and others: using PVH instead of PV solved the problem.
> 
> Thanks
> Klaus
> 
> > -----Ursprüngliche Nachricht-----
> > Von: Xen-users <xen-users-bounces@xxxxxxxxxxxxxxxxxxxx> Im Auftrag von
> > Andy Smith
> > Gesendet: Mittwoch, 27. April 2022 23:54
> > An: xen-users@xxxxxxxxxxxxxxxxxxxx
> > Betreff: Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> > dom0
> >
> > Hi Klaus,
> >
> > On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> > > When using pygrub the error is:
> > > xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD
> > decompress support unavailable
> > >
> > > So it seems that pygrub does not support the ZSTD compressed kernel.
> >
> > pygrub extracts a kernel and initrd from the guest block device and
> > then Xen direct boots it. Your Xen does not support zstd-compressed
> > kernels for direct boot so that doesn't work.
> >
> > You can use the extract-vmlinux script to decompress a guest kernel
> > to vmlinux and direct boot that. It isn't hard to bodge a test into
> > pygrub for zstd (and lz4) compressed kernels and automatically call
> > extract-vmlinux on them, then boot the resulting vmlinux.
> >
> > However, pygrub is a really really bad way to boot guests, so you
> > would be much better off figuring out how to stop using it
> > altogether. Maybe now is a good time for that, since you're going to
> > have to make some changes anyway?
> >
> > TL;DR: PVH mode is probably your saviour as long as you don't need
> > to boot anything older than kernel version 4.20 or so.
> >
> > > So I tried to boot the domU with pvgrub2 by setting:
> > > kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
> >
> > pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
> > to be abandoned in that no one is interested in adding such support
> > to it.
> >
> > Again you can lash something together that decompresses the kernels
> > first. This time a bit different since it needs to run in guest
> > context due to pvgrub. Here's something I came up with to make the
> > guest decompress its kernel when the kernel is installed:
> >
> >     https://github.com/bitfolk/decompress-lz4-kernels
> >
> > > https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states
> > "Support for zstd-compressed dom0 (x86) and domU kernels.".
> > >
> > > So I need some advice/confirmations:
> > > Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to 
> > > add
> > ZSTD support to pygrub?
> >
> > pygrub doesn't boot the kernel, Xen does, so if your Xen supports
> > direct booting LZ4/zstd then that will work. It's not a question of
> > what pygrub supports as pygrub does not parse or manipulate the
> > kernel file in any way, it just passes it to Xen.
> >
> > However you'd still be using pygrub, which is bad.
> >
> > > If pygrub will not support ZSTD, what are my options? Should it
> > > work with pvgrub2 and I am doing something wrong? How to correctly
> > > use pvgrub2.
> >
> > pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
> > there seems to be no interest in supporting that.
> >
> > > If pvgrub2 does not support ZSTD, what are my options? Would it
> > > help to move from PV to PVH,PVHVM,...?
> >
> > Yes. Every guest that uses LZ4 or zstd kernels also supports being
> > booted in PVH mode, so I switched to PVH for every guest that
> > supports it and stopped using my extract kernel hack that was
> > necessary for getting pvgrub to boot them.
> >
> > One of the other reasons why pvhgrub is better is that it uses the
> > standard parts of grub for loading images, so if grub supports the
> > compression format then pvhgrub does too. So no problem loading
> > lz4/zstd compressed kernels, as long as the grub package you started
> > with is new enough.
> >
> > > Can I automate building
> > > domUs similar to xen-create-image also with non-PV domUs?
> >
> > As far as I know xen-tools and xen-create-image are also dead
> > upstream so it would be best not to rely on them. Maybe a project
> > for a later time though, as I can't see any reason why something
> > that xen-create-image makes would not boot under PVH mode.
> >
> > For Debian/Ubuntu-based guests you can just boot the installer
> > kernel/initrd in PVH mode and run through a normal text mode
> > install. Or you can use debootstrap from your dom0 to download
> > everything you need into a directory. Which is basically what
> > xen-create-image is doing.
> >
> > (This all changed with Ubuntu >= 21.10 as they have moved away from
> > the Debian installer. For newer Ubuntu it's going to be either
> > debootstrap or download the Ubuntu cloud image and boot that (with
> > optional cloud-init for any configuration you need).)
> >
> > Cheers,
> > Andy