Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0

[Andy Smith <andy@xxxxxxxxxxxxxx>]

Hi Klaus,

On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> When using pygrub the error is:
> xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD 
> decompress support unavailable
> 
> So it seems that pygrub does not support the ZSTD compressed kernel.

pygrub extracts a kernel and initrd from the guest block device and
then Xen direct boots it. Your Xen does not support zstd-compressed
kernels for direct boot so that doesn't work.

You can use the extract-vmlinux script to decompress a guest kernel
to vmlinux and direct boot that. It isn't hard to bodge a test into
pygrub for zstd (and lz4) compressed kernels and automatically call
extract-vmlinux on them, then boot the resulting vmlinux.

However, pygrub is a really really bad way to boot guests, so you
would be much better off figuring out how to stop using it
altogether. Maybe now is a good time for that, since you're going to
have to make some changes anyway?

TL;DR: PVH mode is probably your saviour as long as you don't need
to boot anything older than kernel version 4.20 or so.

> So I tried to boot the domU with pvgrub2 by setting:
> kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"

pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
to be abandoned in that no one is interested in adding such support
to it.

Again you can lash something together that decompresses the kernels
first. This time a bit different since it needs to run in guest
context due to pvgrub. Here's something I came up with to make the
guest decompress its kernel when the kernel is installed:

    https://github.com/bitfolk/decompress-lz4-kernels

> https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states 
> "Support for zstd-compressed dom0 (x86) and domU kernels.".
> 
> So I need some advice/confirmations:
> Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add 
> ZSTD support to pygrub?

pygrub doesn't boot the kernel, Xen does, so if your Xen supports
direct booting LZ4/zstd then that will work. It's not a question of
what pygrub supports as pygrub does not parse or manipulate the
kernel file in any way, it just passes it to Xen.

However you'd still be using pygrub, which is bad.

> If pygrub will not support ZSTD, what are my options? Should it
> work with pvgrub2 and I am doing something wrong? How to correctly
> use pvgrub2.

pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
there seems to be no interest in supporting that.

> If pvgrub2 does not support ZSTD, what are my options? Would it
> help to move from PV to PVH,PVHVM,...?

Yes. Every guest that uses LZ4 or zstd kernels also supports being
booted in PVH mode, so I switched to PVH for every guest that
supports it and stopped using my extract kernel hack that was
necessary for getting pvgrub to boot them.

One of the other reasons why pvhgrub is better is that it uses the
standard parts of grub for loading images, so if grub supports the
compression format then pvhgrub does too. So no problem loading
lz4/zstd compressed kernels, as long as the grub package you started
with is new enough.

> Can I automate building
> domUs similar to xen-create-image also with non-PV domUs?

As far as I know xen-tools and xen-create-image are also dead
upstream so it would be best not to rely on them. Maybe a project
for a later time though, as I can't see any reason why something
that xen-create-image makes would not boot under PVH mode.

For Debian/Ubuntu-based guests you can just boot the installer
kernel/initrd in PVH mode and run through a normal text mode
install. Or you can use debootstrap from your dom0 to download
everything you need into a directory. Which is basically what
xen-create-image is doing.

(This all changed with Ubuntu >= 21.10 as they have moved away from
the Debian installer. For newer Ubuntu it's going to be either
debootstrap or download the Ubuntu cloud image and boot that (with
optional cloud-init for any configuration you need).)

Cheers,
Andy