[Xen-devel] Problems booting 32-bit PV; just me or more widespread?

[Andy Smith <andy@xxxxxxxxxxxxxx>]

Hi,

I'm sorry this is a long email, but I wanted to explain everything
that I have tried, because it seems like quite a few different
versions of 32-bit upstream Linux kernel no longer boot as PV guest
and I'm surprised I am the first to encounter this. Probably I
have done something wrong.

I cannot get any of the Ubuntu packaged 32-bit mainline kernels
after v4.13.16 that are found at
http://kernel.ubuntu.com/~kernel-ppa/mainline/ to boot in 32-bit PV
mode. All of them from v4.14.0rc1 onwards crash on xl create either
saying "error: no XEN note found." or else immediately producing a
kernel panic like:

.
.
.
[ 0.114370] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, 
max_idle_ns: 7645041785100000 ns
[ 0.114382] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 0.114423] pinctrl core: initialized pinctrl subsystem
[ 0.134326] RTC time: 165:165:165, date: 165/165/65
[ 0.134442] NET: Registered protocol family 16
[ 0.134457] xen:grant_table: Grant tables using version 1 layout
[ 0.134502] Grant table initialized
[ 0.134544] audit: initializing netlink subsys (disabled)
[ 0.134611] audit: type=2000 audit(1535307799.132:1): state=initialized 
audit_enabled=0 res=1
[ 0.134678] EISA bus registered
[ 0.136019] PCI: setting up Xen PCI frontend stub
[ 0.136073] BUG: unable to handle kernel paging request at edc21fd9
[ 0.136084] IP: eisa_bus_probe+0x19/0x36
[ 0.136089] *pdpt = 0000000001ee6027 *pde = 0000000029cc6067 *pte = 
0000000000000000
[ 0.136100] Oops: 0000 [#1] SMP
[ 0.136105] Modules linked in:
[ 0.136111] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-33-generic 
#36-Ubuntu
[ 0.136120] EIP: eisa_bus_probe+0x19/0x36
[ 0.136125] EFLAGS: 00010246 CPU: 0
[ 0.136130] EAX: edc21fd9 EBX: 00000000 ECX: 01e0d000 EDX: 00000200
[ 0.136138] ESI: c1d0d452 EDI: c1dd34a4 EBP: e9c89f24 ESP: e9c89f24
[ 0.136145] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: e021
[ 0.136154] CR0: 80050033 CR2: edc21fd9 CR3: 01e10000 CR4: 00042660
[ 0.136166] Call Trace:
[ 0.136173] do_one_initcall+0x49/0x174
[ 0.136179] ? parse_args+0x143/0x390
[ 0.136187] ? set_debug_rodata+0x14/0x14
[ 0.136193] kernel_init_freeable+0x149/0x1c5
[ 0.136201] ? rest_init+0xa0/0xa0
[ 0.136207] kernel_init+0xd/0xf0
[ 0.136213] ret_from_fork+0x2e/0x38
[ 0.140000] Code: ff b8 df 43 ae c1 e8 35 1b 88 ff e8 20 12 88 ff c9 c3 3e 8d 
74 26 00 55 b9 04 00 00 00 31 d2 b8 d9 ff 0f 00 89 e5 e8 35 8d 35 ff <8b> 10 81 
fa 45 49 53 41 75 0a c7 05 a0 76 ed c1 01 00 00 00 e8
[ 0.140000] EIP: eisa_bus_probe+0x19/0x36 SS:ESP: e021:e9c89f24
[ 0.140000] CR2: 00000000edc21fd9
[ 0.140000] ---[ end trace 8c00b3cb7d4f06ba ]---
[ 0.140013] Kernel panic - not syncing: Attempted to kill init! 
exitcode=0x00000009

(that one was from the currently-packaged linux-image-generic in
Ubuntu 18.04 LTS).

amd64 kernels work no problem.

On the host I'm running Xen 4.10.1 with up to date XSAs.

The guests are booted by either 32-bit or 64-bit pvgrub2 but to take
that out of the picture I copied the kernels and initramfs files to
the host and tried direct kernel boot. In that case I get:

$ sudo xl create -c /var/tmp/debtest1.conf
Parsing config from /var/tmp/debtest1.conf
xc: error: panic: xc_dom_core.c:702: xc_dom_find_loader: no loader found: 
Invalid kernel
libxl: error: libxl_dom.c:718:libxl__build_dom: xc_dom_parse_image failed: No 
such file or directory
libxl: error: libxl_create.c:1264:domcreate_rebuild_done: Domain 43:cannot 
(re-)build domain: -3
libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain 43:Non-existant 
domain
libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain 43:Unable to 
destroy guest
libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain 43:Destruction of 
domain failed

The "No such file or directory" is strange as the file does exist at
the correct path. I can only think that the previous "Invalid
kernel" causes that.

I then began to wonder what the situation was like in Debian.

32-bit Debian stable works fine, but that's a 4.9.x kernel. That
works both from pvgrub2 and from direct kernel boot. A kernel based
on 4.9.17 is available in stretch-backports so I tried that
(linux-image-4.17.0-0.bpo.3-686). That behaves as above (either "No
XEN note" from pvgrub2 or "Invalid kernel" from xl create).

I then used the procedure described in the Debian Kernel Handbook
(https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-kernel-org-package)
to build a kernel from
https://git.kernel.org/torvalds/t/linux-4.19-rc1.tar.gz. That also
fails to boot in the same way.

So, my investigations so far suggest that everything from v4.14.0
onwards will not boot as a 32-bit PV guest. Is this a known issue,
or did I do something wrong? As I say, if it were an upstream kernel
issue then I am surprised I am the first to encounter it, but
perhaps the use of more modern kernel versions and 32-bit PV is
quite low.

If this is not a known issue then perhaps I can git bisect to find
what broke this, or perform any other debugging you'd like me to
try.

I started looking into this because I have a user who was running
32-bit Ubuntu 16.04 LTS. They did a release upgrade to 18.04 from
their working guest, and then found their new kernel did not boot. I
confirmed the behaviour as above and also realised that I had in
fact never tested 32-bit Ubuntu 18.04, only 64-bit, so at the moment
there is no packaged 32-bit kernel in Ubuntu 18.04 that is working
for my setup, and possibly everyone else's. The user has since
reinstalled their guest with 64-bit.

Thanks,
Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel