[BUG] Unable to boot Xen 4.11 (shipped with Ubuntu) on Intel 10i3 CPU

[Ondrej Balaz <blami@xxxxxxxxx>]

Hi,
I recently updated my home server running Ubuntu 20.04 (Focal) with Xen hypervisor 4.11 (installed using Ubuntu packages). Before the upgrade all was running fine and both dom0 and all domUs were booting fine. Upgrade was literally taking harddrive from 6th gen Intel CPU system to 10th gen Intel CPU one and redoing EFI entries from Ubuntu live USB.

After doing so standalone Ubuntu (without Xen multiboot) boots just fine but Ubuntu as dom0 with Xen fails pretty early on with following error (hand-copied from phone snaps I took with loglvl=all as this is barebone system without serial port and I don't know how to dump full logs in case of panic):

(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[01])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override 

(XEN) ACPI: IRQ9 used by override
(XEN) Enabling APIC mode: Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) ERST table was not found
(XEN) ACPI: BGRT: invalidating v1 image at 0x7d7c1018
(XEN) Using ACPI (MADT) for SMP configuration information
...
(XEN) Switched to APIC driver x2apic_cluster
...  
(XEN) Initing memory sharing.

(XEN) alt table ffff82d08042b840 -> ffff82d08042d7ce
...
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.

(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control not enabled 
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled 
(XEN) Intel VT-d Queued Invalidation enabled 
(XEN) Intel VT-d Interrupt Remapping enabled
(XEN) Intel VT-d Posted Interrupt not enabled  
(XEN) Intel VT-d Shared EPT tables enabled
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed.
(XEN) ...trying to set up timer as Virtual Wire IRQ... failed.
(XEN) ...trying to set up timer as ExtINT IRQ...spurious 8259A interrupt IRQ7.
(XEN) CPU0: No irq handler for vector e7 (IRQ -8)
(XEN) IRQ7 a=0001[0001,0000] v=60[ffffffff] t=IO-APIC-edge s=00000002
(XEN)  failed :(.
(XEN)
(XEN) *******************************
(XEN) Panic on CPU 0:
(XEN) IO-APIC + timer doesn't work!  Boot with apic_verbosity=debug and send report.  Then try booting with the `noapic` option
(XEN) *******************************

I suspected that migration of drive could cause problem so I took an empty SSD and installed fresh Ubuntu and added Xen hypervisor, after reboot I ended up with same panic. I tried booting with noapic (gave general page fault) and iommu=0 (said it needs iommu=required/force). Trying to boot this exact fresh install on older (6th gen) Intel CPU succeeded. I happen to have access to one more system with 10th gen Intel CPUs (Lenovo laptop) and no luck booting Xen there too and same panic in the end.

Back to my barebone I tried to match BIOS settings between working and non-working but it didn't help. Virtualization is enabled, both systems are from the same maker (Intel NUC barebones), both systems are EFI enabled/secure boot disabled (the later one doesn't seem to have an option to disable EFI boot and boot using MBR).

Is this something known? Are there any boot options that can potentially fix this?

Any help (including how to dump full Xen boot logs without serial) appreciated.

Thanks,
Ondrej