[xen_netfront] - Xen PVH VM: kernel upgrade 6.9.10 > 6.10.7 results in crash

[Arthur Borsboom <arthurborsboom@xxxxxxxxx>]

After upgrading kernel 6.9.10 to 6.10.7 all Xen PVH VM's became unavailable.

Downgrading the kernel back to 6.9.10 makes the VM's work again.

Snippet stack trace + kernel logs (good and bad) attached.

Sep 01 08:59:21 web3.aramgroup.com kernel: xen_netfront: Initialising Xen virtual ethernet driver
Sep 01 08:59:21 web3.aramgroup.com systemd-udevd[248]: vfb-0: Worker [250] terminated by signal 9 (KILL).
Sep 01 08:59:21 web3.aramgroup.com kernel: BUG: kernel NULL pointer dereference, address: 0000000000000060
Sep 01 08:59:21 web3.aramgroup.com kernel: #PF: supervisor read access in kernel mode
Sep 01 08:59:21 web3.aramgroup.com kernel: #PF: error_code(0x0000) - not-present page
Sep 01 08:59:21 web3.aramgroup.com kernel: PGD 0 P4D 0
Sep 01 08:59:21 web3.aramgroup.com kernel: Oops: Oops: 0000 [#1] PREEMPT SMP PTI
Sep 01 08:59:21 web3.aramgroup.com kernel: CPU: 0 PID: 250 Comm: (udev-worker) Not tainted 6.10.7-arch1-1 #1 2b2df360fbb0436393dc89f6589e9eeea2964ecb
Sep 01 08:59:21 web3.aramgroup.com kernel: RIP: 0010:video_is_primary_device+0x9/0x40
Sep 01 08:59:21 web3.aramgroup.com kernel: Code: 48 89 d8 5b c3 cc cc cc cc 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 <48> 81 7f 60 80 e3 54 90 74 07 31 c0 c3 cc cc cc cc 53 48 89 fb 48
Sep 01 08:59:21 web3.aramgroup.com kernel: RSP: 0000:ffffbb06808d7a60 EFLAGS: 00010246
Sep 01 08:59:21 web3.aramgroup.com kernel: RAX: 0000000000000000 RBX: ffff90ca41367800 RCX: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: RBP: 0000000000000000 R08: 0000000000000060 R09: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: R10: ffffbb06808d7a78 R11: 0000000000000006 R12: ffffbb06808d7a90
Sep 01 08:59:21 web3.aramgroup.com kernel: R13: ffff90ca41367a88 R14: ffff90ca41367a60 R15: ffff90cb41330788
Sep 01 08:59:21 web3.aramgroup.com kernel: FS:  000072bfd74c0880(0000) GS:ffff90ce33a00000(0000) knlGS:0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 01 08:59:21 web3.aramgroup.com kernel: CR2: 0000000000000060 CR3: 0000000001326002 CR4: 00000000003706f0
Sep 01 08:59:21 web3.aramgroup.com kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 01 08:59:21 web3.aramgroup.com kernel: Call Trace:
Sep 01 08:59:21 web3.aramgroup.com kernel:  <TASK>
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? __die_body.cold+0x19/0x27
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? page_fault_oops+0x15a/0x2d0
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? __kernfs_new_node+0x17d/0x200
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? exc_page_fault+0x81/0x190
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? asm_exc_page_fault+0x26/0x30
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? video_is_primary_device+0x9/0x40
Sep 01 08:59:21 web3.aramgroup.com kernel:  do_fb_registered+0x100/0x110
Sep 01 08:59:21 web3.aramgroup.com kernel:  fbcon_fb_registered+0x4d/0x70
Sep 01 08:59:21 web3.aramgroup.com kernel:  register_framebuffer+0x198/0x2a0
Sep 01 08:59:21 web3.aramgroup.com kernel:  xenfb_probe+0x30d/0x430 [xen_fbfront 61323dae510a72b3d2c332a2b0273cf6365e9002]
Sep 01 08:59:21 web3.aramgroup.com kernel:  xenbus_dev_probe+0xe3/0x1d0
Sep 01 08:59:21 web3.aramgroup.com kernel:  really_probe+0xdb/0x340
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? pm_runtime_barrier+0x54/0x90
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? __pfx___driver_attach+0x10/0x10
Sep 01 08:59:21 web3.aramgroup.com kernel:  __driver_probe_device+0x78/0x110
Sep 01 08:59:21 web3.aramgroup.com kernel:  driver_probe_device+0x1f/0xa0
Sep 01 08:59:21 web3.aramgroup.com kernel:  __driver_attach+0xba/0x1c0
Sep 01 08:59:21 web3.aramgroup.com kernel:  bus_for_each_dev+0x8c/0xe0
Sep 01 08:59:21 web3.aramgroup.com kernel:  bus_add_driver+0x112/0x1f0
Sep 01 08:59:21 web3.aramgroup.com kernel:  driver_register+0x72/0xd0
Sep 01 08:59:21 web3.aramgroup.com kernel:  __xenbus_register_frontend+0x2b/0x50
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? __pfx_xenfb_init+0x10/0x10 [xen_fbfront 61323dae510a72b3d2c332a2b0273cf6365e9002]
Sep 01 08:59:21 web3.aramgroup.com kernel:  do_one_initcall+0x58/0x310
Sep 01 08:59:21 web3.aramgroup.com kernel:  do_init_module+0x60/0x220
Sep 01 08:59:21 web3.aramgroup.com kernel:  init_module_from_file+0x89/0xe0
Sep 01 08:59:21 web3.aramgroup.com kernel:  idempotent_init_module+0x121/0x320
Sep 01 08:59:21 web3.aramgroup.com kernel:  __x64_sys_finit_module+0x5e/0xb0
Sep 01 08:59:21 web3.aramgroup.com kernel:  do_syscall_64+0x82/0x190
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? do_user_addr_fault+0x36c/0x620
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? clear_bhb_loop+0x25/0x80
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? clear_bhb_loop+0x25/0x80
Sep 01 08:59:21 web3.aramgroup.com kernel:  ? clear_bhb_loop+0x25/0x80
Sep 01 08:59:21 web3.aramgroup.com kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Sep 01 08:59:21 web3.aramgroup.com kernel: RIP: 0033:0x72bfd6f261fd
Sep 01 08:59:21 web3.aramgroup.com kernel: Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e3 fa 0c 00 f7 d8 64 89 01 48
Sep 01 08:59:21 web3.aramgroup.com kernel: RSP: 002b:00007ffe27bf3868 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Sep 01 08:59:21 web3.aramgroup.com kernel: RAX: ffffffffffffffda RBX: 00005aeffdff9b00 RCX: 000072bfd6f261fd
Sep 01 08:59:21 web3.aramgroup.com kernel: RDX: 0000000000000004 RSI: 000072bfd74ba05d RDI: 0000000000000011
Sep 01 08:59:21 web3.aramgroup.com kernel: RBP: 00007ffe27bf3920 R08: 0000000000000002 R09: 00007ffe27bf38d0
Sep 01 08:59:21 web3.aramgroup.com kernel: R10: 0000000000000007 R11: 0000000000000246 R12: 000072bfd74ba05d
Sep 01 08:59:21 web3.aramgroup.com kernel: R13: 0000000000020000 R14: 00005aeffdff8e90 R15: 00005aeffdfbc4c0
Sep 01 08:59:21 web3.aramgroup.com kernel:  </TASK>
Sep 01 08:59:21 web3.aramgroup.com kernel: Modules linked in: xen_netfront(+) xen_fbfront(+) intel_uncore(-) pcspkr loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 xen_blkfront crc32c_intel
Sep 01 08:59:21 web3.aramgroup.com kernel: CR2: 0000000000000060
Sep 01 08:59:21 web3.aramgroup.com kernel: ---[ end trace 0000000000000000 ]---
Sep 01 08:59:21 web3.aramgroup.com kernel: RIP: 0010:video_is_primary_device+0x9/0x40
Sep 01 08:59:21 web3.aramgroup.com kernel: Code: 48 89 d8 5b c3 cc cc cc cc 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 <48> 81 7f 60 80 e3 54 90 74 07 31 c0 c3 cc cc cc cc 53 48 89 fb 48
Sep 01 08:59:21 web3.aramgroup.com kernel: RSP: 0000:ffffbb06808d7a60 EFLAGS: 00010246
Sep 01 08:59:21 web3.aramgroup.com kernel: RAX: 0000000000000000 RBX: ffff90ca41367800 RCX: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: RBP: 0000000000000000 R08: 0000000000000060 R09: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: R10: ffffbb06808d7a78 R11: 0000000000000006 R12: ffffbb06808d7a90
Sep 01 08:59:21 web3.aramgroup.com kernel: R13: ffff90ca41367a88 R14: ffff90ca41367a60 R15: ffff90cb41330788
Sep 01 08:59:21 web3.aramgroup.com kernel: FS:  000072bfd74c0880(0000) GS:ffff90ce33a00000(0000) knlGS:0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 01 08:59:21 web3.aramgroup.com kernel: CR2: 0000000000000060 CR3: 0000000001326002 CR4: 00000000003706f0
Sep 01 08:59:21 web3.aramgroup.com kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 01 08:59:21 web3.aramgroup.com kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 01 08:59:21 web3.aramgroup.com kernel: note: (udev-worker)[250] exited with irqs disabled
Sep 01 08:59:21 web3.aramgroup.com kernel: xen_netfront: backend supports XDP headroom

Attachment: kernel-6.9.10-good.txt
Description: Text document

Attachment: kernel-6.10.7-bad.txt
Description: Text document