Re: 4.15.0-rc3 can't create domain when dom0 uses newer kernel

[Tamas K Lengyel <tamas@xxxxxxxxxxxxx>]

On Mon, Mar 15, 2021 at 5:37 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Mon, Mar 15, 2021 at 04:18:42PM -0400, Tamas K Lengyel wrote:
> > Hi all,
> > I've ran into a pretty strange issue. I have a freshly installed 4.15.0-rc3
> > system and everything works fine with the stock Ubuntu 20.04
> > vmlinuz-5.4.0-66-generic as the dom0 kernel - except vmtrace, which needs a
> > newer kernel, that's expected. Now if I boot dom0 with
> > vmlinuz-5.8.0-44-generic or vmlinuz-5.10.0-1016-oem I can't start any
> > domains because of some strange disk error:
> >
> > root@t2:/shared/cfg# xl create ubuntu-20.04.cfg
> > Parsing config from ubuntu-20.04.cfg
> > libxl: error: libxl_device.c:1109:device_backend_callback: Domain 1:unable
> > to add device with path /local/domain/0/backend/vbd/1/51712
> > libxl: error: libxl_create.c:1613:domcreate_launch_dm: Domain 1:unable to
> > add disk devices
>
> Can you paste the output with `xl -vvv create ...`?

Here you go:

Parsing config from ubuntu-20.04.cfg
libxl: debug: libxl_create.c:2022:do_domain_create: ao 0x561348383da0: create: how=(nil) callback=(nil) poller=0x561348383e40
libxl: detail: libxl_create.c:627:libxl__domain_make: passthrough: disabled
libxl: debug: libxl_device.c:379:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:413:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy
libxl: debug: libxl_create.c:1284:initiate_domain_create: Domain 5:running bootloader
libxl: debug: libxl_bootloader.c:328:libxl__bootloader_run: Domain 5:not a PV/PVH domain, skipping bootloader
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348386270: deregister unregistered
domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
domainbuilder: detail: xc_dom_kernel_file: filename="/usr/local/lib/xen/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap    : 621 kB
libxl: debug: libxl_dom.c:822:libxl__load_hvm_firmware_module: Loading BIOS: /usr/local/lib/xen/boot/seabios.bin
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x61e04
xc: detail: ELF: memory: 0x100000 -> 0x161e04
domainbuilder: detail: xc_dom_mem_init: mem 10008 MB, pages 0x271800 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x271800 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: range: start=0x0 end=0xf0000000
domainbuilder: detail: range: start=0x100000000 end=0x281800000
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x000000000000038b
xc: detail:   1GB PAGES: 0x0000000000000008

domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x62 at 0x7fc72dd94000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 -> 0x162000  (pfn 0x100 + 0x62 pages)
xc: detail: ELF: phdr 0 at 0x7fc72dd32000 -> 0x7fc72dd8a260
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x162+0x40 at 0x7fc72dd54000
domainbuilder: detail: xc_dom_alloc_segment:   System Firmware module : 0x162000 -> 0x1a2000  (pfn 0x162 + 0x40 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1a2+0x1 at 0x7fc72e336000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x1a2000 -> 0x1a3000  (pfn 0x1a2 + 0x1 pages)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x1a3000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 7008 bytes
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 621 kB
domainbuilder: detail:       domU mmap          : 652 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[0] -> d0 0xfefff
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[1] -> d0 0xfeffc
domainbuilder: detail: xc_dom_release: called

libxl: debug: libxl_device.c:379:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy
libxl: debug: libxl_event.c:813:libxl__ev_xswatch_register: watch w=0x561348389260 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:2060:do_domain_create: ao 0x561348383da0: inprogress: poller=0x561348383e40, flags=i
libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x561348389260 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/>libxl: debug: libxl_event.c:1055:devstate_callback: backend /local/domain/0/backend/vbd/5/51712/state wanted state 2 still waiting state 1
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: backend /local/domain/0/backend/vbd/5/51712/state (hoping for state change to 2): xswait timeout (path=/local/doma>libxl: debug: libxl_event.c:850:libxl__ev_xswatch_deregister: watch w=0x561348389260 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:1039:devstate_callback: backend /local/domain/0/backend/vbd/5/51712/state wanted state 2  timed out
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348389260: deregister unregistered
libxl: debug: libxl_device.c:1096:device_backend_callback: Domain 5:calling device_backend_cleanup
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348389260: deregister unregistered
libxl: error: libxl_device.c:1109:device_backend_callback: Domain 5:unable to add device with path /local/domain/0/backend/vbd/5/51712
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348389360: deregister unregistered
libxl: error: libxl_create.c:1613:domcreate_launch_dm: Domain 5:unable to add disk devices
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348387c28: deregister unregistered
libxl: debug: libxl_event.c:813:libxl__ev_xswatch_register: watch w=0x561348388130 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x561348388130 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/1: event epath=/local/domain/0/backend/vbd/>libxl: debug: libxl_event.c:1055:devstate_callback: backend /local/domain/0/backend/vbd/5/51712/state wanted state 6 still waiting state 5
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: backend /local/domain/0/backend/vbd/5/51712/state (hoping for state change to 6): xswait timeout (path=/local/doma>libxl: debug: libxl_event.c:850:libxl__ev_xswatch_deregister: watch w=0x561348388130 wpath=/local/domain/0/backend/vbd/5/51712/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:1039:devstate_callback: backend /local/domain/0/backend/vbd/5/51712/state wanted state 6  timed out
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348388130: deregister unregistered
libxl: debug: libxl_device.c:1096:device_backend_callback: Domain 5:calling device_backend_cleanup
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348388130: deregister unregistered
libxl: error: libxl_device.c:1109:device_backend_callback: Domain 5:unable to remove device with path /local/domain/0/backend/vbd/5/51712
libxl: debug: libxl_event.c:864:libxl__ev_xswatch_deregister: watch w=0x561348388230: deregister unregistered
libxl: error: libxl_domain.c:1529:devices_destroy_cb: Domain 5:libxl__devices_destroy failed
libxl: debug: libxl_domain.c:1596:devices_destroy_cb: Domain 5:Forked pid 944059 for destroy of domain
libxl: debug: libxl_event.c:2066:libxl__ao_complete: ao 0x561348383da0: complete, rc=-3
libxl: debug: libxl_event.c:2035:libxl__ao__destroy: ao 0x561348383da0: destroy
libxl: debug: libxl_domain.c:1045:libxl_domain_destroy: Domain 5:ao 0x561348382810: create: how=(nil) callback=(nil) poller=0x561348383e40
libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain 5:Non-existant domain
libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain 5:Unable to destroy guest
libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain 5:Destruction of domain failed
libxl: debug: libxl_event.c:2066:libxl__ao_complete: ao 0x561348382810: complete, rc=-21
libxl: debug: libxl_domain.c:1054:libxl_domain_destroy: Domain 5:ao 0x561348382810: inprogress: poller=0x561348383e40, flags=ic

libxl: debug: libxl_event.c:2035:libxl__ao__destroy: ao 0x561348382810: destroy
xencall:buffer: debug: total allocations:603 total releases:603
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:587 misses:3 toobig:13
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0

Thanks,

Tamas