[Xen-devel] [QUESTION] Xen 4.8.4 - Unable to start VM's: spawn failed (rc=-3)

["Alex Braunegg" <alex.braunegg@xxxxxxxxx>]

Hi all,

I have recently upgraded from Xen 4.8.3 to 4.8.4 and now no VM's will start
as they fail with the following error message:

---------
Parsing config from /etc/xen/config/Windows_Server.cfg
libxl: error: libxl_dm.c:2189:device_model_spawn_outcome: domain 4 device
model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -3
libxl: error: libxl_dm.c:2303:kill_device_model: Device Model already exited
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 4
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 4
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 4 failed
---------

I have searched the mailing lists for any idea's / clues and come up short.
I have since rolled back to 4.8.3 & all my VM's start without issue.

Is there any update to the VM configuration file needed when updating to
4.8.4?

--------
builder='hvm'
#memory = 1536
memory = 2048
uuid = '<redacted>'
name = 'Windows_Server'
vif = [ 'bridge=br0, mac=<redacted>' ]
disk = [
'/dev/zvol/storage/xen/Windows_Server/disk_sda,,hda','/storage/samba/xeniso/
en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15
-59754.iso,,hdc,cdrom' ]
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot='dc'
sdl=0
vnc=1
vncconsole=1
vnclisten='0.0.0.0'
stdvga=1
serial='pty'
usbdevice='tablet'
vncpasswd='<redacted>'
vncdisplay=7
localtime=1
#audio='1'
#soundhw='ac97'
--------

Full debug log below in case this is required:

--------
libxl: debug: libxl_create.c:1614:do_domain_create: ao 0x1017c50: create:
how=(nil) callback=(nil) poller=0x1017ce0
libxl: debug: libxl_device.c:367:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:402:libxl__device_disk_set_backend: Disk
vdev=hda, using backend phy
libxl: debug: libxl_device.c:367:libxl__device_disk_set_backend: Disk
vdev=hdc spec.backend=qdisk
libxl: debug: libxl_create.c:970:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV domain,
skipping bootloader
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x1018ac0: deregister unregistered
domainbuilder: detail: xc_dom_allocate: cmdline="(null)", features="(null)"
domainbuilder: detail: xc_dom_kernel_file:
filename="/usr/lib/xen/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap    : 174 kB
libxl: debug: libxl_dom.c:884:libxl__load_hvm_firmware_module: Loading BIOS:
/usr/lib/xen/boot/seabios.bin
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.8, caps xen-3.0-x86_64
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader
... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ... 
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x33c04
xc: detail: ELF: memory: 0x100000 -> 0x133c04
domainbuilder: detail: xc_dom_mem_init: mem 2032 MB, pages 0x7f000 pages, 4k
each
domainbuilder: detail: xc_dom_mem_init: 0x7f000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: xc_dom_malloc            : 4064 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000003f7
xc: detail:   1GB PAGES: 0x0000000000000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x100+0x34 at 0x7f966cc86000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 ->
0x134000  (pfn 0x100 + 0x34 pages)
xc: detail: ELF: phdr 0 at 0x7f966cc52000 -> 0x7f966cc7c180
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x134+0x40 at 0x7f966cc46000
domainbuilder: detail: xc_dom_alloc_segment:   System Firmware module :
0x134000 -> 0x174000  (pfn 0x134 + 0x40 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x174+0x1 at 0x7f966cd38000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x174000 ->
0x175000  (pfn 0x174 + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x175000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_64
domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff
domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 4069 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 174 kB
domainbuilder: detail:       domU mmap          : 468 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:367:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=phy
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x101b920 wpath=/local/domain/0/backend/vbd/7/768/state token=3/0:
register slotnum=3
libxl: debug: libxl_device.c:367:libxl__device_disk_set_backend: Disk
vdev=hdc spec.backend=qdisk
libxl: debug: libxl_linux.c:221:libxl__get_hotplug_script_info: backend_kind
3, no need to execute scripts
libxl: debug: libxl_device.c:1161:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x101d020: deregister unregistered
libxl: debug: libxl_create.c:1640:do_domain_create: ao 0x1017c50:
inprogress: poller=0x1017ce0, flags=i
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x101b920
wpath=/local/domain/0/backend/vbd/7/768/state token=3/0: event
epath=/local/domain/0/backend/vbd/7/768/state
libxl: debug: libxl_event.c:874:devstate_callback: backend
/local/domain/0/backend/vbd/7/768/state wanted state 2 ok
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x101b920 wpath=/local/domain/0/backend/vbd/7/768/state token=3/0:
deregister slotnum=3
libxl: debug: libxl_device.c:1077:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x101b920: deregister unregistered
libxl: debug: libxl_linux.c:183:libxl__hotplug_disk: Args and environment
ready
libxl: debug: libxl_device.c:1175:device_hotplug: calling hotplug script:
/etc/xen/scripts/block add
libxl: debug: libxl_device.c:1176:device_hotplug: extra args:
libxl: debug: libxl_device.c:1184:device_hotplug: env:
libxl: debug: libxl_device.c:1191:device_hotplug:       script:
/etc/xen/scripts/block
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_TYPE: vbd
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_PATH:
backend/vbd/7/768
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_BASE_PATH:
backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block add 
libxl: debug: libxl_event.c:542:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/7/768/state token=3/0: empty slot
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x101ba20: deregister unregistered
libxl: debug: libxl_linux.c:200:libxl__get_hotplug_script_info: num_exec 1,
not running hotplug scripts
libxl: debug: libxl_device.c:1161:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x101ba20: deregister unregistered
libxl: debug: libxl.c:2889:libxl__device_disk_find_local_path: Directly
accessing local RAW disk /dev/zvol/storage/xen/Windows_Server/disk_sda
libxl: debug: libxl.c:2889:libxl__device_disk_find_local_path: Directly
accessing local RAW disk
/storage/samba/xeniso/en_windows_server_2008_r2_standard_enterprise_datacent
er_and_web_x64_dvd_x15-59754.iso
libxl: debug: libxl_dm.c:1500:libxl__build_device_model_args_new: Could not
find user xen-qemuuser-shared, starting QEMU as root
libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: Spawning device-model
/usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
/usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   7
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-7,server,nowait
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -no-shutdown
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-7,server,nowait
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -no-user-config
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   Windows_Server
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
0.0.0.0:7,password,websocket,x509=/etc/pki/xen,to=0
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -display
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   VGA,vgamem_mb=16
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   order=dc
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -usb
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -usbdevice
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   tablet
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
rtl8139,id=nic0,netdev=net0,mac=00:16:3e:1f:89:d4
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
type=tap,id=net0,ifname=vif7.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   2032
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
file=/dev/zvol/storage/xen/Windows_Server/disk_sda,if=ide,index=0,media=disk
,format=raw,cache=writeback
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:
if=ide,index=2,readonly=on,media=cdrom,id=ide-5632,file=/storage/samba/xenis
o/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x
15-59754.iso,format=raw
libxl: debug: libxl_dm.c:2098:libxl__spawn_local_dm: Spawning device-model
/usr/lib/xen/bin/qemu-system-i386 with additional environment:
libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm:
XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x1018db8 wpath=/local/domain/0/device-model/7/state token=3/1: register
slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x1018db8
wpath=/local/domain/0/device-model/7/state token=3/1: event
epath=/local/domain/0/device-model/7/state
libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 7 device model:
spawn watch p=(null)
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x1018db8 wpath=/local/domain/0/device-model/7/state token=3/1: deregister
slotnum=3
libxl: error: libxl_dm.c:2189:device_model_spawn_outcome: domain 7 device
model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model
did not start: -3
libxl: error: libxl_dm.c:2303:kill_device_model: Device Model already exited
libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch
w=0x1022480 wpath=/local/domain/0/backend/vbd/7/768/state token=3/2:
register slotnum=3
libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x1022480
wpath=/local/domain/0/backend/vbd/7/768/state token=3/2: event
epath=/local/domain/0/backend/vbd/7/768/state
libxl: debug: libxl_event.c:874:devstate_callback: backend
/local/domain/0/backend/vbd/7/768/state wanted state 6 ok
libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch
w=0x1022480 wpath=/local/domain/0/backend/vbd/7/768/state token=3/2:
deregister slotnum=3
libxl: debug: libxl_device.c:1077:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x1022480: deregister unregistered
libxl: debug: libxl_linux.c:183:libxl__hotplug_disk: Args and environment
ready
libxl: debug: libxl_device.c:1175:device_hotplug: calling hotplug script:
/etc/xen/scripts/block remove
libxl: debug: libxl_device.c:1176:device_hotplug: extra args:
libxl: debug: libxl_device.c:1184:device_hotplug: env:
libxl: debug: libxl_device.c:1191:device_hotplug:       script:
/etc/xen/scripts/block
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_TYPE: vbd
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_PATH:
backend/vbd/7/768
libxl: debug: libxl_device.c:1191:device_hotplug:       XENBUS_BASE_PATH:
backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block remove 
libxl: debug: libxl_event.c:542:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/7/768/state token=3/2: empty slot
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x1022580: deregister unregistered
libxl: debug: libxl_linux.c:200:libxl__get_hotplug_script_info: num_exec 1,
not running hotplug scripts
libxl: debug: libxl_device.c:1161:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x1022580: deregister unregistered
libxl: debug: libxl_linux.c:221:libxl__get_hotplug_script_info: backend_kind
3, no need to execute scripts
libxl: debug: libxl_device.c:1161:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x1022870: deregister unregistered
libxl: debug: libxl_linux.c:221:libxl__get_hotplug_script_info: backend_kind
6, no need to execute scripts
libxl: debug: libxl_device.c:1161:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
w=0x101dc60: deregister unregistered
libxl: debug: libxl.c:1712:devices_destroy_cb: forked pid 9724 for destroy
of domain 7
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x1017c50: complete,
rc=-3
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x1017c50: destroy
libxl: debug: libxl.c:1445:libxl_domain_destroy: ao 0x101ac00: create:
how=(nil) callback=(nil) poller=0x1017ce0
libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 7
libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 7
libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 7 failed
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x101ac00: complete,
rc=-21
libxl: debug: libxl.c:1454:libxl_domain_destroy: ao 0x101ac00: inprogress:
poller=0x1017ce0, flags=ic
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x101ac00: destroy
xencall:buffer: debug: total allocations:676 total releases:676
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:658 misses:3 toobig:15
--------


Best regards,

Alex




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel