Re: [Xen-devel] [BUG] Qemu crash on xl create with memory allocation error (probably xl not consider all videoram for initial memory check)

[Vitaly Kuznetsov <vkuznets@xxxxxxxxxx>]

Fabio Fantoni <fabio.fantoni@xxxxxxx> writes:

> On dom0 with xen from git 4.4-stable I found unexpected case about
> memory check on xl create.
> If I try to start domU with 2 gb of ram but only 1840384k free (from
> xentop) xl give me the correct error:
>> xl create /etc/xen/W7-04.cfg
>> Parsing config from /etc/xen/W7-04.cfg
>> libxl: notice: libxl_numa.c:494:libxl__get_numa_candidate: NUMA
>> placement failed, performance might be affected
>> xc: error: Could not allocate memory for HVM guest as we cannot
>> claim memory! (12 = Cannot allocate memory): Internal error
>> libxl: error: libxl_dom.c:651:libxl__build_hvm: hvm building failed
>> libxl: error: libxl_create.c:1067:domcreate_rebuild_done: cannot
>> (re-)build domain: -3
>> libxl: error: libxl_dm.c:1486:kill_device_model: unable to find
>> device model pid in /local/domain/30/image/device-model-pid
>> libxl: error: libxl.c:1421:libxl__destroy_domid:
>> libxl__destroy_device_model failed for 30
>
> If I try to start the same domU with 1792 mb of ram xl don't stop for
> xc memory error but starts and fails when qemu crash on start:
>> xl create /etc/xen/W7-04.cfg
>> Parsing config from /etc/xen/W7-04.cfg
>> libxl: notice: libxl_numa.c:494:libxl__get_numa_candidate: NUMA
>> placement failed, performance might be affected
>> libxl: error: libxl_dm.c:1390:device_model_spawn_outcome: domain 31
>> device model: spawn failed (rc=-3)
>> libxl: error: libxl_create.c:1231:domcreate_devmodel_started: device
>> model did not start: -3
>> libxl: error: libxl_dm.c:1494:kill_device_model: Device Model
>> already exited
> /var/log/xen/qemu-dm-W7-04.log
> qemu: hardware error: xen: failed to populate ram at 6c002000

AFAICT it is definitely not the only scenario when QEMU will fail to
start. libxc claims only RAM (that's what failed in your case no. 1) and
QEMU can try allocating additional memory pages on top of that. *In
theory* libxc could claim more and fail earlier but it is a bit
complicated as all this knowledge lives in QEMU.

>
> This problem is happen to me with qxl vga that have big quantity of
> videoram (128mb) but I think can happen also with other emulated vga
> even if with lower probability.
> Exact xen source used is:
> https://github.com/Fantu/Xen/commits/rebase/m2r-testing (based on xen
> 4.4-stable branch)
>
> /etc/xen/W7-04.cfg
>> name='W7-04'
>> builder="hvm"
>> memory=1792
>> vcpus=2
>> acpi_s3=0
>> acpi_s4=0
>> #vif=['model=e1000,bridge=xenbr0']
>> vif=['bridge=xenbr0,mac=00:16:3e:41:b6:2d']
>> disk=['/mnt/vm/disks/W7-04.disk1.xm,raw,hda,rw',',raw,hdb,ro,cdrom']
>> boot='dc'
>> device_model_version="qemu-xen"
>> viridian=1
>> vnc=0
>> keymap="it"
>> on_crash="destroy"
>> vga="qxl"
>> spice=1
>> spicehost='0.0.0.0'
>> spiceport=6004
>> spicedisable_ticketing=1
>> spicevdagent=1
>> spice_clipboard_sharing=0
>> spiceusbredirection=4
>> soundhw="hda"
>> localtime=1
>> usbversion=2
>
> Full xl create with debug:
>> xl -vvv create /etc/xen/W7-04.cfg
>> Parsing config from /etc/xen/W7-04.cfg
>> libxl: debug: libxl_create.c:1387:do_domain_create: ao 0x1dd3f80:
>> create: how=(nil) callback=(nil) poller=0x1dd3fe0
>> libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend:
>> Disk vdev=hda spec.backend=unknown
>> libxl: debug: libxl_device.c:197:disk_try_backend: Disk vdev=hda,
>> backend phy unsuitable as phys path not a block device
>> libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend:
>> Disk vdev=hda, using backend qdisk
>> libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend:
>> Disk vdev=hdb spec.backend=unknown
>> libxl: debug: libxl_device.c:197:disk_try_backend: Disk vdev=hdb,
>> backend phy unsuitable as phys path not a block device
>> libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend:
>> Disk vdev=hdb, using backend qdisk
>> libxl: debug: libxl_create.c:842:initiate_domain_create: running
>> bootloader
>> libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV
>> domain, skipping bootloader
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1dd44c8: deregister unregistered
>> libxl: notice: libxl_numa.c:494:libxl__get_numa_candidate: NUMA
>> placement failed, performance might be affected
>> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x595b0
>> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x1595b0
>> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
>>   Loader:        0000000000100000->00000000001595b0
>>   Modules:       0000000000000000->0000000000000000
>>   TOTAL:         0000000000000000->0000000068000000
>>   ENTRY ADDRESS: 0000000000100000
>> xc: detail: PHYSICAL MEMORY ALLOCATION:
>>   4KB PAGES: 0x0000000000000200
>>   2MB PAGES: 0x000000000000033f
>>   1GB PAGES: 0x0000000000000000
>> xc: detail: elf_load_binary: phdr 0 at 0x7ff0a9342000 -> 0x7ff0a939254d
>> libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend:
>> Disk vdev=hda spec.backend=qdisk
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1dd63e0: deregister unregistered
>> libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend:
>> Disk vdev=hdb spec.backend=qdisk
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1dd7210: deregister unregistered
>> libxl: debug: libxl_dm.c:1322:libxl__spawn_local_dm: Spawning
>> device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> /usr/lib/xen/bin/qemu-system-i386
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -xen-domid
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   32
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-32,server,nowait
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -mon
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> chardev=libxl-cmd,mode=control
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -nodefaults
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -name
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   W7-04
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -k
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   it
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -spice
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> port=6004,tls-port=0,addr=0.0.0.0,disable-ticketing,agent-mouse=on,disable-copy-paste
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm: virtio-serial
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> spicevmc,id=vdagent,name=vdagent
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> virtserialport,chardev=vdagent,name=com.redhat.spice.0
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> qxl-vga,vram_size_mb=64,ram_size_mb=64
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -boot
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   order=dc
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> ich9-usb-ehci1,id=usb,addr=0x1d.0x7,multifunction=on
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,addr=0x1d.0,multifunction=on
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> ich9-usb-uhci2,masterbus=usb.0,firstport=2,addr=0x1d.0x1,multifunction=on
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> ich9-usb-uhci3,masterbus=usb.0,firstport=4,addr=0x1d.0x2,multifunction=on
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> spicevmc,name=usbredir,id=usbrc1
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> usb-redir,chardev=usbrc1,id=usbrc1
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> spicevmc,name=usbredir,id=usbrc2
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> usb-redir,chardev=usbrc2,id=usbrc2
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> spicevmc,name=usbredir,id=usbrc3
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> usb-redir,chardev=usbrc3,id=usbrc3
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -chardev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> spicevmc,name=usbredir,id=usbrc4
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> usb-redir,chardev=usbrc4,id=usbrc4
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -soundhw
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   hda
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -smp
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   2,maxcpus=2
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -device
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> rtl8139,id=nic0,netdev=net0,mac=00:16:3e:41:b6:2d
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -netdev
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> type=tap,id=net0,ifname=vif32.0-emu,script=no,downscript=no
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -machine
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   xenfv
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -m
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   1664
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -drive
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> file=/mnt/vm/disks/W7-04.disk1.xm,if=ide,index=0,media=disk,format=raw,cache=writeback
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:   -drive
>> libxl: debug: libxl_dm.c:1324:libxl__spawn_local_dm:
>> if=ide,index=1,media=cdrom,cache=writeback,id=ide-832
>> libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch
>> w=0x1dd4700 wpath=/local/domain/0/device-model/32/state token=3/0:
>> register slotnum=3
>> libxl: debug: libxl_create.c:1401:do_domain_create: ao 0x1dd3f80:
>> inprogress: poller=0x1dd3fe0, flags=i
>> libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x1dd4700
>> wpath=/local/domain/0/device-model/32/state token=3/0: event
>> epath=/local/domain/0/device-model/32/state
>> libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch
>> w=0x1dd4700 wpath=/local/domain/0/device-model/32/state token=3/0:
>> deregister slotnum=3
>> libxl: error: libxl_dm.c:1390:device_model_spawn_outcome: domain 32
>> device model: spawn failed (rc=-3)
>> libxl: error: libxl_create.c:1231:domcreate_devmodel_started: device
>> model did not start: -3
>> libxl: error: libxl_dm.c:1494:kill_device_model: Device Model
>> already exited
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1dda7f0: deregister unregistered
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1dda910: deregister unregistered
>> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
>> w=0x1ddab50: deregister unregistered
>> libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x1dd3f80:
>> complete, rc=-3
>> libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x1dd3f80: destroy
>> xc: debug: hypercall buffer: total allocations:855 total releases:855
>> xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
>> xc: debug: hypercall buffer: cache current size:4
>> xc: debug: hypercall buffer: cache hits:847 misses:4 toobig:4
>
> If you need more informations and/or tests tell me and I'll post them.
>
> Thanks for any reply and sorry for my bad english.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

-- 
  Vitaly

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel