Re: [Xen-users] After Update: NUMA placement failed, performance might be affected

[WebDawg <webdawg@xxxxxxxxx>]

On Thu, Jan 5, 2017 at 4:19 AM, Helmut Schlang <piwwo@xxxxxx> wrote:

Hello.

After I updated my Xen server that's running on debian jessie, I can not
create PV nor HVM guests anymore. Everything was working perfectly fine
before the update. What does "NUMA placement failed" even mean?

root@hpx01:~# apt-get install xen-hypervisor-4.4-amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
xen-hypervisor-4.4-amd64 is already the newest version.
xen-hypervisor-4.4-amd64 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
root@hpx01:~# apt-get install xen-system-amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
xen-system-amd64 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

root@hpx01:~# uname -a
Linux hpx01 3.16.0-4-686-pae #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
i686 GNU/Linux

root@hpx01:~# xl -vvvv create /xen/windows.cfg
Parsing config from /xen/windows.cfg
libxl: debug: libxl_create.c:1347:do_domain_create: ao 0x8860cd0:
create: how=(nil) callback=(nil) poller=0x8860ee0
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:331:libxl__device_disk_set_backend: Disk
vdev=hda, using backend phy
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk
vdev=hdb spec.backend=unknown
libxl: debug: libxl_device.c:331:libxl__device_disk_set_backend: Disk
vdev=hdb, using backend phy
libxl: debug: libxl_create.c:799:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV
domain, skipping bootloader
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
w=0x88610d0: deregister unregistered
libxl: notice: libxl_numa.c:494:libxl__get_numa_candidate: NUMA
placement failed, performance might be affected
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x59658
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x159658
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->0000000000159658
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->00000000fd500000
  ENTRY ADDRESS: 0000000000100648
xc: detail: Failed allocation for dom 6: 160 extents of order 0
xc: error: Could not allocate memory for HVM guest. (16 = Device or
resource busy): Internal error
libxl: error: libxl_dom.c:655:libxl__build_hvm: hvm building failed
libxl: error: libxl_create.c:1024:domcreate_rebuild_done: cannot
(re-)build domain: -3
libxl: error: libxl_dm.c:1489:kill_device_model: unable to find device
model pid in /local/domain/6/image/device-model-pid
libxl: error: libxl.c:1437:libxl__destroy_domid:
libxl__destroy_device_model failed for 6
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x8860cd0:
complete, rc=-3
libxl: debug: libxl_create.c:1361:do_domain_create: ao 0x8860cd0:
inprogress: poller=0x8860ee0, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x8860cd0: destroy
xc: debug: hypercall buffer: total allocations:138 total releases:138
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:130 misses:4 toobig:4

root@hpx01:~# xl -vvvv create /xen/nagios.cfg
Parsing config from /xen/nagios.cfg
libxl: debug: libxl_create.c:1347:do_domain_create: ao 0x94fec00:
create: how=(nil) callback=(nil) poller=0x94fed58
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:331:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:799:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk
vdev=(null) spec.backend=phy
libxl: debug: libxl.c:2664:libxl__device_disk_local_initiate_attach:
locally attaching PHY disk /dev/main/nagios
libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config
bootloader value: pygrub
libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb:
Checking for bootloader in libexec path: /usr/lib/xen-4.4/bin/pygrub
libxl: debug: libxl_create.c:1361:do_domain_create: ao 0x94fec00:
inprogress: poller=0x94fed58, flags=i
libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch
w=0x94fef48 wpath=/local/domain/7 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:1761:libxl__ao_progress_report: ao
0x94fec00: progress report: ignored
libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing
bootloader: /usr/lib/xen-4.4/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: /usr/lib/xen-4.4/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: --args=root=/dev/xvda1
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: --output=/var/run/xen/bootloader.7.out
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: --output-directory=/var/run/xen/bootloader.7.d
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader
arg: /dev/main/nagios
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x94fef48
wpath=/local/domain/7 token=3/0: event epath=/local/domain/7
libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader
completed
libxl: debug: libxl_bootloader.c:136:bootloader_result_command:
bootloader output contained kernel
/var/run/xen/bootloader.7.d/boot_kernel.Yj1WwT
libxl: debug: libxl_bootloader.c:136:bootloader_result_command:
bootloader output contained ramdisk
/var/run/xen/bootloader.7.d/boot_ramdisk.MLOt5g
libxl: debug: libxl_bootloader.c:136:bootloader_result_command:
bootloader output contained args
root=UUID=5ece8d0f-f3b5-48db-8d30-cacacc6e8560 ro  quiet root=/dev/xvda1
libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader
execution successful
libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch
w=0x94fef48 wpath=/local/domain/7 token=3/0: deregister slotnum=3
libxl: notice: libxl_numa.c:494:libxl__get_numa_candidate: NUMA
placement failed, performance might be affected
domainbuilder: detail: xc_dom_allocate:
cmdline="root=UUID=5ece8d0f-f3b5-48db-8d30-cacacc6e8560 ro  quiet
root=/dev/xvda1 ", features="(null)"
libxl: debug: libxl_dom.c:364:libxl__build_pv: pv kernel mapped 1 path
/var/run/xen/bootloader.7.d/boot_kernel.Yj1WwT
domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary
loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: XZ: Saw data stream end
domainbuilder: detail: _xc_try_lzma_decode: XZ decompress OK, 0x2ed5ec
-> 0xe21960
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0x7cb000
xc: detail: elf_parse_binary: phdr: paddr=0x1800000 memsz=0xee000
xc: detail: elf_parse_binary: phdr: paddr=0x18ee000 memsz=0x13c00
xc: detail: elf_parse_binary: phdr: paddr=0x1902000 memsz=0x616000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x1f18000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff819021f0
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff81000000
xc: detail:     virt_kend        = 0xffffffff81f18000
xc: detail:     virt_entry       = 0xffffffff819021f0
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64:
0xffffffff81000000 -> 0xffffffff81f18000
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 1024 kB
xc: detail: Failed allocation for dom 7: 262144 extents of order 0
xc: error: panic: xc_dom_boot.c:154: xc_dom_boot_mem_init: can't
allocate low memory for domain: Out of memory
libxl: error: libxl_dom.c:426:libxl__build_pv: xc_dom_boot_mem_init
failed: Success
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_create.c:1024:domcreate_rebuild_done: cannot
(re-)build domain: -3
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x94fec00:
complete, rc=-3
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x94fec00: destroy
xc: debug: hypercall buffer: total allocations:126 total releases:126
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:117 misses:4 toobig:5

_______________________________________________

Is there anyway you are out of memory?

http://xen.1045712.n5.nabble.com/Max-memory-td5717925.html

https://wiki.xen.org/wiki/Xen_on_NUMA_Machines

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users