[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [BUG] invocation of xl create with insufficient memory for new domU causes system reboot



02.04.2014 14:25, Georgy V. Repin ÐÐÑÐÑ:

02.04.2014 13:30, Ian Campbell ÐÐÑÐÑ:


On Tue, 2014-04-01 at 14:17 +0400, Georgy V. Repin wrote:

01.04.2014 13:52, Ian Campbell ÐÐÑÐÑ:

On Tue, 2014-04-01 at 13:35 +0400, Georgy V. Repin wrote:

Hello!

1. Problem: invocation of xl create when there is not enough memory to
create new domU causes oom to select process to kiill; when oom
selects
to kill quemu-system-* process, this causes kernel panic (it's my
suppose, because i see no message "kernel panic" in log, but i do see
"CPU: 0 PID: 7899 Comm: qemu-system-i38 Not tainted 3.10.17-gentoo
#1" +
failed call trace), and, therefore (assuming default settings) kernel
reboot (see log at "4.")
Couple of remarks:
- fix checking of available memory for new domU in xl toolstack could
serve as workaround
- real cause is not clear to me, maybe it's not even a xen bug
(quemu ?
kernel ?)


Are you using dom0_mem= together with auto ballooning enabled in the
toolstack?


informika-xen-3 ~ # cat /proc/cmdline
placeholder root=/dev/sda1 ro


dom0_mem is a hypervisor command line option, not a Linux one. Although
given that you have show autobaloon=auto that is moot since it implies
the automatic functionality was indeed added in the version you have.



part of grub.cfg

                if [ "$grub_platform" = "pc" -o "$grub_platform" = ""
]; then
                    xen_rm_opts=
                else
                    xen_rm_opts="no-real-mode edd=off"
                fi
            multiboot    /boot/xen-4.gz placeholder   ${xen_rm_opts}



^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
this code from grub.cfg shows that there is no dom0_mem=  parameter for
/boot/xen-4.gz



            echo    'Loading Linux 3.10.17-gentoo ...'
            module    /boot/vmlinuz-3.10.17-gentoo placeholder
root=/dev/sda1 ro

xl.conf:
# Control whether dom0 is ballooned down when xen doesn't have enough
# free memory to create a domain.  "auto" means only balloon if dom0
# starts with all the host's memory.
#autoballoon="auto"

So:
    - no dom0_mem= as boot param
    - autobaloon is set to default ("auto", i suppose)


Correct, the default is auto.

Please can you show us the output of "xl info" and your guest cfg file
as well as the full output of "xl -vvv create $cfg".


informika-xen-3 ~ # xl info
host                   : informika-xen-3
release                : 3.10.17-gentoo
version                : #1 SMP Sun Dec 8 13:22:31 MSK 2013
machine                : x86_64
nr_cpus                : 16
max_cpu_id             : 15
nr_nodes               : 2
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2400
hw_caps                :
bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 24566
free_memory            : 418
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 3
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder
cc_compiler            : x86_64-pc-linux-gnu-gcc (Gentoo 4.7.3-r1 p1.3,
pie-0.5.5) 4.7.3
cc_compile_by          :
cc_compile_domain      : internal
cc_compile_date        : Sun Jan  5 22:39:16 MSK 2014
xend_config_format     : 4


remark:
     - now there is no oom reaction, because i have reduced memory
amount for one of other domU's (now it's enough memory for all domU's)
     - it seems that "xen_commandline" states that there is no
"dom0_mem=" option.
sorry, now something wrong with my email, i see just part of email i was intended to send in my "sent" box. 

informika-xen-3 ~ # xl -vvv create /etc/xen/vm/www.icsit.ru
Parsing config from /etc/xen/vm/www.icsit.ru
libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x24915a0: create: how=(nil) callback=(nil) poller=0x24903a0 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy 
libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk /dev/storage-a/www.icsit.ru libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x24915a0: inprogress: poller=0x24903a0, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x248f798 wpath=/local/domain/8 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x24915a0: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --args=root=/dev/xvda fastboot libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.8.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.8.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /dev/storage-a/www.icsit.ru libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248f798 wpath=/local/domain/8 token=3/0: event epath=/local/domain/8
libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader completed libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained kernel /var/run/xen/bootloader.8.d/boot_kernel.VfLJyd libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained ramdisk /var/run/xen/bootloader.8.d/boot_ramdisk.HBHXnw libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained args root=/dev/xvda console=hvc0 ro root=/dev/xvda fastboot libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader execution successful libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x248f798 wpath=/local/domain/8 token=3/0: deregister slotnum=3 libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=2, nr_cpus=16, nr_vcpus=65, free_memkb=2466 libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement candidate with 2 nodes, 16 cpus and 2466 KB free selected domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvda console=hvc0 ro root=/dev/xvda fastboot", features="(null)" libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 1 path /var/run/xen/bootloader.8.d/boot_kernel.VfLJyd 

domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc            : 18250 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x4b30aa -> 0x11d2aa0
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0xad6000
xc: detail: elf_parse_binary: phdr: paddr=0x1c00000 memsz=0xe30e0
xc: detail: elf_parse_binary: phdr: paddr=0x1ce4000 memsz=0x143c0
xc: detail: elf_parse_binary: phdr: paddr=0x1cf9000 memsz=0x35f000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x2058000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff81cf9200
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb" 
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff81000000
xc: detail:     virt_kend        = 0xffffffff82058000
xc: detail:     virt_entry       = 0xffffffff81cf9200
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff81000000 -> 0xffffffff82058000 domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000 pages, 4k each 
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff81000000 -> 0xffffffff82058000 (pfn 0x1000 + 0x1058 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1000+0x1058 at 0x7f9aa40e4000 
xc: detail: elf_load_binary: phdr 0 at 0x7f9aa40e4000 -> 0x7f9aa4bba000
xc: detail: elf_load_binary: phdr 1 at 0x7f9aa4ce4000 -> 0x7f9aa4dc70e0
xc: detail: elf_load_binary: phdr 2 at 0x7f9aa4dc8000 -> 0x7f9aa4ddc3c0
xc: detail: elf_load_binary: phdr 3 at 0x7f9aa4ddd000 -> 0x7f9aa4eb6000
domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xffffffff82058000 -> 0xffffffff82fa8000 (pfn 0x2058 + 0xf50 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2058+0xf50 at 0x7f9a9f0b0000 
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x5c1479 -> 0xf4fa10
domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff82fa8000 -> 0xffffffff833a8000 (pfn 0x2fa8 + 0x400 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2fa8+0x400 at 0x7f9a9ecb0000 domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff833a8000 (pfn 0x33a8) domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff833a9000 (pfn 0x33a9) domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff833aa000 (pfn 0x33aa) domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff837fffff, 28 table(s) domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff833ab000 -> 0xffffffff833ca000 (pfn 0x33ab + 0x1f pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x33ab+0x1f at 0x7f9aa9696000 domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff833ca000 (pfn 0x33ca) domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff833cb000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff83800000 
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x33aa, mfn 0x5d4935
domainbuilder: detail: clear_page: pfn 0x33a9, mfn 0x21c2da
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x33a8+0x1 at 0x7f9aa9693000 
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 pfn=0x1001 
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 22564 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 0 bytes
domainbuilder: detail:       domU mmap          : 35 MB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0x1f6b9 
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x33ab mfn 0x21c2d9
domainbuilder: detail: launch_vm: called, ctxt=0x7fff07610e70
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event epath=/local/domain/0/backend/vbd/8/51712/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd/8/51712/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event epath=/local/domain/0/backend/vbd/8/51712/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/8/51712/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x248dd08: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: register slotnum=3 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: event epath=/local/domain/0/backend/vif/8/0/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vif/8/0/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: event epath=/local/domain/0/backend/vif/8/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/8/0/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x248de18: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x24915a0: progress report: ignored libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x24915a0: complete, rc=0 
libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x24915a0: destroy
Daemon running with PID 6087
xc: debug: hypercall buffer: total allocations:346 total releases:346
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:336 misses:4 toobig:6
informika-xen-3 ~ #

informika-xen-3 ~ # cat /etc/xen/vm/www.icsit.ru
bootloader = "pygrub"
memory = 2048
vcpus = 2
name = "www.icsit.ru"
vif = [ 'mac=00:16:3E:5A:04:01, bridge=v119, model=e1000' ]
disk = ['phy:/dev/storage-a/www.icsit.ru,xvda,w']
root = "/dev/xvda"
extra = "fastboot"








--
Ð ÑÐÐÐÐÐÐÐÐ,
        ÐÐÐÑÐÐÐ ÐÐÐÐÐ
        ÐÐÐÐ ÐÐÐÐ ÐÐÐ "ÐÐÑÐÑÐÐÐÐ"
        http://www.informika.ru
        http://www.runnet.ru


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.