|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] Restart domU failure (memory allocation issue)
Hello, all. I am working with DRA7 (OMAP5) platform and I am using XEN 4.4 and Kernel 3.8 for both domains. As we still have 1:1 mapping for domU we need to allocate memory (1024 MB in our case) for it at one chunk. The issue is that I can't create domain again after destroying it (I am destroying domain by a "xl destroy" command) because we no longer have contiguous memory region and as result allocation failed. It seems, we have memory leak. I see that not all memory is freed after domain destroy. I see that some pages (which were allocated for kernel segment) don't returned to the heap. Some debug info: 1. before creating domain: (XEN) Physical memory information: (XEN) Xen heap: 196264kB free (XEN) heap[20]: 1193908kB free (XEN) Dom heap: 1193908kB free (XEN) heap[node=0][zone=0] -> 49066 pages (XEN) heap[node=0][zone=1] -> 0 pages (XEN) heap[node=0][zone=2] -> 0 pages (XEN) heap[node=0][zone=3] -> 0 pages (XEN) heap[node=0][zone=4] -> 0 pages (XEN) heap[node=0][zone=5] -> 0 pages (XEN) heap[node=0][zone=6] -> 0 pages (XEN) heap[node=0][zone=7] -> 0 pages (XEN) heap[node=0][zone=8] -> 0 pages (XEN) heap[node=0][zone=9] -> 0 pages (XEN) heap[node=0][zone=10] -> 0 pages (XEN) heap[node=0][zone=11] -> 0 pages (XEN) heap[node=0][zone=12] -> 0 pages (XEN) heap[node=0][zone=13] -> 0 pages (XEN) heap[node=0][zone=14] -> 0 pages (XEN) heap[node=0][zone=15] -> 0 pages (XEN) heap[node=0][zone=16] -> 0 pages (XEN) heap[node=0][zone=17] -> 0 pages (XEN) heap[node=0][zone=18] -> 0 pages (XEN) heap[node=0][zone=19] -> 0 pages (XEN) heap[node=0][zone=20] -> 298477 pages (XEN) heap[node=0][zone=21] -> 0 pages (XEN) heap[node=0][zone=22] -> 0 pages (XEN) heap[node=0][zone=23] -> 0 pages (XEN) heap[node=0][zone=24] -> 0 pages (XEN) heap[node=0][zone=25] -> 0 pages (XEN) heap[node=0][zone=26] -> 0 pages (XEN) heap[node=0][zone=27] -> 0 pages (XEN) heap[node=0][zone=28] -> 0 pages 2. create domain: / # xl -vvv create /xen/images/DomULinux_nodisk.cfg Parsing config from /xen/images/DomULinux_nodisk.cfg libxl: debug: libxl_create.c:1340:do_domain_create: ao 0x33108: create: how=(nil) callback=(nil) poller=0x32840 libxl: verbose: libxl_create.c:134:libxl__domain_build_info_setdefault: qemu-xen is unavailable, use qemu-xen-traditional instead: No such file or directory libxl: debug: libxl_create.c:797:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x33320: deregister unregistered libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=4, free_memkb=1357 libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement candidate with 1 nodes, 2 cpus and 1357 KB free selected domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)" libxl: debug: libxl_dom.c:361:libxl__build_pv: pv kernel mapped 0 path /xen/images/DomULinux.img domainbuilder: detail: xc_dom_kernel_file: filename="/xen/images/DomULinux.img" domainbuilder: detail: xc_dom_malloc_filemap : 5616 kB domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-armv7l domainbuilder: detail: xc_dom_rambase_init: RAM starts at 80000 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux uImage (ARM) loader ... domainbuilder: detail: xc_dom_probe_uimage_kernel: kernel is not a uImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64) loader ... domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an arm64 Image domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32) loader ... domainbuilder: detail: xc_dom_probe_zimage32_kernel: found an appended DTB domainbuilder: detail: loader probe OK domainbuilder: detail: xc_dom_parse_zimage32_kernel: called domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l: 0x80008000 -> 0x805842df libxl: debug: libxl_arm.c:511:libxl__arch_domain_configure: Skip constructing DTB (DTB is already prepared). // allocate contiguous 1 GB memory for domU: pages from 0x80000 to 0xC0000 domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x40000 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32 domainbuilder: detail: xc_dom_malloc : 2048 kB domainbuilder: detail: Allocating 2^18 pages at one chunk total_pages:0x00040000 i:0 // our print (XEN) >>>>> alloc_heap_pages [735] page: 0x80000 order: 18 // map memory for kernel segment: pages from 0x80008 to 0x80585 // As I understand these pages from domU memory are mapped to dom0 address space domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x80008000 -> 0x80585000 (pfn 0x80008 + 0x57d pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x80008+0x57d at 0xb606b000 domainbuilder: detail: xc_dom_load_zimage_kernel: called domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x80008000-0x80585000 domainbuilder: detail: xc_dom_load_zimage_kernel: copy 5751519 bytes from blob 0xb67e9000 to dst 0xb606b000 domainbuilder: detail: alloc_magic_pages: called domainbuilder: detail: count_pgtables_arm: called domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x80585000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: arch_setup_bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-armv7l <= matches domainbuilder: detail: setup_pgtables_arm: called domainbuilder: detail: clear_page: pfn 0xc0000, mfn 0xc0000 domainbuilder: detail: clear_page: pfn 0xc0001, mfn 0xc0001 domainbuilder: detail: start_info_arm: called domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 2082 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 5616 kB domainbuilder: detail: domU mmap : 5620 kB domainbuilder: detail: vcpu_arm32: called domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x80008000 // unmap memory for kernel segment // before call munmap() domainbuilder: detail: !!!!! xc_dom_unmap_one [604] ptr: 0xb606b000 count: 0x57d // before call xc_domain_cacheflush() domainbuilder: detail: !!!!! xc_dom_unmap_one [611] page: 0x80008 count: 0x57d domainbuilder: detail: !!!!! xc_dom_unmap_all [619] domainbuilder: detail: launch_vm: called, ctxt=0xb6d6a004 domainbuilder: detail: xc_dom_release: called ... ... ... libxl: debug: libxl_event.c:1761:libxl__ao_progress_report: ao 0x33108: progress report: ignored libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x33108: complete, rc=0 libxl: debug: libxl_create.c:1354:do_domain_create: ao 0x33108: inprogress: poller=0x32840, flags=ic libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x33108: destroy xc: debug: hypercall buffer: total allocations:237 total releases:237 xc: debug: hypercall buffer: current allocations:0 maximum allocations:4 xc: debug: hypercall buffer: cache current size:4 xc: debug: hypercall buffer: cache hits:230 misses:4 toobig:3 3. after creating domain: (XEN) Physical memory information: (XEN) Xen heap: 196140kB free (XEN) heap[20]: 143208kB free (XEN) Dom heap: 143208kB free (XEN) heap[node=0][zone=0] -> 49035 pages (XEN) heap[node=0][zone=1] -> 0 pages (XEN) heap[node=0][zone=2] -> 0 pages (XEN) heap[node=0][zone=3] -> 0 pages (XEN) heap[node=0][zone=4] -> 0 pages (XEN) heap[node=0][zone=5] -> 0 pages (XEN) heap[node=0][zone=6] -> 0 pages (XEN) heap[node=0][zone=7] -> 0 pages (XEN) heap[node=0][zone=8] -> 0 pages (XEN) heap[node=0][zone=9] -> 0 pages (XEN) heap[node=0][zone=10] -> 0 pages (XEN) heap[node=0][zone=11] -> 0 pages (XEN) heap[node=0][zone=12] -> 0 pages (XEN) heap[node=0][zone=13] -> 0 pages (XEN) heap[node=0][zone=14] -> 0 pages (XEN) heap[node=0][zone=15] -> 0 pages (XEN) heap[node=0][zone=16] -> 0 pages (XEN) heap[node=0][zone=17] -> 0 pages (XEN) heap[node=0][zone=18] -> 0 pages (XEN) heap[node=0][zone=19] -> 0 pages (XEN) heap[node=0][zone=20] -> 35802 pages (XEN) heap[node=0][zone=21] -> 0 pages (XEN) heap[node=0][zone=22] -> 0 pages (XEN) heap[node=0][zone=23] -> 0 pages (XEN) heap[node=0][zone=24] -> 0 pages (XEN) heap[node=0][zone=25] -> 0 pages (XEN) heap[node=0][zone=26] -> 0 pages (XEN) heap[node=0][zone=27] -> 0 pages (XEN) heap[node=0][zone=28] -> 0 pages / # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 128 2 r----- 3.2 win 1 1024 2 -b---- 0.8 4. destroy domain: libxl: debug: libxl.c:1250:libxl_domain_destroy: ao 0x32850: create: how=(nil) callback=(nil) poller=0x326d0 // our prints (XEN) <<<<< free_heap_pages [831] page: 0x80000 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80001 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80002 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80003 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80004 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80005 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80006 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80007 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80585 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80586 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80587 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80588 order: 0 (XEN) <<<<< free_heap_pages [831] page: 0x80589 order: 0 ... ... ... We can see that pages which were using for kernel segment (from 0x80008 to 0x80585) are absent here. 5. after destroying domain: (XEN) Physical memory information: (XEN) Xen heap: 196140kB free (XEN) heap[20]: 1186164kB free (XEN) Dom heap: 1186164kB free (XEN) heap[node=0][zone=0] -> 49035 pages (XEN) heap[node=0][zone=1] -> 0 pages (XEN) heap[node=0][zone=2] -> 0 pages (XEN) heap[node=0][zone=3] -> 0 pages (XEN) heap[node=0][zone=4] -> 0 pages (XEN) heap[node=0][zone=5] -> 0 pages (XEN) heap[node=0][zone=6] -> 0 pages (XEN) heap[node=0][zone=7] -> 0 pages (XEN) heap[node=0][zone=8] -> 0 pages (XEN) heap[node=0][zone=9] -> 0 pages (XEN) heap[node=0][zone=10] -> 0 pages (XEN) heap[node=0][zone=11] -> 0 pages (XEN) heap[node=0][zone=12] -> 0 pages (XEN) heap[node=0][zone=13] -> 0 pages (XEN) heap[node=0][zone=14] -> 0 pages (XEN) heap[node=0][zone=15] -> 0 pages (XEN) heap[node=0][zone=16] -> 0 pages (XEN) heap[node=0][zone=17] -> 0 pages (XEN) heap[node=0][zone=18] -> 0 pages (XEN) heap[node=0][zone=19] -> 0 pages (XEN) heap[node=0][zone=20] -> 296541 pages (XEN) heap[node=0][zone=21] -> 0 pages (XEN) heap[node=0][zone=22] -> 0 pages (XEN) heap[node=0][zone=23] -> 0 pages (XEN) heap[node=0][zone=24] -> 0 pages (XEN) heap[node=0][zone=25] -> 0 pages (XEN) heap[node=0][zone=26] -> 0 pages (XEN) heap[node=0][zone=27] -> 0 pages (XEN) heap[node=0][zone=28] -> 0 pages / # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 128 2 r----- 4.1 (null) 1 5 2 --p--d 0.9 6. Create domain again: / # xl -vvv create /xen/images/DomULinux_nodisk.cfg Parsing config from /xen/images/DomULinux_nodisk.cfg libxl: debug: libxl_create.c:1340:do_domain_create: ao 0x33108: create: how=(nil) callback=(nil) poller=0x32840 libxl: verbose: libxl_create.c:134:libxl__domain_build_info_setdefault: qemu-xen is unavailable, use qemu-xen-traditional instead: No such file or directory libxl: debug: libxl_create.c:797:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x33320: deregister unregistered libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=6, free_memkb=1349 libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement candidate with 1 nodes, 2 cpus and 1349 KB free selected domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)" libxl: debug: libxl_dom.c:361:libxl__build_pv: pv kernel mapped 0 path /xen/images/DomULinux.img domainbuilder: detail: xc_dom_kernel_file: filename="/xen/images/DomULinux.img" domainbuilder: detail: xc_dom_malloc_filemap : 5616 kB domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-armv7l domainbuilder: detail: xc_dom_rambase_init: RAM starts at 80000 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux uImage (ARM) loader ... domainbuilder: detail: xc_dom_probe_uimage_kernel: kernel is not a uImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64) loader ... domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an arm64 Image domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32) loader ... domainbuilder: detail: xc_dom_probe_zimage32_kernel: found an appended DTB domainbuilder: detail: loader probe OK domainbuilder: detail: xc_dom_parse_zimage32_kernel: called domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l: 0x80008000 -> 0x805842df libxl: debug: libxl_arm.c:511:libxl__arch_domain_configure: Skip constructing DTB (DTB is already prepared). domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x40000 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32 domainbuilder: detail: xc_dom_malloc : 2048 kB domainbuilder: detail: Allocating 2^18 pages at one chunk total_pages:0x00040000 i:0 // our print: we no longer have contiguous memory region ( (XEN) >>>>> alloc_heap_pages [674] page: NULL order: 18 (XEN) memory.c:160:d0 Could not allocate order=18 extent: id=2 memflags=0 (0 of 1) xc: detail: Failed allocation for dom 2: 1 extents of order 18 domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x80008000 -> 0x80585000 (pfn 0x80008 + 0x57d pages) (XEN) dom2 IPA 0x0000000080008000 (XEN) P2M @ 02bddb00 mfn:0xdeed8 (XEN) 1ST[0x2] = 0x0000000000000000 [ 51.851562] Failed to map pfn to mfn rc:0:-22 pfn:cc995 mfn:80008 (XEN) dom2 IPA 0x0000000080009000 (XEN) P2M @ 02bddb00 mfn:0xdeed8 (XEN) 1ST[0x2] = 0x0000000000000000 [ 51.867187] Failed to map pfn to mfn rc:0:-22 pfn:cc994 mfn:80009 (XEN) dom2 IPA 0x000000008000a000 (XEN) P2M @ 02bddb00 mfn:0xdeed8 (XEN) 1ST[0x2] = 0x0000000000000000 [ 51.882812] Failed to map pfn to mfn rc:0:-22 pfn:cc993 mfn:8000a (XEN) dom2 IPA 0x000000008000b000 (XEN) P2M @ 02bddb00 mfn:0xdeed8 (XEN) 1ST[0x2] = 0x0000000000000000 ... ... ... xc: error: panic: xc_dom_boot.c:191: xc_dom_boot_domU_map: failed to mmap domU pages 0x80008+0x57d [mmap, errno=14 (Bad address)]: Internal error libxl: error: libxl_dom.c:427:libxl__build_pv: xc_dom_build_image failed: Bad address domainbuilder: detail: xc_dom_release: called libxl: error: libxl_create.c:1022:domcreate_rebuild_done: cannot (re-)build domain: -3 libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x33108: complete, rc=-3 libxl: debug: libxl_create.c:1354:do_domain_create: ao 0x33108: inprogress: poller=0x32840, flags=ic libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x33108: destroy xc: debug: hypercall buffer: total allocations:92 total releases:92 xc: debug: hypercall buffer: current allocations:0 maximum allocations:4 xc: debug: hypercall buffer: cache current size:4 xc: debug: hypercall buffer: cache hits:85 misses:4 toobig:3 Please, could anyone explain me what is wrong? Can I perform any steps manually to free this memory region? Yes, we have to drop 1:1 mapping, but from my point of view this behavior looks odd (what happened with these pages after domain create/destroy?). Thank you. -- Oleksandr Tyshchenko | Embedded Developer GlobalLogic www.globallogic.com _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |