[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Xen 4.4.1 on Debian Jessie: freeze using "xl create" on second VM




>
> El 23/2/16 a les 21:30, Brandon Adkins ha escrit:
> >
> >
> >>
> >> El 22/2/16 a les 20:11, Brandon Adkins ha escrit:
> >>>
> >>>> El 22/2/16 a les 18:40, Brandon Adkins ha escrit:
> >>>>>
> >>>>>>
> >>>>>> El 19/2/16 a les 17:38, Brandon Adkins ha escrit:
> >>>>>>> I am having issues migrating my Xen setup from Debian Wheezy to Debian Jessie. The errors I see are not very concrete, so it doesn't give me much details.Here is what I am seeing:First, when I make a new Xen VM and connect to its console, I see "xenconsole: Could not open tty `': No such file or directory". It seems to boot fine after that, but after returning to Dom0, the console is kinda jacked up. This mimics exactly the issue described athttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794071 . Apparently there is a fix as of Oct 2015, but just not released on Jessie yet? Will that be released on Jessie sometime soon?Second, and this is the bigger issue: I can make 1 VM like this fine, but when I try to make a second one, it just seems to hang. If I use "xl create -c" so I see the console the whole time, I see the bootloader menu appear, and when the kernel selection is finished, it prints the error about the tty, and then nothing else. I don't know if the tty issue is re
> la
> >> te
> >>>> d
> >>>>>> at all."xl list" shows the second VM in the list, but it is marked as only having 1 vcpu. If I try to change this with "xl vcpu-set 2" (or any number higher), it is just ignored. I can't find any log files to indicate there is a problem at all. The VM just doesn't start. It was created from the same template as the first VM, so VM1 and VM2 are basically copies of each other, yet only the first one will run.I have no idea if I am doing something wrong or if this is a bug in Jessie. I will say that everything that I am doing now has worked perfectly for years using Debian Wheezy. The only change I needed was to start using "xl" instead of "xm".Anyone have clues on figuring this out?Thanks,Beau
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> Can you please paste the config file of those two VMs?
> >>>>>>
> >>>>>> Also, can you paste the output when creating them with:
> >>>>>>
> >>>>>> # xl -vvv create <config.cfg>
> >>>>>>
> >>>>>> Are there any errors displayed by Xen or Linux in the serial console or
> >>>>>> dmesg?
> >>>>>>
> >>>>>> Roger.
> >>>>>
> >>>>
> >>>> Hello,
> >>>>
> >>>> I'm sorry but your MUA seems to completely mangle the messages, and I
> >>>> cannot parse the output you posted. Could you try to use a different MUA
> >>>> that adds proper line breaks?
> >>>>
> >>>> Thanks, Roger.
> >>>>
> >>> Trying again
> >>>
> >>> Hi Roger,
> >>> Thanks for the help. Here are the 2 config files:guest0:bootloader = '/usr/lib/xen-4.4/bin/pygrub'vcpus = '6'cpus = '2-7'memory = '6000'root = '/dev/xvda2 ro'disk = [ 'phy:/dev/xenvolgroup/guest0disk,xvda2,w', 'phy:/dev/xenvolgroup/guest0swap,xvda1,w' ]name = 'guest0'vif = [ 'mac=AE:57:EC:01:54:8C,bridge=xenbr0' ]-----------------------------------------guest1:bootloader = '/usr/lib/xen-4.4/bin/pygrub'vcpus = '6'cpus = '2-7'memory = '6000'root = '/dev/xvda2 ro'disk = [ 'phy:/dev/xenvolgroup/guest1disk,xvda2,w', 'phy:/dev/xenvolgroup/guest1swap,xvda1,w' ]name = 'guest1'vif = [ 'mac=76:7B:D6:6F:34:C6,bridge=xenbr0' ]---------------------------------------------Output of xl -vvv create:libxl: debug: libxl_create.c:1342:do_domain_create: ao 0x22ff8b0: create: how=(nil) callback=(nil) poller=0x22ff9b0libxl: debug: libxl_device.c:251:libxl__device_disk_set_back
> en
> >> d: Disk vdev=xvda2 spec.backend=unknownlibxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phylibxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknownlibxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phylibxl: debug: libxl_create.c:797:initiate_domain_create: running bootloaderlibxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phylibxl: debug: libxl.c:2634:libxl__device_disk_local_initiate_attach: locally attaching PHY disk /dev/xenvolgroup/guest1disklibxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: /usr/lib/xen-4.4/bin/pygrublibxl: debug: libxl_create.c:1356:do_domain_create: ao 0x22ff8b0: inprogress: poller=0x22ff9b0, flags=ilibxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x22f5178 wpath=/local/domain/127 token=3/0: register slotnum=3libxl: deb
> ug:
> >> libxl_event.c:1761:libxl__ao_progress_report: ao 0x22ff8b0: progress report: ignoredlibxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib/xen-4.4/bin/pygrublibxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib/xen-4.4/bin/pygrublibxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --args=root=/dev/xvda2 rolibxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.127.outlibxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.127.dlibxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /dev/xenvolgroup/guest1disklibxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x22f5178 wpath=/local/domain/127 token=3/0: event epath=/local/domain/127libxl: debug: libxl_bootloader.c:
> 634
> >> :bootloader_finished: bootloader completedlibxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained kernel /var/run/xen/bootloader.127.d/boot_kernel.m3pOqrlibxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained ramdisk /var/run/xen/bootloader.127.d/boot_ramdisk.DQT0dDlibxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained args root=/dev/xvda2 ro crashkernel=auto root=/dev/xvda2 rolibxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader execution successfullibxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x22f5178wpath=/local/domain/127 token=3/0: deregister slotnum=3domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvda2 ro crashkernel=auto root=/dev/xvda2 ro ", features="(null)"libxl: debug: libxl_dom.c:364:libxl__build_pv: pv kernel mapped 1 path /var/run/xen/bootloader.127.d/boot_kernel.m3pOqrdomainbuilder: detail: xc_dom_kernel_mem:
> ca
> >> lleddomainbuilder: detail: xc_dom_ramdisk_mem: calleddomainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64domainbuilder: detail: xc_dom_parse_image: calleddomainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...domainbuilder: detail: loader probe faileddomainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...domainbuilder: detail: xc_dom_malloc : 8861 kBdomainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x3e7bbc -> 0x8a7564domainbuilder: detail: loader probe OKxc: detail: elf_parse_binary: phdr: paddr=0x400000 memsz=0x66f000xc: detail: elf_parse_binary: phdr: paddr=0xa6f000 memsz=0x381000xc: detail: elf_parse_binary: memory: 0x400000 -> 0xdf0000xc: detail: elf_xen_parse_note: GUEST_OS = "linux"xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"xc: detail: elf_xen_parse_note: VIRT_BASE = 0xc0000000
> xc:
> >> detail: elf_xen_parse_note: ENTRY = 0xc0b11000xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xc0402000xc: detail: elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"xc: detail: elf_xen_parse_note: PAE_MODE = "yes"xc: detail: elf_xen_parse_note: LOADER = "generic"xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1xc: detail: elf_xen_parse_note: HV_START_LOW = 0xf5800000xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0xc: detail: elf_xen_addr_calc_check: addresses:xc: detail: virt_base = 0xc0000000xc: detail: elf_paddr_offset = 0x0xc: detail: virt_offset = 0xc0000000xc: detail: virt_kstart = 0xc0400000xc: detail: virt_kend = 0xc0df0000xc: detail: virt_entry = 0xc0b11000xc: detail: p2m_base = 0xffffffffffffffffdomainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_32p: 0xc0400000 -> 0xc0df0000domainbuilder: detail: xc_dom_mem_ini
> t:
> >> mem 6000 MB, pages 0x177000 pages, 4k eachdomainbuilder: detail: xc_dom_mem_init: 0x177000 pagesdomainbuilder: detail: xc_dom_boot_mem_init: calleddomainbuilder: detail: x86_compat: guest xen-3.0-x86_32p, address size 32domainbuilder: detail: xc_dom_malloc : 12000 kBdomainbuilder: detail: xc_dom_build_image: calleddomainbuilder: detail: xc_dom_alloc_segment: kernel : 0xc0400000 -> 0xc0df0000 (pfn 0x400 + 0x9f0 pages)domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x400+0x9f0 at 0x7fd07b610000xc: detail: elf_load_binary: phdr 0 at 0x7fd07b610000 -> 0x7fd07bc7f000xc: detail: elf_load_binary: phdr 1 at 0x7fd07bc7f000 -> 0x7fd07bdb2000domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xc0df0000 -> 0xc2ee0000 (pfn 0xdf0 + 0x20f0 pages)domainbuilder: detail: xc_dom_malloc : 197 kBdomainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xdf0+0x20f0 at 0x7fd079520000domainbuilder: detail: xc_dom_do_gunzip: unzip
> ok,
> >> 0xd262fe -> 0x20ef410domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xc2ee0000 -> 0xc34bc000 (pfn 0x2ee0 + 0x5dc pages)domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2ee0+0x5dc at 0x7fd0800d3000domainbuilder: detail: xc_dom_alloc_page : start info : 0xc34bc000 (pfn 0x34bc)domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xc34bd000 (pfn 0x34bd)domainbuilder: detail: xc_dom_alloc_page : console : 0xc34be000 (pfn 0x34be)domainbuilder: detail: nr_page_tables: 0x00000000ffffffff/32: 0x0000000000000000 -> 0x00000000ffffffff, 1 table(s)domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0x00000000c0000000 -> 0x00000000ffffffff, 1 table(s)domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0x00000000c0000000 -> 0x00000000c37fffff, 28 table(s)domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xc34bf000 -> 0xc34dd000 (pfn 0x34bf + 0x1e pages)domainbuilder: detail: xc_dom_pfn_to_ptr_retcount:
> do
> >> mU mapping: pfn 0x34bf+0x1e at 0x7fd084ad6000domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xc34dd000 (pfn 0x34dd)domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xc34de000domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xc3800000domainbuilder: detail: xc_dom_boot_image: calleddomainbuilder: detail: arch_setup_bootearly: doing nothingdomainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p <= matchesdomainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32pdomainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64domainbuilder: detail: xc_dom_update_guest_p2m: dst 32bit, pages 0x177000domainbuilder: detail: clear_page: pfn 0x34be, mfn 0x17cd7adomainbuilder: detail: clear_page: pfn 0x34bd, mfn 0x17cd7bdomainbuilde
> r:
> >> detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x34bc+0x1at 0x7fd084b31000domainbuilder: detail: start_info_x86_32: calleddomainbuilder: detail: setup_hypercall_page: vaddr=0xc0402000 pfn=0x402domainbuilder: detail: domain builder memory footprintdomainbuilder: detail: allocateddomainbuilder: detail: malloc : 21157 kBdomainbuilder: detail: anon mmap : 0 bytesdomainbuilder: detail: mappeddomainbuilder: detail: file mmap : 0 bytesdomainbuilder: detail: domU mmap : 48 MBdomainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xbf615domainbuilder: detail: shared_info_x86_32: calleddomainbuilder: detail: vcpu_x86_32: calleddomainbuilder: detail: vcpu_x86_32: cr3: pfn 0x34bf mfn 0x17cd79domainbuilder: detail: launch_vm: called, ctxt=0x7fd084ad4004domainbuilder: detail: xc_dom_release: calledlibxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phylibxl: debug:
> li
> >> bxl_event.c:570:libxl__ev_xswatch_register: watch w=0x22f6068 wpath=/local/domain/0/backend/vbd/127/51714/state token=3/1: register slotnum=3libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=phylibxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=phylibxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x22f7df8 wpath=/local/domain/0/backend/vbd/127/51713/state token=2/2: register slotnum=2libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x22f6068 wpath=/local/domain/0/backend/vbd/127/51714/state token=3/1: event epath=/local/domain/0/backend/vbd/127/51714/statelibxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/127/51714/state wanted state 2 oklibxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x22f6068wpath=/local/domain/0/backend/vbd/127/51714/state token=3/1: deregister slotnum=3libxl: debug: libxl_event.c:618:li
> bxl
> >> __ev_xswatch_deregister: watch w=0x22f6068: deregister unregisteredlibxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block addlibxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/127/51714/state token=3/1: empty slotlibxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x22f7df8 wpath=/local/domain/0/backend/vbd/127/51713/state token=2/2: event epath=/local/domain/0/backend/vbd/127/51713/statelibxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/127/51713/state wanted state 2 oklibxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x22f7df8wpath=/local/domain/0/backend/vbd/127/51713/state token=2/2: deregister slotnum=2libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22f7df8: deregister unregisteredlibxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block addlibxl: debug: libxl_event.c:
> 483
> >> :watchfd_callback: watch epath=/local/domain/0/backend/vbd/127/51713/state token=2/2: empty slotlibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22f60f0: deregister unregisteredlibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22f60f0: deregister unregisteredlibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22f7e80: deregister unregisteredlibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22f7e80: deregister unregisteredlibxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x22fa7c8 wpath=/local/domain/0/backend/vif/127/0/state token=2/3: register slotnum=2libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x22fa7c8 wpath=/local/domain/0/backend/vif/127/0/state token=2/3: event epath=/local/domain/0/backend/vif/127/0/statelibxl: debug: libxl_event.c:657:devstate_watch_callback: backend /local/domain/0/backend/vif/127/0/state wanted state 2 still waiting state 1libxl: debug:
> lib
> >> xl_event.c:514:watchfd_callback: watch w=0x22fa7c8 wpath=/local/domain/0/backend/vif/127/0/state token=2/3: event epath=/local/domain/0/backend/vif/127/0/statelibxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vif/127/0/state wanted state 2 oklibxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x22fa7c8wpath=/local/domain/0/backend/vif/127/0/state token=2/3: deregister slotnum=2libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22fa7c8: deregister unregisteredlibxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge onlinelibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22fa850: deregister unregisteredlibxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x22fa850: deregister unregisteredlibxl: debug: libxl_event.c:1761:libxl__ao_progress_report: ao 0x22ff8b0: progress report: ignoredlibxl: debug: libxl_event.c:1591:libxl__a
> o_c
> >> omplete: ao 0x22ff8b0: complete, rc=0libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x22ff8b0: destroyxc: debug: hypercall buffer: total allocations:470 total releases:470xc: debug: hypercall buffer: current allocations:0 maximum allocations:3xc: debug: hypercall buffer: cache current size:2xc: debug: hypercall buffer: cache hits:462 misses:2 toobig:6Parsing config from guest.cfg-----------------------------------------------Looking into dmesg, the only interesting thing I see related to this are these 2 lines everytime I try to start the VM: xen:balloon: Cannot add additional memory (-17) xen:balloon: Cannot add additional memory (-17)
> >>> Thanks!Beau
> >>
> >> Hello,
> >>
> >> Can you also post the output of xl info before creating the guests?
> >> AFAICT you are
> >> giving each guest 6000MB of RAM; are you sure you have enough RAM for
> >> Dom0 plus this two guests?
> >>
> >> From the output you provided, I can only see the trace of one guest
> >> booting [0] (one execution of xl), what happens with the other guest?
> >>
> >> Roger.
> >>
> >> [0]
> >> http://lists.xenproject.org/archives/html/xen-users/2016-02/msg00116.html
> >>
> >
> > Roger, yes thats right. Both guests get 6000MB of RAM. The system has 16GB of RAM, and I have allocated 4GB to the host.
> > Guest0 boots and operates perfectly. It is only guest1 that has issues. It does act like maybe there isnt enough RAM for the second VM, but from my calculations, it looks like it should be fine. Here is the output of "xl info" with no guests running
>
> You haven't provided the log when creating the second guest. I would try
> reducing the memory assigned to the second guest, if you have 4GB
> assigned to Dom0 and 6GB assigned to each guest you don't leave any
> memory for Xen (Xen takes over some memory in order to run).
>
> Roger.

Hi Roger,

The log messages I entered before are for the second guest. I didn't include any logs for the second guest since that one works perfectly.

But maybe I am allocating my RAM too tightly? The system has 16 GB of RAM, or 16,384MB. I allocated 4096MB to the host. That leaves me with 12,228MB. Each VM gets 6000, so thats 228MB unaccounted for. Is there some minimum amount that needs to remain above that?

Looking at "xl info" with both guests started, it says my total memory is 16,353, and that my free memory is 45.

I also just tried reducing the RAM for both guests down to 5500. I got the same error when starting the second guest. Thats about 1 GB of unallocated space, so I don't think thats my issue.

I also just tried stopping both guests, and starting guest1 first, and guest0 second. Guest1 came up fine, and now guest0 is stuck. Its definitely related to the second vm that starts, and not a specific configuration.

I am really at a loss of where to look. Is there anything in the config that would cause the CPUs to get assigned to the first guest, so the second one is left with no CPUs?

Best,
Beau
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.