Xen project Mailing List

>
> El 27/2/16 a les 0:47, Brandon Adkins ha escrit:
> >
> >
> >> Subject: Re: [Xen-users] Xen 4.4.1 on Debian Jessie: freeze using "xl
> > create" on second VM
> >> To: badkins79@xxxxxxx; xen-users@xxxxxxxxxxxxx
> >> From: roger.pau@xxxxxxxxxx
> >> Date: Fri, 26 Feb 2016 09:32:13 +0100
> >>
> >> El 26/2/16 a les 6:25, Brandon Adkins ha escrit:
> >> >
> >> >
> >> >>
> >> >> El 24/2/16 a les 18:13, Brandon Adkins ha escrit:
> >> >> > Hi Roger,
> >> >> > The log messages I entered before are for the second guest. I didn't
> >> > include any logs for the second guest since that one works perfectly.
> >> >> > But maybe I am allocating my RAM too tightly? The system has 16 GB
> >> > of RAM, or 16,384MB. I allocated 4096MB to the host. That leaves me with
> >> > 12,228MB. Each VM gets 6000, so thats 228MB unaccounted for. Is there
> >> > some minimum amount that needs to remain above that?
> >> >> > Looking at "xl info" with both guests started, it says my total
> >> > memory is 16,353, and that my free memory is 45.
> >> >> > I also just tried reducing the RAM for both guests down to 5500. I
> >> > got the same error when starting the second guest. Thats about 1 GB of
> >> > unallocated space, so I don't think thats my issue.
> >> >> > I also just tried stopping both guests, and starting guest1 first,
> >> > and guest0 second. Guest1 came up fine, and now guest0 is stuck. Its
> >> > definitely related to the second vm that starts, and not a specific
> >> > configuration.
> >> >> > I am really at a loss of where to look. Is there anything in the
> >> > config that would cause the CPUs to get assigned to the first guest, so
> >> > the second one is left with no CPUs?
> >> >> > Best,Beau
> >> >> >
> >> >>
> >> >> I don't see any error in the trace, AFAICT libxl has correctly created
> >> >> the domain. Can you paste the output of `xl list` after creating both
> >> >> guests?
> >> >>
> >> >> Roger.
> >> >
> >> > Hi Roger,
> >> >
> >> > Yes, I can't find any errors either, which makes debugging it almost
> >> > impossible for me. Here are some outputs:
> >> >>xl list
> >> > Name ID Mem VCPUs State
> >> > Time(s)
> >> > Domain-0 0 4087 2 r-----
> >> > 3308.0
> >> > guest1 132 5500 6 -b----
> >> > 533.0
> >> > guest0 133 5500 1 r-----
> >> > 130103.4
> >> >
> >> > Interesting to see the nonfunctional guest0 has such a high Time column.
> >> > Its like its spinning in a tight loop, while the working guest1 is
> >> > running fine. Note that this is the output of my last test setup, where
> >> > I gave both vms 5500MB of RAM, and tried starting guest1 first.
> >> >
> >> > xl vcpu-list
> >> > Name ID VCPU CPU State Time(s) CPU
> >> > Affinity
> >> > Domain-0 0 0 0 -b- 1554.4 0
> >> > Domain-0 0 1 1 r-- 1749.6 1
> >> > Domain-0 0 2 - --p 0.5 2
> >> > Domain-0 0 3 - --p 0.6 3
> >> > Domain-0 0 4 - --p 0.4 4
> >> > Domain-0 0 5 - --p 0.6 5
> >> > Domain-0 0 6 - --p 1.7 6
> >> > Domain-0 0 7 - --p 0.8 7
> >> > guest1 132 0 7 -b- 220.4 2-7
> >> > guest1 132 1 5 -b- 86.0 2-7
> >> > guest1 132 2 4 -b- 73.5 2-7
> >> > guest1 132 3 2 -b- 45.2 2-7
> >> > guest1 132 4 6 -b- 61.8 2-7
> >> > guest1 132 5 5 -b- 46.3 2-7
> >> > guest0 133 0 3 r-- 130279.1 2-7
> >> > guest0 133 1 - --p 0.0 2-7
> >> > guest0 133 2 - --p 0.0 2-7
> >> > guest0 133 3 - --p 0.0 2-7
> >> > guest0 133 4 - --p 0.0 2-7
> >> > guest0 133 5 - --p 0.0 2-7
> >> >
> >> > For some reason, even though I assigned 6 CPUs to both VMs, only the
> >> > first one started seems to get it right.
> >> >
> >> > I believe I must have some sort of screwed up Xen configuration. Surely
> >> > I am not the first person to try using Xen on Debian Jessie.
> >>
> >> Something seems wrong regarding guest0, but it looks like an issue in
> >> the guest kernel. Can you paste the output of xl dmesg after creating
> >> both guests?
> >>
> >> AFAICT from the traces you have provided, you are trying to create 32bit
> >> PV guests. Do they boot fine if you convert them to HVM? I would for
> >> example try to change the config files so they look like:
> >>
> >> guest0:
> >> builder='hvm'
> >> vcpus = '6'
> >> cpus = '2-7'
> >> memory = '6000'
> >> disk = [ 'phy:/dev/xenvolgroup/guest0disk,xvda2,w',
> >> 'phy:/dev/xenvolgroup/guest0swap,xvda1,w' ]
> >> name = 'guest0'
> >> vif = [ 'mac=AE:57:EC:01:54:8C,bridge=xenbr0' ]
> >> _on_poweroff_ = 'destroy'
> >> _on_reboot_ = 'restart'
> >> _on_crash_ = 'restart'
> >>
> >> vnc=1
> >> -----------------------------------------
> >> guest1:
> >> builder='hvm'
> >> vcpus = '6'
> >> cpus = '2-7'
> >> memory = '6000'
> >> disk = [ 'phy:/dev/xenvolgroup/guest1disk,xvda2,w',
> >> 'phy:/dev/xenvolgroup/guest1swap,xvda1,w' ]
> >> name = 'guest1'
> >> vif = [ 'mac=76:7B:D6:6F:34:C6,bridge=xenbr0' ]
> >> _on_poweroff_ = 'destroy'
> >> _on_reboot_ = 'restart'
> >> _on_crash_ = 'restart'
> >>
> >> vnc=1
> >> -----------------------------------------
> >>
> >> You can then attach to the guest display using a VNC client.
> >>
> >> Roger.
> >
> > Roger. This is a 64bit machine loaded with 64bit debian jessie. The
> > guest VMs are 32bit CentOS 6. However, both guest0 and guest1 are
> > identical. What could happen in the guest kernel that would allow one
> > instance to run just fine, but the second one to freeze on boot.
> >
> > I tried the "xl dmesg", but its difficult to tell what output in there
> > is related to what guest. But the end of the output contained lots of
> > the following 2 messages. Not sure if these mean anything:
> > (XEN) traps.c:2514:d132 Domain attempted WRMSR 0000000000000140 from
> > 0x0000000000000001 to 0x0000000000000000.
> > (XEN) event_fifo.c:50:d0 domain 0, port 19 already on a queue
> >
> > I tried using hvm to build the VMs, but that failed even on the first
> > VM. My output was:
> > Parsing config from /home/lpwcm/xen/guest0/guest.cfg
> > libxl: error: libxl_dm.c:1393:device_model_spawn_outcome: domain 138
> > device model: spawn failed (rc=-3)
> > libxl: error: libxl_create.c:1186:domcreate_devmodel_started: device
> > model did not start: -3
> > libxl: error: libxl_dm.c:1497:kill_device_model: Device Model already exited
>
> Can you check the logs in /var/log/xen/? You should see some logs with
> the following nomenclature "qemu-dm-<guest_name>.log"? Are there any
> errors reported there?
>
> Also, can you try again but with a higher debug value (using xl -vvv
> create ...) and post the result?
>
> Roger.
>

Hey Roger,

There was one entry in the qemu-dm logfile:

qemu-system-i386: -drive file=/dev/xenvolgroup/guest0swap,if=ide,index=0,media=disk,format=raw,cache=writeback: drive with bus=0, unit=0 (index=0) exists

Here is the output using -vvv:

Parsing config from /home/lpwcm/xen/guest0/guest.cfg

libxl: debug: libxl_create.c:1342:do_domain_create: ao 0x23a38b0: create: how=(nil) callback=(nil) poller=0x23a3950

libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=unknown

libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phy

libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown

libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phy

libxl: debug: libxl_create.c:797:initiate_domain_create: running bootloader

libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV domain, skipping bootloader

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x2399188: deregister unregistered

xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x59658

xc: detail: elf_parse_binary: memory: 0x100000 -> 0x159658

xc: detail: VIRTUAL MEMORY ARRANGEMENT:

Loader: 0000000000100000->0000000000159658

Modules: 0000000000000000->0000000000000000

TOTAL: 0000000000000000->0000000157400000

ENTRY ADDRESS: 0000000000100648

xc: detail: PHYSICAL MEMORY ALLOCATION:

4KB PAGES: 0x0000000000000200

2MB PAGES: 0x00000000000004b9

1GB PAGES: 0x0000000000000003

xc: detail: elf_load_binary: phdr 0 at 0x7f9c2de99000 -> 0x7f9c2dee960d

libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phy

libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x239a998 wpath=/local/domain/0/backend/vbd/139/51714/state token=3/0: register slotnum=3

libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=phy

libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x239b938 wpath=/local/domain/0/backend/vbd/139/51713/state token=2/1: register slotnum=2

libxl: debug: libxl_create.c:1356:do_domain_create: ao 0x23a38b0: inprogress: poller=0x23a3950, flags=i

libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x239a998 wpath=/local/domain/0/backend/vbd/139/51714/state token=3/0: event epath=/local/domain/0/backend/vbd/139/51714/state

libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/139/51714/state wanted state 2 ok

libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x239a998 wpath=/local/domain/0/backend/vbd/139/51714/state token=3/0: deregister slotnum=3

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239a998: deregister unregistered

libxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block add

libxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/139/51714/state token=3/0: empty slot

libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x239b938 wpath=/local/domain/0/backend/vbd/139/51713/state token=2/1: event epath=/local/domain/0/backend/vbd/139/51713/state

libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/139/51713/state wanted state 2 ok

libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x239b938 wpath=/local/domain/0/backend/vbd/139/51713/state token=2/1: deregister slotnum=2

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239b938: deregister unregistered

libxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block add

libxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/139/51713/state token=2/1: empty slot

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239aa20: deregister unregistered

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239b9c0: deregister unregistered

libxl: debug: libxl_dm.c:1325:libxl__spawn_local_dm: Spawning device-model /usr/bin/qemu-system-i386 with arguments:

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: /usr/bin/qemu-system-i386

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -xen-domid

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: 139

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -chardev

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-139,server,nowait

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -mon

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -nodefaults

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -name

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: guest0

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -vnc

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: 127.0.0.1:0,to=99

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -display

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: none

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -device

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: cirrus-vga

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -global

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: vga.vram_size_mb=8

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -boot

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: order=cda

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -smp

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: 6,maxcpus=6

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -device

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: rtl8139,id=nic0,netdev=net0,mac=ae:57:ec:01:54:8c

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -netdev

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: type=tap,id=net0,ifname=vif139.0-emu,script=no,downscript=no

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -machine

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: xenfv

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -m

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: 5492

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -drive

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: file=/dev/xenvolgroup/guest0disk,if=ide,index=0,media=disk,format=raw,cache=writeback

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: -drive

libxl: debug: libxl_dm.c:1327:libxl__spawn_local_dm: file=/dev/xenvolgroup/guest0swap,if=ide,index=0,media=disk,format=raw,cache=writeback

libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x23993c0 wpath=/local/domain/0/device-model/139/state token=2/2: register slotnum=2

libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x23993c0 wpath=/local/domain/0/device-model/139/state token=2/2: event epath=/local/domain/0/device-model/139/state

libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x23993c0 wpath=/local/domain/0/device-model/139/state token=2/2: deregister slotnum=2

libxl: error: libxl_dm.c:1393:device_model_spawn_outcome: domain 139 device model: spawn failed (rc=-3)

libxl: error: libxl_create.c:1186:domcreate_devmodel_started: device model did not start: -3

libxl: error: libxl_dm.c:1497:kill_device_model: Device Model already exited

libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x239f718 wpath=/local/domain/0/backend/vbd/139/51714/state token=2/3: register slotnum=2

libxl: debug: libxl_event.c:570:libxl__ev_xswatch_register: watch w=0x239f998 wpath=/local/domain/0/backend/vbd/139/51713/state token=3/4: register slotnum=3

libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x239f718 wpath=/local/domain/0/backend/vbd/139/51714/state token=2/3: event epath=/local/domain/0/backend/vbd/139/51714/state

libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/139/51714/state wanted state 6 ok

libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x239f718 wpath=/local/domain/0/backend/vbd/139/51714/state token=2/3: deregister slotnum=2

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239f718: deregister unregistered

libxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove

libxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/139/51714/state token=2/3: empty slot

libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x239f998 wpath=/local/domain/0/backend/vbd/139/51713/state token=3/4: event epath=/local/domain/0/backend/vbd/139/51713/state

libxl: debug: libxl_event.c:653:devstate_watch_callback: backend /local/domain/0/backend/vbd/139/51713/state wanted state 6 ok

libxl: debug: libxl_event.c:606:libxl__ev_xswatch_deregister: watch w=0x239f998 wpath=/local/domain/0/backend/vbd/139/51713/state token=3/4: deregister slotnum=3

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239f998: deregister unregistered

libxl: debug: libxl_device.c:1023:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove

libxl: debug: libxl_event.c:483:watchfd_callback: watch epath=/local/domain/0/backend/vbd/139/51713/state token=3/4: empty slot

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239f7a0: deregister unregistered

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239fa20: deregister unregistered

libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x239ef80: deregister unregistered

libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x23a38b0: complete, rc=-3

libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x23a38b0: destroy

xc: debug: hypercall buffer: total allocations:1381 total releases:1381

xc: debug: hypercall buffer: current allocations:0 maximum allocations:2

xc: debug: hypercall buffer: cache current size:2

xc: debug: hypercall buffer: cache hits:1377 misses:2 toobig:2

Best regards,

Beau

Re: [Xen-users] Xen 4.4.1 on Debian Jessie: freeze using "xl create" on second VM