> Subject: Re: [Xen-users] Xen 4.4.1 on Debian Jessie: freeze using "xl create" on second VM
> To: badkins79@xxxxxxx; xen-users@xxxxxxxxxxxxx
> From: roger.pau@xxxxxxxxxx
> Date: Fri, 26 Feb 2016 09:32:13 +0100
>
> El 26/2/16 a les 6:25, Brandon Adkins ha escrit:
> >
> >
> >>
> >> El 24/2/16 a les 18:13, Brandon Adkins ha escrit:
> >> > Hi Roger,
> >> > The log messages I entered before are for the second guest. I didn't
> > include any logs for the second guest since that one works perfectly.
> >> > But maybe I am allocating my RAM too tightly? The system has 16 GB
> > of RAM, or 16,384MB. I allocated 4096MB to the host. That leaves me with
> > 12,228MB. Each VM gets 6000, so thats 228MB unaccounted for. Is there
> > some minimum amount that needs to remain above that?
> >> > Looking at "xl info" with both guests started, it says my total
> > memory is 16,353, and that my free memory is 45.
> >> > I also just tried reducing the RAM for both guests down to 5500. I
> > got the same error when starting the second guest. Thats about 1 GB of
> > unallocated space, so I don't think thats my issue.
> >> > I also just tried stopping both guests, and starting guest1 first,
> > and guest0 second. Guest1 came up fine, and now guest0 is stuck. Its
> > definitely related to the second vm that starts, and not a specific
> > configuration.
> >> > I am really at a loss of where to look. Is there anything in the
> > config that would cause the CPUs to get assigned to the first guest, so
> > the second one is left with no CPUs?
> >> > Best,Beau
> >> >
> >>
> >> I don't see any error in the trace, AFAICT libxl has correctly created
> >> the domain. Can you paste the output of `xl list` after creating both
> >> guests?
> >>
> >> Roger.
> >
> > Hi Roger,
> >
> > Yes, I can't find any errors either, which makes debugging it almost
> > impossible for me. Here are some outputs:
> >>xl list
> > Name ID Mem VCPUs State
> > Time(s)
> > Domain-0 0 4087 2 r-----
> > 3308.0
> > guest1 132 5500 6 -b----
> > 533.0
> > guest0 133 5500 1 r-----
> > 130103.4
> >
> > Interesting to see the nonfunctional guest0 has such a high Time column.
> > Its like its spinning in a tight loop, while the working guest1 is
> > running fine. Note that this is the output of my last test setup, where
> > I gave both vms 5500MB of RAM, and tried starting guest1 first.
> >
> > xl vcpu-list
> > Name ID VCPU CPU State Time(s) CPU
> > Affinity
> > Domain-0 0 0 0 -b- 1554.4 0
> > Domain-0 0 1 1 r-- 1749.6 1
> > Domain-0 0 2 - --p 0.5 2
> > Domain-0 0 3 - --p 0.6 3
> > Domain-0 0 4 - --p 0.4 4
> > Domain-0 0 5 - --p 0.6 5
> > Domain-0 0 6 - --p 1.7 6
> > Domain-0 0 7 - --p 0.8 7
> > guest1 132 0 7 -b- 220.4 2-7
> > guest1 132 1 5 -b- 86.0 2-7
> > guest1 132 2 4 -b- 73.5 2-7
> > guest1 132 3 2 -b- 45.2 2-7
> > guest1 132 4 6 -b- 61.8 2-7
> > guest1 132 5 5 -b- 46.3 2-7
> > guest0 133 0 3 r-- 130279.1 2-7
> > guest0 133 1 - --p 0.0 2-7
> > guest0 133 2 - --p 0.0 2-7
> > guest0 133 3 - --p 0.0 2-7
> > guest0 133 4 - --p 0.0 2-7
> > guest0 133 5 - --p 0.0 2-7
> >
> > For some reason, even though I assigned 6 CPUs to both VMs, only the
> > first one started seems to get it right.
> >
> > I believe I must have some sort of screwed up Xen configuration. Surely
> > I am not the first person to try using Xen on Debian Jessie.
>
> Something seems wrong regarding guest0, but it looks like an issue in
> the guest kernel. Can you paste the output of xl dmesg after creating
> both guests?
>
> AFAICT from the traces you have provided, you are trying to create 32bit
> PV guests. Do they boot fine if you convert them to HVM? I would for
> example try to change the config files so they look like:
>
> guest0:
> builder='hvm'
> vcpus = '6'
> cpus = '2-7'
> memory = '6000'
> disk = [ 'phy:/dev/xenvolgroup/guest0disk,xvda2,w',
> 'phy:/dev/xenvolgroup/guest0swap,xvda1,w' ]
> name = 'guest0'
> vif = [ 'mac=AE:57:EC:01:54:8C,bridge=xenbr0' ]
> _on_poweroff_ = 'destroy'
> _on_reboot_ = 'restart'
> _on_crash_ = 'restart'
>
> vnc=1
> -----------------------------------------
> guest1:
> builder='hvm'
> vcpus = '6'
> cpus = '2-7'
> memory = '6000'
> disk = [ 'phy:/dev/xenvolgroup/guest1disk,xvda2,w',
> 'phy:/dev/xenvolgroup/guest1swap,xvda1,w' ]
> name = 'guest1'
> vif = [ 'mac=76:7B:D6:6F:34:C6,bridge=xenbr0' ]
> _on_poweroff_ = 'destroy'
> _on_reboot_ = 'restart'
> _on_crash_ = 'restart'
>
> vnc=1
> -----------------------------------------
>
> You can then attach to the guest display using a VNC client.
>
> Roger.
Roger. This is a 64bit machine loaded with 64bit debian jessie. The guest VMs are 32bit CentOS 6. However, both guest0 and guest1 are identical. What could happen in the guest kernel that would allow one instance to run just fine, but the second one to freeze on boot.
I tried the "xl dmesg", but its difficult to tell what output in there is related to what guest. But the end of the output contained lots of the following 2 messages. Not sure if these mean anything:
(XEN) traps.c:2514:d132 Domain attempted WRMSR 0000000000000140 from 0x0000000000000001 to 0x0000000000000000.
(XEN) event_fifo.c:50:d0 domain 0, port 19 already on a queue
I tried using hvm to build the VMs, but that failed even on the first VM. My output was:
Parsing config from /home/lpwcm/xen/guest0/guest.cfg
libxl: error: libxl_dm.c:1393:device_model_spawn_outcome: domain 138 device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1186:domcreate_devmodel_started: device model did not start: -3
libxl: error: libxl_dm.c:1497:kill_device_model: Device Model already exited
This led me to http://lists.xenproject.org/archives/html/xen-devel/2013-06/msg03008.html . This thread makes me think I need some extra packages installed which are not installed automatically as xen dependencies. Could this be causing my issue in the PV case? The only packages I had specified in my apt-get line was "xen-linux-system" so I got that and all its dependencies. Also, does it matter that I also installed sysvinit-core so the host uses that instead of the new systemd init that is the default on jessie?
Again, I think its worth mentioning that these exact same images and configurations worked correctly on Debian Wheezy.
Best regards,
Beau
