Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Tue, Apr 27, 2010 at 12:41:07AM -0700, Yuvraj Agarwal wrote:
>    Hi All,
> 
> 
> 
>    We are setting up a system with a large number of very small VMs for a
>    project. We worked through a number of limitations, including those
>    imposed by blktap2 devices and the number of dynamic IRQ (set kernel
>    config NR_CPUS) etc. After these changes we were able to get to  154 domUs
>    (!), but as soon as we start up the 155^th domU the system crashes. We
>    edited each domU config to have one blktap2 device (the disk image) and
>    two virtual network interfaces each.
> 
> 
> 
>    We are using the standard XEN 4.0 (stable/release version)  with the
>    2.6.31.13 (pvops) kernel.
> 
> 
> 
>    Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
>    but as far as we can see we don't quite know what might be going causing
>    the system to crash (no console access anymore and system becomes
>    unresponsive and needs to be power-cycled).  I have pasted only the
>    relevant bits of information (the last domU that did successfully start
>    and the next one that failed). It may be the case that all the log
>    messages weren't flushed before the system crashed...
> 
> 
> 
>    Does anyone know where this limit of 155 domU is coming from and how we
>    can fix/increase it?
> 

Please paste your dom0 grub.conf.
Are you using memory ballooning? 

-- Pasi

>    thanks
> 
>    Yuvraj

> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created 
> /dev/xen/blktap-2/blktap152 device
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created 
> /dev/xen/blktap-2/tapdev152 device
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: new interface: ring: 251, device: 
> 253, minor: 152
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: I/O queue driver: lio
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: block-aio 
> open('/home/xen/domains/testing-ss-155.ucsd.edu/disk.img')
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: 
> open(/home/xen/domains/testing-ss-155.ucsd.edu/disk.img) with O_DIRECT
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Image size: #012#011pre 
> sector_shift  [1073741824]#012#011post sector_shift [2097152]
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: opened image 
> /home/xen/domains/testing-ss-155.ucsd.edu/disk.img (1 users, state: 
> 0x00000001, type: 0)
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: VBD CHAIN:
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: 
> /home/xen/domains/testing-ss-155.ucsd.edu/disk.img: 0
> Apr 26 20:08:09  BlackBox logger: /etc/xen/scripts/block: add 
> XENBUS_PATH=backend/vbd/153/770
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: online 
> XENBUS_PATH=backend/vif/153/0
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful 
> vif-bridge online for vif153.0, bridge testbr.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing 
> backend/vif/153/0/hotplug-status connected to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing 
> backend/vbd/153/770/physical-device fd:98 to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing 
> backend/vbd/153/770/hotplug-status connected to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: online 
> XENBUS_PATH=backend/vif/153/1
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful 
> vif-bridge online for vif153.1, bridge dummy.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing 
> backend/vif/153/1/hotplug-status connected to xenstore.
> 
> 
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created 
> /dev/xen/blktap-2/blktap153 device
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created 
> /dev/xen/blktap-2/tapdev153 device
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: new interface: ring: 251, device: 
> 253, minor: 153
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: I/O queue driver: lio
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: block-aio 
> open('/home/xen/domains/testing-ss-156.ucsd.edu/disk.img')
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: 
> open(/home/xen/domains/testing-ss-156.ucsd.edu/disk.img) with O_DIRECT
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Image size: #012#011pre 
> sector_shift  [1073741824]#012#011post sector_shift [2097152]
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: opened image 
> /home/xen/domains/testing-ss-156.ucsd.edu/disk.img (1 users, state: 
> 0x00000001, type: 0)
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: VBD CHAIN:
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: 
> /home/xen/domains/testing-ss-156.ucsd.edu/disk.img: 0
> Apr 26 20:10:17  BlackBox logger: /etc/xen/scripts/block: add 
> XENBUS_PATH=backend/vbd/154/770
> Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing 
> backend/vbd/154/770/physical-device fd:99 to xenstore.
> Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing 
> backend/vbd/154/770/hotplug-status connected to xenstore.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online 
> XENBUS_PATH=backend/vif/154/0
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful 
> vif-bridge online for vif154.0, bridge testbr.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing 
> backend/vif/154/0/hotplug-status connected to xenstore.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online 
> XENBUS_PATH=backend/vif/154/1
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup 
> failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful 
> vif-bridge online for vif154.1, bridge dummy.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing 
> backend/vif/154/1/hotplug-status connected to xenstore.

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel