[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Weird ram/dom-u limit problem




Hello all,

I've been having a very odd problem the last few days and I'm at a loss on how to proceed, so I thought I'd ask the user community. I'm hitting what *appears* to be some kind of wall on either the number of vm's or the amount of ram they're using (personally I think it's ram, but I don't see how).

We have two identical CentOS 5.4 xen servers (xen-3.0.3-94.el5_4.2) in production with 7 vm's and dom0 on each server. I'll post the full config on the bottom of the email, but here are the basics.

xentop - 13:58:32   Xen 3.1.2-164.6.1.el5
8 domains: 1 running, 7 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16776420k total, 5582324k used, 11194096k free    CPUs: 4 @ 3000MHz

dom0_mem=1g
loops = 64

Now, if I attempt to install an 8th vm with 512MB or ram on either server the install runs for awhile and then suddenly load on the xen server begins to grow and grow and grow. Going from a load average: 0.09, 0.15, 0.10 to load average: 10.58, 7.88, 4.70. The box itself begin to become very slow and bogged down. top shows that it's because of cpu iowait, but I can't find any particular reason for it.

top - 13:35:18 up 63 days, 13:54,  3 users,  load average: 10.58, 7.88, 4.70
Tasks: 196 total,   1 running, 195 sleeping,   0 stopped,   0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 24.9%id, 74.5%wa, 0.0%hi, 0.0%si, 0.7%st

75% is ridiculously high and it continues to get worse until the install finally barely finishes after much longer than normal. An average install takes 20 minutes, they're taking around 35-40 now and I fear that if I add yet another vm it will be even worse. Once the install finishes the box returns to normal and I can boot the new vm normally as well.

If I shutdown one of the existing vm's using 512MB of ram, then the install works perfectly and the load never goes crazy. Turn the 7th back on , and installing the 8th is horrible again. I can duplicate this on both servers which is very weird, but I can't figure out *why* it's happening. The fact that both servers have the problem leads me to believe its a config problem.

I'm leaning towards something running out of ram and swapping hard, which is why load takes awhile to start climbing and the cpu wait gets very high, yet I have *more* than enough memory allocated to everything. The installs are very io intensive, but I can't understand why that 8th vm is causing hell.

Is there some weird limit to the amount of vm's or ram I can allocate? Does dom0 need more than 1g and that's why it's getting cranky? Is there a config option I'm not aware of?

Any help would be appreciated, this is really worrying me. Thanks.

Swap:
# free
             total       used       free     shared    buffers     cached
Mem:       1048576    1039176       9400          0      10924     777816
-/+ buffers/cache:     250436     798140
Swap:     10482404        188   10482216

Full config:

# xm info
host                   : xen2.rutgers.edu
release                : 2.6.18-164.6.1.el5xen
version                : #1 SMP Tue Nov 3 16:48:13 EST 2009
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 3000
hw_caps : 178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory           : 16383
free_memory            : 10931
node_to_cpu            : node0:0-3
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-164.6.1.el5
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Nov  3 16:04:14 EST 2009
xend_config_format     : 2


--
================================
David Halik
System Administrator
OIT-CSS Rutgers University
dhalik@xxxxxxxxxxxxxxx
================================


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.