|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] [Fwd: high load average on hvm gentoo linux domU]
Hi All,
This is my first time on this (or in fact any as I recall!) list so be
gentle :)
I am seeing some peculiar load behaviour with a gentoo linux domu which
is acting as mailserver. The load average starts out fine but then grows
over the course of several hours topping out at around 120 (one hundred
and twenty!)
The strange thing is the machine seems responsive, imap connections
working ok, mail being delivered, snappy shell access etc
This makes me suspicious that the load average is more a reflection of
the vm load on the cpu slice its being given (ie its only getting say
0.01 of the cpu time, and would be running at say 0.11 if it was not
virtualised) maybe I'm way off though!
The setup is not overly complex:
# xm list
Name ID Mem VCPUs State
Time(s)
Domain-0 0 1708 1 r-----
1682.4
gentoo-spud 5 1024 1 -b----
2792.4
solaris 2 1024 1 -b----
1757.4
its a gentoo-linux dom0, its running on amd-vt hardware, do both domUs
are unmodified guests...
xen # cat /etc/xen/gentoo-spud
name = "gentoo-spud";
memory = 1024;
disk = [ "file:/var/xen/gentoo/spud/domU,xvda,w" ];
kernel = "/var/xen/gentoo/spud/vmlinuz";
root = "/dev/xvda ro";
vif = [ "mac=00:16:3E:00:00:12" ];
xen # cat /etc/xen/solaris.py
name = 'solaris'
memory = '1024'
disk = [ 'file:/var/xen/solaris/domU,0,w' , 'phy:/dev/sdb,1,w' ,
'phy:/dev/sdc,2,w' ,'phy:/dev/sdd,3,w' ]
vif = [ 'mac=00:16:3E:00:00:11' ]
The solaris vm is providing zfs on the 3 physical devices ie
solaris # zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 832G 200G 632G 24% ONLINE -
solaris # zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c0d1 ONLINE 0 0 0
c0d2 ONLINE 0 0 0
c0d3 ONLINE 0 0 0
solaris # zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 133G 412G 24.0K /tank
tank/mail 1.32G 412G 1.32G /export/zfs/mail
the tank/mail share is exported over nfs:
solaris # zfs get sharenfs tank/mail
NAME PROPERTY VALUE SOURCE
tank/mail sharenfs rw,anon=0 local
the linux box is running:
postfix
courier-imapd-ssl
courier-authlib
amavisd
spamassassin
clamav
mail is routed through postfix on port 25 to amavis on port 10024 which
takes care of spam and av then back into postfix on 10025 for local delivery
all domains/users are virtual on postfix with delivery to
/home/vmail/<user@domain>/
/home/vmail is a symbolic link to /mnt/zfs/mail
/mnt/zfs/mail is nfs mounted share from the solaris vm
through a process of elimination I found that restarting
courier-imapd-ssl would cause the loadavg to head back towards zero,
where it would stay for short while but then eventually start to
escalate again.
I can't find any explanation....
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 0 0 495912 14488 185348 0 0 34 64 399 296 2 1
96 0
0 0 0 494296 14488 185352 0 0 0 0 269 97 5 1
93 0
0 0 0 494296 14488 185352 0 0 0 0 207 82 0 0
99 0
0 0 0 494288 14488 185400 0 0 0 0 131 64 0 0
99 0
0 0 0 494288 14488 185452 0 0 0 0 128 60 0 0
100 0
0 0 0 494304 14488 185472 0 0 0 8 130 74 0 0
99 0
0 0 0 494304 14488 185472 0 0 0 0 144 67 0 0
100 0
0 0 0 494368 14488 185512 0 0 0 0 127 62 0 0
99 0
0 0 0 494260 14488 185512 0 0 0 0 141 73 0 0
100 0
0 0 0 494252 14488 185556 0 0 0 0 143 65 0 0
99 0
0 0 0 494252 14520 185576 0 0 0 32 116 68 0 0
99 0
0 0 0 494300 14520 185596 0 0 0 0 127 69 0 0
100 0
0 0 0 494192 14520 185596 0 0 0 0 141 68 0 0
99 0
0 0 0 494184 14520 185640 0 0 0 0 153 60 0 0
99 0
0 0 0 494184 14520 185640 0 0 0 0 122 59 0 0
100 0
0 0 0 494216 14520 185680 0 0 0 0 124 72 0 0
99 0
0 0 0 494108 14520 185680 0 0 0 0 123 67 0 0
100 0
0 0 0 494092 14520 185720 0 0 0 0 117 63 0 0
99 0
0 0 0 494092 14520 185720 0 0 0 0 119 64 0 0
100 0
0 0 0 494124 14520 185760 0 0 0 0 123 66 0 0
99 0
0 0 0 494124 14520 185760 0 0 0 0 125 70 0 0
99 0
0 0 0 494016 14520 185800 0 0 0 0 118 60 0 0
100 0
0 0 0 494016 14520 185800 0 0 0 0 93 42 0 0
97 0
0 0 0 494032 14520 185820 0 0 0 0 25 18 0 0
99 0
0 0 0 494032 14520 185820 0 0 0 0 11 12 0 0
100 0
0 0 0 494032 14520 185828 0 0 0 0 89 55 0 0
100 0
0 0 0 494032 14520 185828 0 0 0 0 138 68 0 1
98 0
0 0 0 494040 14520 185872 0 0 0 0 118 66 0 0
99 0
0 0 0 493932 14520 185872 0 0 0 0 122 63 0 0
99 0
0 0 0 493916 14520 185912 0 0 0 0 129 62 0 0
100 0
0 0 0 493916 14552 185932 0 0 0 44 126 76 0 0
99 0
0 0 0 493916 14552 185956 0 0 0 0 123 65 0 0
100 0
0 0 0 493808 14552 185956 0 0 0 0 119 58 0 0
99 0
0 0 0 493800 14552 185996 0 0 0 0 132 71 0 0
100 0
0 0 0 493800 14552 185996 0 0 0 0 120 62 0 0
99 0
0 0 0 493800 14552 186040 0 0 0 36 131 73 0 0
99 0
0 0 0 493692 14552 186040 0 0 0 0 133 68 0 0
100 0
0 0 0 493676 14552 186080 0 0 0 0 130 66 0 0
99 0
0 0 0 493676 14552 186080 0 0 0 0 125 63 0 0
99 0
0 0 0 493684 14552 186124 0 0 0 0 139 79 0 0
100 0
0 0 0 493684 14552 186124 0 0 0 0 126 68 0 0
99 0
0 0 0 493584 14552 186164 0 0 0 0 125 62 0 0
99 0
0 0 0 493584 14552 186164 0 0 0 0 127 60 0 0
100 0
0 0 0 493584 14552 186204 0 0 0 0 130 80 0 0
99 0
0 0 0 493584 14552 186204 0 0 0 0 132 69 0 0
99 0
0 0 0 493460 14564 186248 0 0 0 12 119 74 0 0
100 0
0 0 0 493460 14564 186248 0 0 0 0 129 68 0 0
99 0
0 0 0 493452 14564 186288 0 0 0 0 145 75 0 0
100 0
thought maybe it was nfs...?
# iostat -n 1
Linux 2.6.21-xen (spud) 05/28/08
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 10363.60 13.49 0.00
0.00 11.30 13.49
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 24.18 24.19 0.00
0.00 0.00 18.67
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 0.00 0.05 0.00
0.00 0.00 5.56
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 1.98 0.06 0.00
0.00 0.00 0.05
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 4137.37 0.06 0.00
0.00 0.00 0.07
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 12051.17 4.16 0.00
0.00 0.00 4.16
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 6151.21 7.25 0.00
0.00 0.00 7.26
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 15.48 15.49 0.00
0.00 0.00 15.49
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 26.00 26.00 0.00
0.00 0.00 26.00
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 26.98 27.00 0.00
0.00 0.00 21.00
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 26.79 26.79 0.00
0.00 0.00 29.09
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 20.99 21.00 0.00
0.00 0.00 18.77
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 21.20 21.24 0.00
0.00 0.00 27.23
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 0.00 0.05 0.00
0.00 0.00 0.05
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 1.98 0.05 0.00
0.00 0.00 0.05
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 0.00 0.05 0.00
0.00 0.00 0.05
Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
wBlk_dir/s rBlk_svr/s wBlk_svr/s
swanney:/export/zfs/mail 2.75 2.79 0.00
0.00 0.00 2.79
the machine is hardly begin touched - only two mailboxes on it, very
litle mail passing through.
any pointers would be really appreciated
Simon Bettison
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |