[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] [Fwd: high load average on hvm gentoo linux domU]



Hi All,

This is my first time on this (or in fact any as I recall!) list so be
gentle :)

I am seeing some peculiar load behaviour with a gentoo linux domu which
is acting as mailserver. The load average starts out fine but then grows
over the course of several hours topping out at around 120 (one hundred
and twenty!)

The strange thing is the machine seems responsive, imap connections
working ok, mail being delivered, snappy shell access etc

This makes me suspicious that the load average is more a reflection of
the vm load on the cpu slice its being given (ie its only getting say
0.01 of the cpu time, and would be running at say 0.11 if it was not
virtualised) maybe I'm way off though!

The setup is not overly complex:

# xm list
Name                                        ID   Mem VCPUs      State  
Time(s)
Domain-0                                     0  1708     1     r-----  
1682.4
gentoo-spud                                  5  1024     1     -b----  
2792.4
solaris                                      2  1024     1     -b----  
1757.4

its a gentoo-linux dom0, its running on amd-vt hardware, do both domUs
are unmodified guests...

xen # cat /etc/xen/gentoo-spud
name    = "gentoo-spud";
memory  = 1024;
disk = [ "file:/var/xen/gentoo/spud/domU,xvda,w" ];
kernel  = "/var/xen/gentoo/spud/vmlinuz";
root = "/dev/xvda ro";
vif = [ "mac=00:16:3E:00:00:12" ];

xen # cat /etc/xen/solaris.py
name = 'solaris'
memory = '1024'
disk = [ 'file:/var/xen/solaris/domU,0,w' , 'phy:/dev/sdb,1,w' ,
'phy:/dev/sdc,2,w' ,'phy:/dev/sdd,3,w' ]
vif = [ 'mac=00:16:3E:00:00:11' ]

The solaris vm is providing zfs on the 3 physical devices ie


solaris # zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank   832G   200G   632G    24%  ONLINE  -

solaris # zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c0d1    ONLINE       0     0     0
            c0d2    ONLINE       0     0     0
            c0d3    ONLINE       0     0     0

solaris # zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
tank         133G   412G  24.0K  /tank
tank/mail   1.32G   412G  1.32G  /export/zfs/mail

the tank/mail share is exported over nfs:

solaris # zfs get sharenfs tank/mail
NAME       PROPERTY  VALUE      SOURCE
tank/mail  sharenfs  rw,anon=0  local

the linux box is running:
postfix
courier-imapd-ssl
courier-authlib
amavisd
spamassassin
clamav

mail is routed through postfix on port 25 to amavis on port 10024 which
takes care of spam and av then back into postfix on 10025 for local delivery

all domains/users are virtual on postfix with delivery to
/home/vmail/<user@domain>/

/home/vmail is a symbolic link to /mnt/zfs/mail
/mnt/zfs/mail is nfs mounted share from the solaris vm

through a process of elimination I found that restarting
courier-imapd-ssl would cause the loadavg to head back towards zero,
where it would stay for short while but then eventually start to
escalate again.

I can't find any explanation....

# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 0  0      0 495912  14488 185348    0    0    34    64  399  296  2  1
96  0
 0  0      0 494296  14488 185352    0    0     0     0  269   97  5  1
93  0
 0  0      0 494296  14488 185352    0    0     0     0  207   82  0  0
99  0
 0  0      0 494288  14488 185400    0    0     0     0  131   64  0  0
99  0
 0  0      0 494288  14488 185452    0    0     0     0  128   60  0  0
100  0
 0  0      0 494304  14488 185472    0    0     0     8  130   74  0  0
99  0
 0  0      0 494304  14488 185472    0    0     0     0  144   67  0  0
100  0
 0  0      0 494368  14488 185512    0    0     0     0  127   62  0  0
99  0
 0  0      0 494260  14488 185512    0    0     0     0  141   73  0  0
100  0
 0  0      0 494252  14488 185556    0    0     0     0  143   65  0  0
99  0
 0  0      0 494252  14520 185576    0    0     0    32  116   68  0  0
99  0
 0  0      0 494300  14520 185596    0    0     0     0  127   69  0  0
100  0
 0  0      0 494192  14520 185596    0    0     0     0  141   68  0  0
99  0
 0  0      0 494184  14520 185640    0    0     0     0  153   60  0  0
99  0
 0  0      0 494184  14520 185640    0    0     0     0  122   59  0  0
100  0
 0  0      0 494216  14520 185680    0    0     0     0  124   72  0  0
99  0
 0  0      0 494108  14520 185680    0    0     0     0  123   67  0  0
100  0
 0  0      0 494092  14520 185720    0    0     0     0  117   63  0  0
99  0
 0  0      0 494092  14520 185720    0    0     0     0  119   64  0  0
100  0
 0  0      0 494124  14520 185760    0    0     0     0  123   66  0  0
99  0
 0  0      0 494124  14520 185760    0    0     0     0  125   70  0  0
99  0
 0  0      0 494016  14520 185800    0    0     0     0  118   60  0  0
100  0
 0  0      0 494016  14520 185800    0    0     0     0   93   42  0  0
97  0
 0  0      0 494032  14520 185820    0    0     0     0   25   18  0  0
99  0
 0  0      0 494032  14520 185820    0    0     0     0   11   12  0  0
100  0
 0  0      0 494032  14520 185828    0    0     0     0   89   55  0  0
100  0
 0  0      0 494032  14520 185828    0    0     0     0  138   68  0  1
98  0
 0  0      0 494040  14520 185872    0    0     0     0  118   66  0  0
99  0
 0  0      0 493932  14520 185872    0    0     0     0  122   63  0  0
99  0
 0  0      0 493916  14520 185912    0    0     0     0  129   62  0  0
100  0
 0  0      0 493916  14552 185932    0    0     0    44  126   76  0  0
99  0
 0  0      0 493916  14552 185956    0    0     0     0  123   65  0  0
100  0
 0  0      0 493808  14552 185956    0    0     0     0  119   58  0  0
99  0
 0  0      0 493800  14552 185996    0    0     0     0  132   71  0  0
100  0
 0  0      0 493800  14552 185996    0    0     0     0  120   62  0  0
99  0
 0  0      0 493800  14552 186040    0    0     0    36  131   73  0  0
99  0
 0  0      0 493692  14552 186040    0    0     0     0  133   68  0  0
100  0
 0  0      0 493676  14552 186080    0    0     0     0  130   66  0  0
99  0
 0  0      0 493676  14552 186080    0    0     0     0  125   63  0  0
99  0
 0  0      0 493684  14552 186124    0    0     0     0  139   79  0  0
100  0
 0  0      0 493684  14552 186124    0    0     0     0  126   68  0  0
99  0
 0  0      0 493584  14552 186164    0    0     0     0  125   62  0  0
99  0
 0  0      0 493584  14552 186164    0    0     0     0  127   60  0  0
100  0
 0  0      0 493584  14552 186204    0    0     0     0  130   80  0  0
99  0
 0  0      0 493584  14552 186204    0    0     0     0  132   69  0  0
99  0
 0  0      0 493460  14564 186248    0    0     0    12  119   74  0  0
100  0
 0  0      0 493460  14564 186248    0    0     0     0  129   68  0  0
99  0
 0  0      0 493452  14564 186288    0    0     0     0  145   75  0  0
100  0

thought maybe it was nfs...?

# iostat -n 1
Linux 2.6.21-xen (spud)         05/28/08

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail     10363.60        13.49         0.00        
0.00        11.30        13.49

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        24.18        24.19         0.00        
0.00         0.00        18.67

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         0.00         0.05         0.00        
0.00         0.00         5.56

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         1.98         0.06         0.00        
0.00         0.00         0.05

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail      4137.37         0.06         0.00        
0.00         0.00         0.07

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail     12051.17         4.16         0.00        
0.00         0.00         4.16

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail      6151.21         7.25         0.00        
0.00         0.00         7.26

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        15.48        15.49         0.00        
0.00         0.00        15.49

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        26.00        26.00         0.00        
0.00         0.00        26.00

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        26.98        27.00         0.00        
0.00         0.00        21.00

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        26.79        26.79         0.00        
0.00         0.00        29.09

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        20.99        21.00         0.00        
0.00         0.00        18.77

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail        21.20        21.24         0.00        
0.00         0.00        27.23

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         0.00         0.05         0.00        
0.00         0.00         0.05

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         1.98         0.05         0.00        
0.00         0.00         0.05

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         0.00         0.05         0.00        
0.00         0.00         0.05

Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
swanney:/export/zfs/mail         2.75         2.79         0.00        
0.00         0.00         2.79

the machine is hardly begin touched - only two mailboxes on it, very
litle mail passing through.

any pointers would be really appreciated

Simon Bettison



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.