[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] 99% iowait on one core in 8 core processor



On Wed, 2012-07-11 at 09:56 -0400, Matthias wrote:
> Hi,
> 
> I experianced the same.
> 
> I figured the reason behind this has nothing to do with what (v)cpus
> you assign to the domUs. In fact, I think that the real reason is that
> all I/O emulation from the domUs (like HDD, Network and Stuff) is all
> handled by CPU0 of dom0, even in a multi-core-architecture.

This should not be the case. I/O emulation is done by qemu running in
domain 0 and should be scheduled on any dom0 vcpu. If it is not then
this is something to investigate. Normally this would require explicit
admin action to pin the affinity of the process though.

One easy thing to look at would be /proc/interrupts to check that the
irq associated with ioreq upcalls for each guest are being properly
balanced (if not the installing irqbalanced might help)

Ian.

> 
> You can test this with trying something I/O intensive like a hdd
> benchmark in the domU and you will see the dom0-cpu0 utilization
> rising.
> 
> Using PVHVM or GPLPV-Driver only reduces the amount of utilization but
> it is still handled by cpu0 only. Due to that fact i leave cpu0 to my
> dom0 alone and only assign the other cpu-cores to the domUs.
> 
> 
> Note: This is only an assumption I made after seeing my machines
> behaviour. If the above isn't true, please correct me.
> 
> 
> 2012/7/11 Rajesh Kumar <rajesh@xxxxxxxx>:
> > Hi All,
> >
> > We have a xen server and using 8 core processor.
> >
> > I can see that there is 99% iowait on only  core 0.
> >
> >
> > 02:28:49 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   
> > %idle    intr/s
> > 02:28:54 AM  all    0.00    0.00    0.00   12.65    0.00    0.02    2.24   
> > 85.08   1359.88
> > 02:28:54 AM    0    0.00    0.00    0.00   96.21    0.00    0.20    3.19    
> > 0.40    847.11
> > 02:28:54 AM    1    0.00    0.00    0.00    6.41    0.00    0.00    9.42   
> > 84.17    219.56
> > 02:28:54 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
> > 100.00      2.59
> > 02:28:54 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    2.12   
> > 97.88     76.25
> > 02:28:54 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    1.20   
> > 98.80    118.56
> > 02:28:54 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
> > 100.00      3.59
> > 02:28:54 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    2.02   
> > 97.98     89.62
> > 02:28:54 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.20   
> > 99.80      2.59
> >
> > 02:28:54 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   
> > %idle    intr/s
> > 02:28:59 AM  all    0.00    0.00    0.00   12.48    0.00    0.00    2.78   
> > 84.74   1317.43
> > 02:28:59 AM    0    0.00    0.00    0.00   98.80    0.00    0.00    0.80    
> > 0.40    885.17
> > 02:28:59 AM    1    0.00    0.00    0.00    0.00    0.00    0.00   11.38   
> > 88.62    151.30
> > 02:28:59 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.20   
> > 99.80      2.81
> > 02:28:59 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    7.21   
> > 92.79     94.79
> > 02:28:59 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    2.20   
> > 97.80    170.34
> > 02:28:59 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
> > 100.00      4.41
> > 02:28:59 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
> > 100.00      5.81
> > 02:28:59 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
> > 100.00      2.81
> >
> >
> > I have even tries changing the cpus mapped to the DomU with out no effect.
> >
> > Name                              CPU  CPU Affinity
> > 4pulse                              1           1
> > 2music                             3            3
> > Domain-0                        0               0
> > Domain-0                        1               1
> > Domain-0                        2               2
> > Domain-0                        3               3
> > Domain-0                        4               4
> > Domain-0                        5               5
> > Domain-0                        6               6
> > Domain-0                        7               7
> > analshah                         6              any cpu
> > arunvelayudhan            7             any cpu
> > backup                            7             any cpu
> > crickruns                         3             1-3
> > crickruns                         2             1-3
> > crickruns                         1             1-3
> > crickruns                         2             1-3
> > crickruns                         1             1-3
> > crickruns                         1             1-3
> > dedicatedjv                     7               any cpu
> > yeluthu                             4           3-5
> > yeluthu                             3           3-5
> > yeluthu                             3           3-5
> > yeluthu                             3           3-5
> > yeluthu                             3           3-5
> > yeluthu                             3           3-5
> > freshnfresh                      3              any cpu
> > monitoring                       7              any cpu
> > reporter                                  6             5-7
> > reporter                          7             5-7
> > reporter                          6             5-7
> > reporter                          6             5-7
> > reporter                          7             5-7
> > reporter                                  7             5-7
> > reporter                          5             5-7
> > reporter                          7             5-7
> > radio03                            7            any cpu
> > saampeter                       2               1-2
> > saampeter                       2               1-2
> >
> >
> > Thanks,
> > Rajesh
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@xxxxxxxxxxxxx
> > http://lists.xen.org/xen-users
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.