[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] 99% iowait on one core in 8 core processor



Just checked my /proc/interrupts:

EVERY xen related thing (blkif-backend,the domUs, xhci_hcd, all
xen-interfaces, evtchn:xenstored, evtchn:qemu-dm is all bound to cpu0,
only zeros for the other cpu-cores..

Sys is debian wheezy, xen is current testing from hg, kernel is
openSuse3.4.2 with the xen patches they ship with it..

will try your irqbalanced-suggestion next..

2012/7/12 Ian Campbell <ian.campbell@xxxxxxxxxx>:
> On Wed, 2012-07-11 at 09:56 -0400, Matthias wrote:
>> Hi,
>>
>> I experianced the same.
>>
>> I figured the reason behind this has nothing to do with what (v)cpus
>> you assign to the domUs. In fact, I think that the real reason is that
>> all I/O emulation from the domUs (like HDD, Network and Stuff) is all
>> handled by CPU0 of dom0, even in a multi-core-architecture.
>
> This should not be the case. I/O emulation is done by qemu running in
> domain 0 and should be scheduled on any dom0 vcpu. If it is not then
> this is something to investigate. Normally this would require explicit
> admin action to pin the affinity of the process though.
>
> One easy thing to look at would be /proc/interrupts to check that the
> irq associated with ioreq upcalls for each guest are being properly
> balanced (if not the installing irqbalanced might help)
>
> Ian.
>
>>
>> You can test this with trying something I/O intensive like a hdd
>> benchmark in the domU and you will see the dom0-cpu0 utilization
>> rising.
>>
>> Using PVHVM or GPLPV-Driver only reduces the amount of utilization but
>> it is still handled by cpu0 only. Due to that fact i leave cpu0 to my
>> dom0 alone and only assign the other cpu-cores to the domUs.
>>
>>
>> Note: This is only an assumption I made after seeing my machines
>> behaviour. If the above isn't true, please correct me.
>>
>>
>> 2012/7/11 Rajesh Kumar <rajesh@xxxxxxxx>:
>> > Hi All,
>> >
>> > We have a xen server and using 8 core processor.
>> >
>> > I can see that there is 99% iowait on only  core 0.
>> >
>> >
>> > 02:28:49 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   
>> > %idle    intr/s
>> > 02:28:54 AM  all    0.00    0.00    0.00   12.65    0.00    0.02    2.24   
>> > 85.08   1359.88
>> > 02:28:54 AM    0    0.00    0.00    0.00   96.21    0.00    0.20    3.19   
>> >  0.40    847.11
>> > 02:28:54 AM    1    0.00    0.00    0.00    6.41    0.00    0.00    9.42   
>> > 84.17    219.56
>> > 02:28:54 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
>> > 100.00      2.59
>> > 02:28:54 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    2.12   
>> > 97.88     76.25
>> > 02:28:54 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    1.20   
>> > 98.80    118.56
>> > 02:28:54 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
>> > 100.00      3.59
>> > 02:28:54 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    2.02   
>> > 97.98     89.62
>> > 02:28:54 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.20   
>> > 99.80      2.59
>> >
>> > 02:28:54 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   
>> > %idle    intr/s
>> > 02:28:59 AM  all    0.00    0.00    0.00   12.48    0.00    0.00    2.78   
>> > 84.74   1317.43
>> > 02:28:59 AM    0    0.00    0.00    0.00   98.80    0.00    0.00    0.80   
>> >  0.40    885.17
>> > 02:28:59 AM    1    0.00    0.00    0.00    0.00    0.00    0.00   11.38   
>> > 88.62    151.30
>> > 02:28:59 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.20   
>> > 99.80      2.81
>> > 02:28:59 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    7.21   
>> > 92.79     94.79
>> > 02:28:59 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    2.20   
>> > 97.80    170.34
>> > 02:28:59 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
>> > 100.00      4.41
>> > 02:28:59 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
>> > 100.00      5.81
>> > 02:28:59 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00  
>> > 100.00      2.81
>> >
>> >
>> > I have even tries changing the cpus mapped to the DomU with out no effect.
>> >
>> > Name                              CPU  CPU Affinity
>> > 4pulse                              1           1
>> > 2music                             3            3
>> > Domain-0                        0               0
>> > Domain-0                        1               1
>> > Domain-0                        2               2
>> > Domain-0                        3               3
>> > Domain-0                        4               4
>> > Domain-0                        5               5
>> > Domain-0                        6               6
>> > Domain-0                        7               7
>> > analshah                         6              any cpu
>> > arunvelayudhan            7             any cpu
>> > backup                            7             any cpu
>> > crickruns                         3             1-3
>> > crickruns                         2             1-3
>> > crickruns                         1             1-3
>> > crickruns                         2             1-3
>> > crickruns                         1             1-3
>> > crickruns                         1             1-3
>> > dedicatedjv                     7               any cpu
>> > yeluthu                             4           3-5
>> > yeluthu                             3           3-5
>> > yeluthu                             3           3-5
>> > yeluthu                             3           3-5
>> > yeluthu                             3           3-5
>> > yeluthu                             3           3-5
>> > freshnfresh                      3              any cpu
>> > monitoring                       7              any cpu
>> > reporter                                  6             5-7
>> > reporter                          7             5-7
>> > reporter                          6             5-7
>> > reporter                          6             5-7
>> > reporter                          7             5-7
>> > reporter                                  7             5-7
>> > reporter                          5             5-7
>> > reporter                          7             5-7
>> > radio03                            7            any cpu
>> > saampeter                       2               1-2
>> > saampeter                       2               1-2
>> >
>> >
>> > Thanks,
>> > Rajesh
>> > _______________________________________________
>> > Xen-users mailing list
>> > Xen-users@xxxxxxxxxxxxx
>> > http://lists.xen.org/xen-users
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.