Re: [Xen-users] 99% iowait on one core in 8 core processor

[Matthias <matthias.kannenberg@xxxxxxxxxxxxxx>]

Some updates:

- Deactivating CONFIg_HOTPlUG_CPU and suspend to ram in kernel didn'T
change anything.
- I tried to change the smp_affinity to just a single other cpu and
this worked. So basically I was wrong that there is something changing
the settings back constantly but linux is simply discarding every
assignment to more then one cpu (1,2,4, etc works, 3,5,etc is
discarded)

so the problem is that my linux can not assign irqs to multiple cpus.
I think this is also why irqbalance does not make a difference because
it tries the same.

I will check if i can reproduce this behaviour with a stock kernel
without xen or if this is really xen related in the evening. But if
you have any other idea what could cause this, I'm open for
suggestions

@Rajesh can you check with your setup if you have the same case or if
this is a different problem? simply do a 'cat /proc/interrupts' to
check the cpu affinity and if everything is done on cpu0 try a 'echo
"3" > /proc/irq/<some irq number from the other command>/smp_affinity'
and afterwards check if the smp_affinity now really has '3' as
content.

2012/7/13 Matthias <matthias.kannenberg@xxxxxxxxxxxxxx>:
> Found a hint somewhere that my problem might be related to the
> CONFIG_HOTPLUG_CPU kernel option.. recompiling the kernel right now and will
> update tomorrow if this resolved the issue..
>
> Am 13.07.2012 00:27 schrieb "Matthias" <matthias.kannenberg@xxxxxxxxxxxxxx>:
>
>> Hi Ian,
>>
>> sorry, but it's still not working..
>>
>> I rebooted the server, then manually started irqbalanced just to be
>> sure, then started xen and the domUs.. still the same as above: every
>> I/O related is done by cpu0 (even after hours of running 5 domUs). i
>> see some spikes on the other cpu's, but i think this is due to domUs
>> using their assigned vcpus and xentop shows a distribution of cputime,
>> too..
>>
>> I started irqbalance in debug modus once and it complained that my
>> hardware is not numa compatible. Might this be an issue? From what I
>> read, xen should support proper scheduling on non-numa hardware and
>> only the numa-support is new and might be a bit quirky..
>>
>> checked my smp_affinity stuff then: currently, it shows the following for
>> a domU
>> smp_affinity: 01
>> smp_affinity_list: 0
>>
>> tried to change smp_affinity to 3f (=111111 for my 6vcpu) and it was
>> changed immediately back to 01.. thought that was irqbalance going
>> rogue but after stopd the deamon, this still happens..
>>
>> so my take is: something is setting the irq to only use cpu0 and
>> changes i do manually or which are done by irqbalance are overwritten
>> constantly making irqbalance useless..
>>
>> Any idea what this can be? supposently something within xen?
>>
>>
>>
>> 2012/7/12 Ian Campbell <ian.campbell@xxxxxxxxxx>:
>> > On Thu, 2012-07-12 at 13:04 -0400, Matthias wrote:
>> >>
>> >> Any other idea how we can make xen utilize the other (v)cpus for it's
>> >> I/O stuff?
>> >
>> > Are you sure irqbalanced is running? Some versions had a bug and would
>> > crash on a Xen system (they crash if there is no irq 0 or something like
>> > that). Even if it is running it can take some time for irqbalanced to
>> > realise that things are unbalanced and start moving stuff around.
>> >
>> > There are ways in Linux to manually balance IRQs. You have to much
>> > around with /proc/irq/*/smp_affinity*
>> >
>> > Really irqbalanced should be doing this for you though.
>> >
>> > Ian.
>> >
>> >

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users