[Xen-users] Interrupt handling in Xen

[Adi Kriegisch <adi@xxxxxxxxxxxxxxx>]

Hi!

This should probably be added to "best practices"[1] or "common
problems"[2].

Two servers with 10x1GBit ethernet each were able to achieve around
1150MB/s (byte not bit) in network benchmarks conducted with iperf
(2.6.26/stock Debian Lenny).
Booting Xen (3.4.3-rc3 from Debian/unstable) with kernel 2.6.32.11 (pv-ops)
as Dom0 it turned out that the servers could not do more than about
700-800MB/s. (A long way of debugging/testing/benchmarking followed -- I
tried lots of things including Xen-4.0.0 ;-)
Looking at /proc/interrupts lead to the solution:
            CPU0       CPU1       CPU2       CPU3
1500:    1859518          0          0          0  xen-pirq-msi       eth9
1501:    2295607          0          0          0  xen-pirq-msi       eth8
1502:    2720851          0          0          0  xen-pirq-msi       eth7
1503:    3064408          0          0          0  xen-pirq-msi       eth6
1504:    4573347          0          0          0  xen-pirq-msi       eth5
1505:    4672864          0          0          0  xen-pirq-msi       eth4

all interrups are pinned on CPU0; booting the system without Xen shows that
interrupts are equaly distributed over all the cores.
So the solution is quite easy: either manually balance interrupts by eg:
echo <BITMASK> > /proc/irq/<INTERRUPT>/smp_affinity
where <BITMASK> (in hex) is
1 -> CPU0
2 -> CPU1
4 -> CPU2
8 -> CPU3
and so on. This is actually a bit mask that allows to attach several
CPUs/cores to handle interrupts on native linux. In Xen this is disabled:
If you specify a mask like "3" (=1+2 which means CPU0 and CPU1) the first
matching CPU will be chosen -- CPU0 in this case.

The easier and for most cases better solution will be to install
irqbalance[3] which Debian Lenny did not do automatically.

After installation of irqbalance I measured values around 1100MB/s which is
about 50MB/s less than native. I am not sure if manual balancing will bring
better results or if I should do more thorrow testing. On the other hand,
having 1100MB/s seems to be fast enough for what I plan to do! ;-)
There was however an interesting difference between Xen-3.4.3-rc3 and
Xen-4.0.0: With the very same kernel (2.6.32.11 pv-ops/Debian) Xen-3.4.3
got stuck for fractions of seconds while under high load whereas with
Xen-4.0.0 I did not experience anything similar. In terms of numbers
Xen-4.0.0 was between 1 and 10MB/s faster -- which is less than 1%.

best regards,
  Adi Kriegisch

[1] http://wiki.xensource.com/xenwiki/XenBestPractices
[2] http://wiki.xensource.com/xenwiki/XenCommonProblems
[3] http://www.irqbalance.org


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users