Re: [Xen-devel] [Xen-users] Changing netback tx interrupts affinity on Dom0

[Jintack Lim <jintack@xxxxxxxxxxxxxxx>]

Thanks Julien.

On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@xxxxxxxxxx> wrote:

On 03/09/15 17:06, Jintack Lim wrote:
> Hi,

Hi Jintack,

> On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@xxxxxxxxxx

> <mailto:ian.campbell@xxxxxxxxxx>> wrote:
>
>Â Â ÂOn Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
>Â Â Â> Hi,
>Â Â Â>
>Â Â Â> While I was running Apache server,
>Â Â Â> I found that one of Dom0 vcpu is running 100% to handle irqs,
>Â Â Â> and those irqs are set to be processed only on that specific vcpu.
>Â Â Â>
>Â Â Â> Referring to this document,
>Â Â Â> http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
>Â Â Â> I tried to change smp_affinity by writing a value to /proc/irq/<irq
>Â Â Â> -no>/smp_affinity,
>Â Â Â> however the smp_affinity value was not changed.
>Â Â Â>
>Â Â Â> I'm working on Xen 4.5 on ARMv8,
>Â Â Â> and the irq is netback tx interrupt.
>Â Â Â>
>Â Â Â> # cat /proc/irq/106/smp_affinity
>Â Â Â> 1
>Â Â Â>
>Â Â Â> # cat /proc/interrupts
>Â Â Â> ...
>Â Â Â> 106:Â Â Â 53849Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q0-tx
>Â Â Â> 107:Â Â Â Â Â 1Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q0-rx
>Â Â Â> 108:Â Â Â 61460Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q1-tx
>Â Â Â> 109:Â Â Â Â Â 1Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q1-rx
>Â Â Â> 110:Â Â Â 67118Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q2-tx
>Â Â Â> 111:Â Â Â Â Â 1Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q2-rx
>Â Â Â> 112:Â Â Â 58273Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q3-tx
>Â Â Â> 113:Â Â Â Â Â 1Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0Â Âxen-dyn-event
>Â Â Â> vif1.1-q3-rx
>Â Â Â> ...
>Â Â Â>
>Â Â Â> What would be the way to change smp_affinity?

All those interrupts are in-fine event channels. This is a bug in Linux
which I sent a fix a month ago and it's queued in xentip for Linux 4.3 [1]

The event channel rebinding was not working because we don't have vector
callback on ARM and therefore the driver was only allowing event channel
to be routed on VCPU0.

After applying this patch, I was able to set smp_affinity.

In addition, the default value of smp_affinity for netback tx interrupts on Dom0 becomes "f" (I have four vcpus on Dom0).

What I meant by 'netback tx interrupts' isÂvif1.1-q<num>-tx

This is slightly different from x86, because netback tx interrupts are pinned to separate vcpus on x86.

I'm not sure this question is appropriate for devel-list,

but why the policy to set smp_affinity for netback tx interrupts are different on ARM and x86?

Â

commit 4a5b69464e51f4a8dd432e8c2a1468630df1a53c
Author: Julien Grall <julien.grall@xxxxxxxxxx>
Date:Â ÂTue Jul 28 10:10:42 2015 +0100

xen/events: Support event channel rebind on ARM
Currently, the event channel rebind code is gated with the presence of
the vector callback.

The virtual interrupt controller on ARM has the concept of per-CPU
interrupt (PPI) which allow us to support per-VCPU event
channel.Therefore there is no need of vector callback for ARM.

Xen is already using a free PPI to notify the guest VCPU of an event.
Furthermore, the xen code initialization in Linux (see
arch/arm/xen/enlighten.c) is requesting correctly a per-CPU IRQ.

Introduce new helper xen_support_evtchn_rebind to allow architecture
decide whether rebind an event is support or not. It will always return
true on ARM and keep the same behavior on x86.

This is also allow us to drop the usage of xen_have_vector_callback
entirely in the ARM code.

Signed-off-by: Julien Grall <julien.grall@xxxxxxxxxx>
Signed-off-by: David Vrabel <david.vrabel@xxxxxxxxxx>

Regards,

[1] https://lkml.org/lkml/2015/7/28/205

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel