[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt





From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
To: Justin Acker <ackerj67@xxxxxxxxx>
Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>; boris.ostrovsky@xxxxxxxxxx
Sent: Tuesday, September 1, 2015 4:56 PM
Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt

On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
> Taking this to the dev list from users.
>
> Is there a way to force or enable pirq delivery to a set of cpus as opposed to single device from being a assigned a single pirq so that its interrupt can be distributed across multiple cpus? I believe the device drivers do support multiple queues when run natively without the Dom0 loaded. The device in question is the xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 is loaded. The behavior seems to pass through to the DomU if pass through is enabled. I found some similar threads, but most relate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried IRQ balance and when configured/adjusted it will balance individual pirqs, but not multiple interrupts.

Yes. You can do it with smp affinity:


Yes, this does allow for assigning a specific interrupt to a single cpu, but it will not spread the interrupt load across a defined group or all cpus. Is it possible to define a range of CPUs or spread the interrupt load for a device across all cpus as it does with a native kernel without the Dom0 loaded?

I don't follow the "behavior seems to pass through to the DomU if pass through is enabled" ?

The device interrupts are limited to a single pirq if the device is used directly in the Dom0. If the device is passed through to a DomU - i.e. the xhci_hcd controller - then the DomU cannot spread the interrupt load across the cpus in the VM.

>
>
>
> With irqbalance enabled in Dom0:

What version? There was a bug in it where it would never distribute the IRQs properly
across the CPUs.

irqbalance version 1.0.7.

Boris (CC-ed) might remember the upstream patch that made this work properly?



>
>            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7     
>  76:      11304          0     149579          0          0          0          0          0  xen-pirq-msi       0000:00:1f.2
>  77:       1243          0          0      35447          0          0          0          0  xen-pirq-msi       radeon
>  78:      82521          0          0          0          0          0          0          0  xen-pirq-msi       xhci_hcd
>  79:         23          0          0          0          0          0          0          0  xen-pirq-msi       mei_me
>  80:         11          0          0          0          0        741          0          0  xen-pirq-msi       em1
>  81:        350          0          0          0       1671          0          0          0  xen-pirq-msi       iwlwifi
>  82:        275          0          0          0          0          0          0          0  xen-pirq-msi       snd_hda_intel
>
> With native 3.19 kernel:
>
> Without Dom0 for the same system from the first message:
>
> # cat /proc/interrupts
>            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7     
>   0:         33          0          0          0          0          0          0          0  IR-IO-APIC-edge      timer
>   8:          0          0          0          0          0          0          1          0  IR-IO-APIC-edge      rtc0
>   9:         20          0          0          0          0          1          1          1  IR-IO-APIC-fasteoi   acpi
>  16:         15          0          8          1          4          1          1          1  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
>  18:     703940       5678    1426226       1303    3938243     111477     757871        510  IR-IO-APIC  18-fasteoi   ath9k
>  23:         11          2          3          0          0         17          2          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
>  24:          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar0
>  25:          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar1
>  26:      20419       1609      26822        567      62281       5426      14928        395  IR-PCI-MSI-edge      0000:00:1f.2
>  27:   17977230     628258   44247270     120391 1597809883   14440991  152189328      73322  IR-PCI-MSI-edge      xhci_hcd
>  28:        563          0          0          0          1          0          6          0  IR-PCI-MSI-edge      i915
>  29:         14          0          0          4          2          4          0          0  IR-PCI-MSI-edge      mei_me
>  30:      39514       1744      60339        157     129956      19702      72140         83  IR-PCI-MSI-edge      eth0
>  31:          3          0          0          1         54          0          0          2  IR-PCI-MSI-edge      snd_hda_intel
>  32:      28145        284      53316         63     139165       4410      25760         27  IR-PCI-MSI-edge      eth1-rx-0
>  33:       1032         43       2392          5       1797        265       1507         20  IR-PCI-MSI-edge      eth1-tx-0
>  34:          0          1          0          0          0          1          2          0  IR-PCI-MSI-edge      eth1
>  35:          5          0          0         12        148          6          2          1  IR-PCI-MSI-edge      snd_hda_intel
>
>
> The USB controller is an Intel C210:
>
> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
>     Subsystem: Dell Device 053e
>     Flags: bus master, medium devsel, latency 0, IRQ 78
>     Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
>     Capabilities: [70] Power Management version 2
>     Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
>     Kernel driver in use: xhci_hcd
>     Kernel modules: xhci_pci
>      On Tuesday, September 1, 2015 11:50 AM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:
>   
>
>  On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
> > Thanks Ian,
> >
> > I appreciate the explanation. I believe the device drivers do support
> > multiple queues when run natively without the Dom0 loaded. The device in
> > question is the xhci_hcd driver for which I/O transfers seem to be slowed
> > when the Dom0 is loaded. The behavior seems to pass through to the DomU
> > if pass through is enabled. I found some similar threads, but most relate
> > to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0
> > kernel arguments, but none distributed the pirqs. Based on the reading
> > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to
> > avoid an I/O storm. I tried IRQ balance and when configured/adjusted it
> > will balance individual pirqs, but not multiple interrupts.
> >
> > Is there a way to force or enable pirq delivery to a set of cpus as you
> > mentioned above or omit a single device from being a assigned a PIRQ so
> > that its interrupt can be distributed across all cpus?
>
> A PIRQ is the way an interrupt is exposed to a PV guest, without it there
> would be no interrupt at all.
>
> I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
> interact, in particular WRT configuring which set of CPUs can have the IRQ
> delivered.
>
> If no one else chimes in soon I'd suggest taking this to the dev list, at
> the very least someone who knows what they are talking about (i.e. other
> than me) might be able to help.
>
> Ian.

>
>
>


> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.