Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt

[Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>]

On 09/01/2015 04:56 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
> > Taking this to the dev list from users.
> >
> > Is there a way to force or enable pirq delivery to a set of cpus as opposed to 
> > single device from being a assigned a single pirq so that its interrupt can be 
> > distributed across multiple cpus? I believe the device drivers do support 
> > multiple queues when run natively without the Dom0 loaded. The device in 
> > question is the xhci_hcd driver for which I/O transfers seem to be slowed when 
> > the Dom0 is loaded. The behavior seems to pass through to the DomU if pass 
> > through is enabled. I found some similar threads, but most relate to Ethernet 
> > controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, 
> > but none distributed the pirqs. Based on the reading relating to IRQs for Xen, 
> > I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried 
> > IRQ balance and when configured/adjusted it will balance individual pirqs, but 
> > not multiple interrupts.
> Yes. You can do it with smp affinity:
>
> https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
>
> I don't follow the "behavior seems to pass through to the DomU if pass through is 
> enabled" ?
>
> > With irqbalance enabled in Dom0:
> What version? There was a bug in it where it would never distribute the IRQs 
> properly
> across the CPUs.
>
> Boris (CC-ed) might remember the upstream patch that made this work properly?
I think we ended up taking the latest version of irqbalance as the one 
that added support for Xen guests was still not quite working. Besides, 
that patch was for xen-dyn-events, not for pirqs.
-boris


> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
> > CPU6       CPU7
> >   76:      11304          0     149579          0          0          0         
> >  0          0  xen-pirq-msi       0000:00:1f.2
> >   77:       1243          0          0      35447          0          0         
> >  0          0  xen-pirq-msi       radeon
> >   78:      82521          0          0          0          0          0         
> >  0          0  xen-pirq-msi       xhci_hcd
> >   79:         23          0          0          0          0          0         
> >  0          0  xen-pirq-msi       mei_me
> >   80:         11          0          0          0          0        741         
> >  0          0  xen-pirq-msi       em1
> >   81:        350          0          0          0       1671          0         
> >  0          0  xen-pirq-msi       iwlwifi
> >   82:        275          0          0          0          0          0         
> >  0          0  xen-pirq-msi       snd_hda_intel
> >
> > With native 3.19 kernel:
> >
> > Without Dom0 for the same system from the first message:
> >
> > # cat /proc/interrupts
> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
> > CPU6       CPU7
> >    0:         33          0          0          0          0          0         
> >  0          0  IR-IO-APIC-edge      timer
> >    8:          0          0          0          0          0          0         
> >  1          0  IR-IO-APIC-edge      rtc0
> >    9:         20          0          0          0          0          1         
> >  1          1  IR-IO-APIC-fasteoi   acpi
> >   16:         15          0          8          1          4          1         
> >  1          1  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
> >   18:     703940       5678    1426226       1303    3938243     111477     
> > 757871        510  IR-IO-APIC  18-fasteoi   ath9k
> >   23:         11          2          3          0          0         17         
> >  2          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
> >   24:          0          0          0          0          0          0         
> >  0          0  DMAR_MSI-edge      dmar0
> >   25:          0          0          0          0          0          0         
> >  0          0  DMAR_MSI-edge      dmar1
> >   26:      20419       1609      26822        567      62281       5426      
> > 14928        395  IR-PCI-MSI-edge      0000:00:1f.2
> >   27:   17977230     628258   44247270     120391 1597809883   14440991  
> > 152189328      73322  IR-PCI-MSI-edge      xhci_hcd
> >   28:        563          0          0          0          1          0         
> >  6          0  IR-PCI-MSI-edge      i915
> >   29:         14          0          0          4          2          4         
> >  0          0  IR-PCI-MSI-edge      mei_me
> >   30:      39514       1744      60339        157     129956      19702      
> > 72140         83  IR-PCI-MSI-edge      eth0
> >   31:          3          0          0          1         54          0         
> >  0          2  IR-PCI-MSI-edge      snd_hda_intel
> >   32:      28145        284      53316         63     139165       4410      
> > 25760         27  IR-PCI-MSI-edge      eth1-rx-0
> >   33:       1032         43       2392          5       1797        265       
> > 1507         20  IR-PCI-MSI-edge      eth1-tx-0
> >   34:          0          1          0          0          0          1         
> >  2          0  IR-PCI-MSI-edge      eth1
> >   35:          5          0          0         12        148          6         
> >  2          1  IR-PCI-MSI-edge      snd_hda_intel
> >
> >
> > The USB controller is an Intel C210:
> >
> > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family 
> > USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
> >      Subsystem: Dell Device 053e
> >      Flags: bus master, medium devsel, latency 0, IRQ 78
> >      Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
> >      Capabilities: [70] Power Management version 2
> >      Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> >      Kernel driver in use: xhci_hcd
> >      Kernel modules: xhci_pci
> >       On Tuesday, September 1, 2015 11:50 AM, Ian Campbell 
> > <ian.campbell@xxxxxxxxxx> wrote:
> >     
> >
> >   On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
> > > Thanks Ian,
> > >
> > > I appreciate the explanation. I believe the device drivers do support
> > > multiple queues when run natively without the Dom0 loaded. The device in
> > > question is the xhci_hcd driver for which I/O transfers seem to be slowed
> > > when the Dom0 is loaded. The behavior seems to pass through to the DomU
> > > if pass through is enabled. I found some similar threads, but most relate
> > > to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0
> > > kernel arguments, but none distributed the pirqs. Based on the reading
> > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to
> > > avoid an I/O storm. I tried IRQ balance and when configured/adjusted it
> > > will balance individual pirqs, but not multiple interrupts.
> > >
> > > Is there a way to force or enable pirq delivery to a set of cpus as you
> > > mentioned above or omit a single device from being a assigned a PIRQ so
> > > that its interrupt can be distributed across all cpus?
> > A PIRQ is the way an interrupt is exposed to a PV guest, without it there
> > would be no interrupt at all.
> >
> > I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
> > interact, in particular WRT configuring which set of CPUs can have the IRQ
> > delivered.
> >
> > If no one else chimes in soon I'd suggest taking this to the dev list, at
> > the very least someone who knows what they are talking about (i.e. other
> > than me) might be able to help.
> >
> > Ian.
> >
> >
> >
> >    
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxx
> > http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel