|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt
From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> To: Justin Acker <ackerj67@xxxxxxxxx> Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>; boris.ostrovsky@xxxxxxxxxx Sent: Tuesday, September 1, 2015 4:56 PM Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote: > Taking this to the dev list from users. > > Is there a way to force or enable pirq delivery to a set of cpus as opposed to single device from being a assigned a single pirq so that its interrupt can be distributed across multiple cpus? I believe the device drivers do support multiple queues when run natively without the Dom0 loaded. The device in question is the xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 is loaded. The behavior seems to pass through to the DomU if pass through is enabled. I found some similar threads, but most relate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried IRQ balance and when configured/adjusted it will balance individual pirqs, but not multiple interrupts. Yes. You can do it with smp affinity: Yes, this does allow for assigning a specific interrupt to a single cpu, but it will not spread the interrupt load across a defined group or all cpus. Is it possible to define a range of CPUs or spread the interrupt load for a device across all cpus as it does with a native kernel without the Dom0 loaded? I don't follow the "behavior seems to pass through to the DomU if pass through is enabled" ? The device interrupts are limited to a single pirq if the device is used directly in the Dom0. If the device is passed through to a DomU - i.e. the xhci_hcd controller - then the DomU cannot spread the interrupt load across the cpus in the VM. > > > > With irqbalance enabled in Dom0: What version? There was a bug in it where it would never distribute the IRQs properly across the CPUs. irqbalance version 1.0.7. Boris (CC-ed) might remember the upstream patch that made this work properly? > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > 76: 11304 0 149579 0 0 0 0 0 xen-pirq-msi 0000:00:1f.2 > 77: 1243 0 0 35447 0 0 0 0 xen-pirq-msi radeon > 78: 82521 0 0 0 0 0 0 0 xen-pirq-msi xhci_hcd > 79: 23 0 0 0 0 0 0 0 xen-pirq-msi mei_me > 80: 11 0 0 0 0 741 0 0 xen-pirq-msi em1 > 81: 350 0 0 0 1671 0 0 0 xen-pirq-msi iwlwifi > 82: 275 0 0 0 0 0 0 0 xen-pirq-msi snd_hda_intel > > With native 3.19 kernel: > > Without Dom0 for the same system from the first message: > > # cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > 0: 33 0 0 0 0 0 0 0 IR-IO-APIC-edge timer > 8: 0 0 0 0 0 0 1 0 IR-IO-APIC-edge rtc0 > 9: 20 0 0 0 0 1 1 1 IR-IO-APIC-fasteoi acpi > 16: 15 0 8 1 4 1 1 1 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 > 18: 703940 5678 1426226 1303 3938243 111477 757871 510 IR-IO-APIC 18-fasteoi ath9k > 23: 11 2 3 0 0 17 2 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 > 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 > 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 > 26: 20419 1609 26822 567 62281 5426 14928 395 IR-PCI-MSI-edge 0000:00:1f.2 > 27: 17977230 628258 44247270 120391 1597809883 14440991 152189328 73322 IR-PCI-MSI-edge xhci_hcd > 28: 563 0 0 0 1 0 6 0 IR-PCI-MSI-edge i915 > 29: 14 0 0 4 2 4 0 0 IR-PCI-MSI-edge mei_me > 30: 39514 1744 60339 157 129956 19702 72140 83 IR-PCI-MSI-edge eth0 > 31: 3 0 0 1 54 0 0 2 IR-PCI-MSI-edge snd_hda_intel > 32: 28145 284 53316 63 139165 4410 25760 27 IR-PCI-MSI-edge eth1-rx-0 > 33: 1032 43 2392 5 1797 265 1507 20 IR-PCI-MSI-edge eth1-tx-0 > 34: 0 1 0 0 0 1 2 0 IR-PCI-MSI-edge eth1 > 35: 5 0 0 12 148 6 2 1 IR-PCI-MSI-edge snd_hda_intel > > > The USB controller is an Intel C210: > > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Dell Device 053e > Flags: bus master, medium devsel, latency 0, IRQ 78 > Memory at f7f20000 (64-bit, non-prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > Kernel modules: xhci_pci > On Tuesday, September 1, 2015 11:50 AM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote: > > > On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote: > > Thanks Ian, > > > > I appreciate the explanation. I believe the device drivers do support > > multiple queues when run natively without the Dom0 loaded. The device in > > question is the xhci_hcd driver for which I/O transfers seem to be slowed > > when the Dom0 is loaded. The behavior seems to pass through to the DomU > > if pass through is enabled. I found some similar threads, but most relate > > to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 > > kernel arguments, but none distributed the pirqs. Based on the reading > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to > > avoid an I/O storm. I tried IRQ balance and when configured/adjusted it > > will balance individual pirqs, but not multiple interrupts. > > > > Is there a way to force or enable pirq delivery to a set of cpus as you > > mentioned above or omit a single device from being a assigned a PIRQ so > > that its interrupt can be distributed across all cpus? > > A PIRQ is the way an interrupt is exposed to a PV guest, without it there > would be no interrupt at all. > > I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs > interact, in particular WRT configuring which set of CPUs can have the IRQ > delivered. > > If no one else chimes in soon I'd suggest taking this to the dev list, at > the very least someone who knows what they are talking about (i.e. other > than me) might be able to help. > > Ian. > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@xxxxxxxxxxxxx > http://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |