|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt
Thanks Ian, I appreciate the explanation. I believe the device drivers do support multiple queues when run natively without the Dom0 loaded. The device in question is the xhci_hcd driver for which I/O transfers seem to be slowed when the Dom0 is loaded. The behavior seems to pass through to the DomU if pass through is enabled. I found some similar threads, but most relate to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, but none distributed the pirqs. Based on the reading relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to avoid an I/O storm. I tried IRQ balance and when configured/adjusted it will balance individual pirqs, but not multiple interrupts. Is there a way to force or enable pirq delivery to a set of cpus as you mentioned above or omit a single device from being a assigned a PIRQ so that its interrupt can be distributed across all cpus? Without Dom0 for the same system from the first message: # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 33 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 8: 0 0 0 0 0 0 1 0 IR-IO-APIC-edge rtc0 9: 20 0 0 0 0 1 1 1 IR-IO-APIC-fasteoi acpi 16: 15 0 8 1 4 1 1 1 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 18: 703940 5678 1426226 1303 3938243 111477 757871 510 IR-IO-APIC 18-fasteoi ath9k 23: 11 2 3 0 0 17 2 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 20419 1609 26822 567 62281 5426 14928 395 IR-PCI-MSI-edge 0000:00:1f.2 27: 17977230 628258 44247270 120391 1597809883 14440991 152189328 73322 IR-PCI-MSI-edge xhci_hcd 28: 563 0 0 0 1 0 6 0 IR-PCI-MSI-edge i915 29: 14 0 0 4 2 4 0 0 IR-PCI-MSI-edge mei_me 30: 39514 1744 60339 157 129956 19702 72140 83 IR-PCI-MSI-edge eth0 31: 3 0 0 1 54 0 0 2 IR-PCI-MSI-edge snd_hda_intel 32: 28145 284 53316 63 139165 4410 25760 27 IR-PCI-MSI-edge eth1-rx-0 33: 1032 43 2392 5 1797 265 1507 20 IR-PCI-MSI-edge eth1-tx-0 34: 0 1 0 0 0 1 2 0 IR-PCI-MSI-edge eth1 35: 5 0 0 12 148 6 2 1 IR-PCI-MSI-edge snd_hda_intel NMI: 219 3431 2704 14 2288 73 350 10 Non-maskable interrupts LOC: 3217463 34396332 27060410 1455125 12973313 611058 2497009 704568 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 219 3431 2704 14 2288 73 350 10 Performance monitoring interrupts IWI: 12 3 22 0 49 5 31 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 2424805 445319 3613972 11673 3374860 35514 225975 3640 Rescheduling interrupts CAL: 1699 1793 1824 1714 1802 1875 1942 1758 Function call interrupts TLB: 9356 2750 44389 6766 18585 2983 30122 1911 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 265 265 265 265 265 265 265 265 Machine check polls THR: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 0 MIS: 0 On Tuesday, September 1, 2015 9:26 AM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote: On Sat, 2015-08-15 at 14:31 +0000, Justin Acker wrote: > Hello, > > Is there a configuration option or patch to control the xhci_hcd > interrupt smp affinity? It looks like the Dom0 and DomU, if passed > through, will only use a single interrupt on the xhci_hcd controller > (usually the first unless smp affinity is manually set). > The xhci_hcd interrupts appear to be scheduled across all CPUs when > booting with a native kernel. I've noticed other devices seem to schedule > interrupts across all CPUs when in the Dom0 and DomU. This suggests that you do have irqbalanced running and it is doing stuff, the interrupt counts you showed suggest it is rebalancing some stuff Perhaps irqbalanced has just decided that xchi_hcd should be on CPU0? After all something should run there... The proc file is cumulative, meaning that while multiple CPUs may have a non-zero count for a given interrupt not all of them will be incrementing right now. So I'm not completely sure that what you are seeing isn't just normal behaviour. I'm not sure if Xen pirqs support delivery to a set of CPUs rather than just a single one at a time, TBH I'm not even really sure what the behaviour of MSIs for the native case is. Devices which want to benefit from multiple CPUs typically need to have multiple queues and multiple associated interrupts. Sorry, there's a lot more "not sure"s in there than I would like... Ian. > Using Xen 4.5 and Kernel 3.19. > > 76: 11304 0 149579 0 0 0 > 0 0 xen-pirq-msi 0000:00:1f.2 > 77: 1243 0 0 35447 0 0 > 0 0 xen-pirq-msi radeon > 78: 82521 0 0 0 0 0 > 0 0 xen-pirq-msi xhci_hcd > 79: 23 0 0 0 0 0 > 0 0 xen-pirq-msi mei_me > 80: 11 0 0 0 0 741 > 0 0 xen-pirq-msi em1 > 81: 350 0 0 0 1671 0 > 0 0 xen-pirq-msi iwlwifi > 82: 275 0 0 0 0 0 > 0 0 xen-pirq-msi snd_hda_intel > > > > The USB controller is an Intel C210: > > 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Dell Device 053e > Flags: bus master, medium devsel, latency 0, IRQ 78 > Memory at f7f20000 (64-bit, non-prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > Kernel modules: xhci_pci > > > > _______________________________________________ > Xen-users mailing list > Xen-users@xxxxxxxxxxxxx > http://lists.xen.org/xen-users _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |