Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Wed, 2015-09-02 at 08:53 -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Sep 01, 2015 at 11:09:38PM +0000, Justin Acker wrote:
> > 
> >       From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> >  To: Justin Acker <ackerj67@xxxxxxxxx> 
> > Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>; 
> > boris.ostrovsky@xxxxxxxxxx 
> >  Sent: Tuesday, September 1, 2015 4:56 PM
> >  Subject: Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU 
> > limited to single interrupt
> >    
> > On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
> > > Taking this to the dev list from users. 
> > > 
> > > Is there a way to force or enable pirq delivery to a set of cpus as 
> > > opposed to single device from being a assigned a single pirq so that 
> > > its interrupt can be distributed across multiple cpus? I believe the 
> > > device drivers do support multiple queues when run natively without 
> > > the Dom0 loaded. The device in question is the xhci_hcd driver for 
> > > which I/O transfers seem to be slowed when the Dom0 is loaded. The 
> > > behavior seems to pass through to the DomU if pass through is 
> > > enabled. I found some similar threads, but most relate to Ethernet 
> > > controllers. I tried some of the x2apic and x2apic_phys dom0 kernel 
> > > arguments, but none distributed the pirqs. Based on the reading 
> > > relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done 
> > > to avoid an interrupt storm. I tried IRQ balance and when 
> > > configured/adjusted it will balance individual pirqs, but not 
> > > multiple interrupts.
> > 
> > Yes. You can do it with smp affinity:
> > 
> > https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
> > Yes, this does allow for assigning a specific interrupt to a single 
> > cpu, but it will not spread the interrupt load across a defined group 
> > or all cpus. Is it possible to define a range of CPUs or spread the 
> > interrupt load for a device across all cpus as it does with a native 
> > kernel without the Dom0 loaded?
> 
> It should be. Did you try giving it an mask that puts the interrupts on 
> all the CPUs?
> (0xf) ?
> > 
> > I don't follow the "behavior seems to pass through to the DomU if pass 
> > through is enabled" ?
> > The device interrupts are limited to a single pirq if the device is 
> > used directly in the Dom0. If the device is passed through to a DomU - 
> > i.e. the xhci_hcd controller - then the DomU cannot spread the 
> > interrupt load across the cpus in the VM. 
> 
> Why? How are you seeing this? The method by which you use smp affinity 
> should
> be exactly the same.
> 
> And it looks to me that the device has a single pirq as well when booting 
> as baremetal right?
> 
> So the issue here is that you want to spread the interrupt delivery to happen 
> across
> all of the CPUs. The smp_affinity should do it. Did you try modifying it by 
> hand (you may
> want to kill irqbalance when you do this just to make sure it does not write 
> its own values in)?

It sounds then like the real issue is that under native irqbalance is
writing smp_affinity values with potentially multiple bits set while under
Xen it is only setting a single bit?

Justin, is the contents of /proc/irq/<IRQ>/smp_affinity for the IRQ in
question under Native and Xen consistent with that supposition?

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel