Re: [Xen-devel] [PATCH 2/3] xen: Have schedulers revise initial placement

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Mon, 2016-07-18 at 22:36 +0100, Andrew Cooper wrote:
> On 18/07/2016 19:55, Dario Faggioli wrote:
> > 
> > On Mon, 2016-07-18 at 19:10 +0100, Andrew Cooper wrote:
> > > 
> > > On 16/07/16 15:12, Dario Faggioli wrote:
> > > > 
> > > > On Fri, 2016-07-15 at 19:07 +0100, Andrew Cooper wrote:
> > > > So you have to always keep IRQ enabled, for all scheduling
> > > > operations,
> > > > which is ok for _almost_ all of them, with the only exception
> > > > of
> > > > the
> > > > wakeup of a vcpu.
> > > I know that it is all or nothing.  What specific action about
> > > waking
> > > a
> > > vcpu requires holding a lock?
> > > 
> > > If it is simply re-queueing the vcpu onto the runable queue,
> > > there
> > > are a
> > > number of lockless queuing algorithms which can be used.
> > > 
> > Yes, it's vcpu_wake() that does vcpu_schedule_lock_irqsave()
> > before,
> > among other things, calling SCHED_OP(wake, v).
> Right - this looks easy to fix.  Use a per-pcpu single linked list,
> which can be mutated safely with cmpxchg() rather than locks, have
> vcpu_wake() add v to the linked list, and schedule itself a
> SCHEDULE_SOFTIRQ, (possibly a new SCHEDULE_WAKE_SOFTIRQ with higher
> priority than SCHEDULE_SOFTIRQ).
> 
That's exactly what I'm doing in one of the variant I've implemented so
far (with a dedicated lock, rather than cmpxchg, but I was indeed
looking at removing that):

http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=blobdiff;f=xen/include/xen/softirq.h;h=e62c247fdc41dd4e36ceb5f8094647fa62530c56;hp=0895a162031b64ad6acff6be9d17707dad7a89bf;hb=431423e69c9fe4503d26bd4c657699284e0223b7;hpb=a282bcd6f7751d2ff428ca6c1e14120aa6fcc5fc

:-)

> Then in the schedule softirq can take the vcpu schedule lock without
> disabling interrupts and run the internals of what is currently
> vcpu_wake().  The current content of vcpu_wake() very large for what
> is
> typically executed from interrupt context.
> 
Totally agree.

Something I was reasoning on, and trying to assess by means of
benchmarks is, for instance, when taking out the vcpus from the queue
and waking them, whether to do that always "to completion" (considering
that it's probably even possible that new vcpus are added to the queue
itself while I'm trying to drain it), or in batches (and after each
batch, let Xen do something else and re-trigger the softirq).

And this (and other similar "subtleties") is where the numbers I could
get were not conclusive.

I'll dust off the code and let you know. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel