Re: [Xen-devel] Design and Question: Eliminate Xen (RTDS) scheduler overhead on dedicated CPU

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Tue, Mar 24, 2015 at 11:27:35AM -0400, Meng Xu wrote:
> 2015-03-24 7:54 GMT-04:00 George Dunlap <George.Dunlap@xxxxxxxxxxxxx>:
> 
> > On Tue, Mar 24, 2015 at 3:50 AM, Meng Xu <xumengpanda@xxxxxxxxx> wrote:
> > > Hi Dario and George,
> > >
> > > I'm exploring the design choice of eliminating the Xen scheduler
> > overhead on
> > > the dedicated CPU. A dedicated CPU is a PCPU that has a full capacity
> > VCPU
> > > pinned onto it and no other VCPUs will run on that PCPU.
> >
> > Hey Meng!  This sounds awesome, thanks for looking into it.
> >
> 
> :-) I think it is a useful feature for extreme low latency applications.
> â
> 
> >
> >
> > > [Problems]
> > > The issue I'm encountering is as follows:
> > > After I implemented the dedicated cpu feature, I compared the latency of
> > a
> > > cpu-intensive task in domU on dedicated CPU (denoted as R_dedcpu) and the
> > > latency on non-dedicated CPU (denoted as R_nodedcpu). The expected result
> > > should be R_dedcpu < R_nodedcpu since we avoid the scheduler overhead.
> > > However, the actual result is R_dedcpu > R_nodedcpu, and R_dedcpu -
> > > R_nodedcpu ~= 1000 cycles.
> > >
> > > After adding some trace to every function that may raise the
> > > SCHEDULE_SOFTIRQ, I found:
> > > When a cpu is not marked as dedicated cpu and the scheduler on it is not
> > > disabled, the vcpu_block() is triggered 2896 times during 58280322928ns
> > > (i.e., triggered once every 20,124,421ns in average) on the dedicated
> > cpu.
> > > However,
> > > When I disable the scheduler on a dedicated cpu, the function
> > > vcpu_block(void) @schedule.c will be triggered very frequently; the
> > > vcpu_block(void) is triggered 644824 times during 8,918,636,761ns (i.e.,
> > > once every 13831ns in average) on the dedicated cpu.
> > >
> > > To sum up the problem I'm facing, the vcpu_block(void) is trigger much
> > > faster and more frequently when the scheduler is disabled on a cpu than
> > when
> > > the scheduled is enabled.
> > >
> > > [My question]
> > > I'm very confused at the reason why vcpu_block(void) is triggered so
> > > frequently when the scheduler is disabled.  The vcpu_block(void) is
> > called
> > > by the SCHEDOP_block hypercall, but why this hypercall will be triggered
> > so
> > > frequently?
> > >
> > > It will be great if you know the answer directly. (This is just a pure
> > hope
> > > and I cannot really expect it. :-) )
> > > But I really appreciate it if you could give me some directions on how I
> > > should figure it out. I grepped vcpu_block(void) and SCHEDOP_block  in
> > the
> > > xen code base, but didn't found much call to them.
> > >
> > > What confused me most is that  the dedicated VCPU should be blocked less
> > > frequently instead of more frequently when the scheduler is disabled on
> > the
> > > dedicated CPU, because the dedicated VCPU is always running on the CPU
> > now
> > > without the hypervisor scheduler's interference.
> >
> > So if I had to guess, I would guess that you're not actually blocking
> > when the guest tries to block.  Normally if the guest blocks, it
> > blocks in a loop like this:
> >
> > do {
> >   enable_irqs();
> >   hlt;
> >   disable_irqs;
> > } while (!interrup_pending);
> >
> > For a PV guest, the hlt() would be replaced with a PV block() hypercall.
> >
> > Normally, when a guest calls block(), then it's taken off the
> > runqueue; and if there's nothing on the runqueue, then the scheduler
> > will run the idle domain; it's the idle domain that actually does the
> > blocking.
> >
> > If you've hardwired it always to return the vcpu in question rather
> > than the idle domain, then it will never block -- it will busy-wait,
> > calling block millions of times.
> >
> > The simplest way to get your prototype working, in that case, would be
> > to return the idle vcpu for that pcpu if the guest is blocked.
> >
> 
> âExactly! Thank you so much for pointing this out!  I did hardwired it
> always to return the vcpu that is supposed to be blocked. Now I totally
> understand what happened. :-) â

Or you could change the kernel to do an idle poll instead of using 'hlt'.

And idle poll is just:
 do {
        nop;pause;
 } while  (!interrupt_pending);

On Linux I believe you do 'idle=poll' to make it do that.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel