Re: [Xen-devel] [Patch 2 of 2]: PV-domain SMP performance Linux-part

[George Dunlap <George.Dunlap@xxxxxxxxxxxxx>]

On Fri, Jan 16, 2009 at 5:41 PM, Jeremy Fitzhardinge <jeremy@xxxxxxxx> wrote:
> Yes, that's more or less right.  Each lock has a count of how many cpus are
> waiting for the lock; if its non-zero on unlock, the unlocker kicks all the
> waiting cpus via IPI.  There's a per-cpu variable of "lock I am waiting
> for"; the kicker looks at each cpu's entry and kicks it if its waiting for
> the lock being unlocked.
>
> The locking side does the expected "spin for a while, then block on
> timeout".  The timeout is settable if you have the appropriate debugfs
> option enabled (which also produces quite a lot of detailed stats about
> locking behaviour).  The IPI is never delivered as an event BTW; the locker
> uses the event poll hypercall to block until the event is pending (this
> hypercall had some performance problems until relatively recent versions of
> Xen; I'm not sure which release versions has the fix).
>
> The lock itself is a simple byte spinlock, with no fairness guarantees; I'm
> assuming (hoping) that the pathological cases that ticket locks were
> introduced to solve will be mitigated by the timeout/blocking path (and/or
> less likely in a virtual environment anyway).
>
> I measured a small performance improvement within the domain with this patch
> (kernbench-type workload), but an overall 10% reduction in system-wide CPU
> use with multiple competing domains.

This is in the pv-ops kernel; is it in the Xen 2.6.18 kernel yet?

The advantage of the block approach over yielding is that you don't
have these crazy priority problems:  The reason v0 (who is waiting for
the spinlock) is running right now and v1 (which holds the spinlock)
is not is usually because v1 is out of credits and v0 isn't; so
calling "schedule" often just results in v0 being chosen as the "best
candidate" over again.  The solution in the patch I sent is to
temporarily reduce the priority on a yield; but that's inherently a
little unpredictable.  (Another option might be to re-balance credits
to vcpus on a yield.)

The disadvantage of this approach is that it is rather complicated,
and would have to be re-implemented for each OS.  In theory it should
be able to be implemented in Windows, but it may not be that simple.
And it's got to be implemented all-or-nothing for each spinlock; i.e.,
if any caller of the spin_lock() for a given lock blocks, all callers
of spin_unlock() on that lock need to know to wake the blocker up.  I
don't expect that to be a problem in Windows, but it may be.

Another thing to consider is how the approach applies to a related
problem, that of "syncronous" IPI function calls: i.e., when v0 sends
an IPI to v1 to do something, and spins waiting for it to be done,
expecting it to be finished pretty quickly.  But v1 is over credits,
so it doesn't get to run, and v0 burns its credits waiting.

At any rate, I'm working on the scheduler now, and I'll be considering
the "don't-deschedule" option in due time. :-)

Peace,
 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel