Re: [Xen-devel] [RFC PATCH 2/2] gnttab: refactor locking for better scalability

[Matt Wilson <msw@xxxxxxxxx>]

On Tue, Nov 12, 2013 at 08:07:03AM +0000, Keir Fraser wrote:
> On 12/11/2013 07:18, "Matt Wilson" <msw@xxxxxxxxx> wrote:
> 
> >> Is there any concern about writer starvation here? I know our spinlocks
> >> aren't 'fair' but our rwlocks are guaranteed to starve out writers if there
> >> is a steady continuous stream of readers. Perhaps we should write-bias our
> >> rwlock, or at least make that an option. We could get fancier but would
> >> probably hurt performance.
> > 
> > Yes, I'm a little concerned about writer starvation. But so far even
> > in the presence of very frequent readers it seems like the infrequent
> > writers are able to get the lock when they need to. However, I've not
> > tested the iommu=strict path yet. I'm thinking that in that case
> > there's just going to be frequent writers, so there's less risk of
> > readers starving writers. For what it's worth, when mapcount() gets in
> > the picture with persistent grants, I'd expect to see some pretty
> > significant performance degradation for map/unmap operations. This was
> > also observed in [1] under different circumstances.
> 
> The average case isn't the only concern here, but also the worst case, which
> could maybe tie up a CPU for unbounded time. Could a malicious guest set up
> such a workload? I'm just thinking we don't want to end up with a DoS XSA on
> this down the line. :)

Indeed.

> > But right now I'm more curious about cache line bouncing between all
> > the readers. I've not done any study of inter-arrival times for
> > typical workloads (much less some more extreme workloads like we've
> > been testing), but lock profiling of grant table operations when a
> > spinlock was used showed some pretty long hold times, which should
> > translate fairly well to decent rwlock performance. I'm by no means an
> > expert in this area so I'm eager to hear the thoughts of others.
> 
> In the read-heavy case the only improvement would be with the old
> Linux-style biglock (spinlock per CPU; writers must take all spinlocks), or
> working out a lock-free scheme for readers (perhaps making use of RCU).
> 
>  -- Keir
> 
> > I should also mention that some of the improvement I mentioned from
> > 3,000 MB/s to 3,600 MB/s was due to avoiding the m2p override spinlock
> > in dom0.
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel