Re: [Xen-devel] netfront/netback multiqueue exhausting grants

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Thu, 2016-01-21 at 10:56 +0000, David Vrabel wrote:
> On 20/01/16 12:23, Ian Campbell wrote:
> > There have been a few reports recently[0] which relate to a failure of
> > netfront to allocate sufficient grant refs for all the queues:
> > 
> > [ÂÂÂÂ0.533589] xen_netfront: can't alloc rx grant refs
> > [ÂÂÂÂ0.533612] net eth0: only created 31 queues
> > 
> > Which can be worked around by increasing the number of grants on the
> > hypervisor command line or by limiting the number of queues permitted
> > by
> > either back or front using a module param (which was broken but is now
> > fixed on both sides, but I'm not sure it has been backported everywhere
> > such that it is a reliable thing to always tell users as a workaround).
> > 
> > Is there any plan to do anything about the default/out of the box
> > experience? Either limiting the number of queues or making both ends
> > cope
> > more gracefully with failure to create some queues (or both) might be
> > sufficient?
> > 
> > I think the crash after the above in the first link at [0] is fixed? I
> > think that was the purpose of ca88ea1247df "xen-netfront: update
> > num_queues
> > to real created" which was in 4.3.
> 
> I think the correct solution is to increase the default maximum grant
> table size.

That could well make sense, but then there will just be another higher
limit, so we should perhaps do both.

i.e. factoring in:
 * performance i.e. ability for N queues to saturate whatever sort of link
   contemporary Linux can saturate these days, plus some headroom, or
   whatever other ceiling seems sensible)
 * grant table resource consumption i.e. (sensible max number of blks * nr
   gnts per blk + sensible max number of vifs * nr gnts per vif + other
   devs needs) < per guest grant limit) to pick both the default gnttab
   size and the default max queuers.

(or s/sensible/supportable/g etc).

> Although, unless you're using the not-yet-applied per-cpu rwlock patches
> multiqueue is terrible on many (multisocket) systems and the number of
> queue should be limited in netback to 4 or even just 2.

Presumably the guest can't tell, so it can't do this.

I think when you say "terrible" you don't mean "worse than without mq" but
rather "not realising the expected gains from a larger nunber of queues",
right?.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel