Re: [Xen-devel] blkfront resource use

[Roger Pau Monne <roger.pau@xxxxxxxxxx>]

On Mon, Feb 06, 2017 at 03:53:58AM -0700, Jan Beulich wrote:
> Konrad, Roger,
> 
> we've recently had a report of stuck I/O (with slightly over a hundred
> frontend instances in a single guest), which turned out to be a simple
> lack of configured grants on the system. This raises two questions:
> 
> 1) Shouldn't blkfront cope with this situation, by throttling I/O (but not
> getting stuck), perhaps by forcing requests to be split (or splitting
> them itself) if gnttab_alloc_grant_references() continues to fail over
> a measurable time period?

Right, this was more problematic to implement before, but now that ARM
introduced the code to split bios in blkfront, it shouldn't be that hard to
split a bio into smaller ones if not enough grants can be allocated for the
whole request.

> 2) Isn't the current grant hunger of blkfront a little extreme? An
> instance with all defaults can consume 1k grants (32 segs times
> 32 reqs), but the scaling with ring size, queue count, and number
> of segs can easily get this to 64k or more. A default configured

If blkback is using the default values, a single frontend should only use up to
1056 persistent grants (see max_persistent_grants/xen_blkif_max_pgrants) per
queue. TBH, I didn't really like the change to account persistent grants
per-queue instead of globally [0], because it makes it harder to impose limits
on grant utilization.

> host, however, allows only for 128k grants (256 frames each
> holding 512 entries). Interestingly I've found 
> https://groups.google.com/forum/#!topic/linux.kernel/N6Q171xkIkM
> when looking around - is there a reason this or something similar
> never made it into the driver? Without such adjustment a single
> spike in I/O can lead to a significant amount of grants to be "lost"
> in the queue of a single frontend instance.

IIRC we didn't go for that solution and instead implemented a limit in blkback
that can be set by the system administrator. But yes, it is still possible for
a single blkfront instance to use a huge amount of grants, although only
temporarily. When the IO spike is done (ie: the bio is done) blkfront should
release the grants. If this is a system doing a huge amount of IO the default
amount of grant tables pages should probably be increased.

Roger.

[0] 
https://github.com/torvalds/linux/commit/73716df7da4f60dd2d59a9302227d0394f1b8fcc

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel