Re: [Xen-devel] [Xen-users] Grant reference batch transmission

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Wed, Mar 11, 2015 at 10:24:48AM +0000, Ian Campbell wrote:
> On Wed, 2015-03-11 at 10:12 +0000, Gareth Stockwell wrote:
> > Hi Ian,
> > 
> > Thanks for your reply.
> > 
> > On Tue, Mar 10, 2015 at 16:22:40, Ian Campbell wrote:
> > >
> > > > What is the recommended way for the donor to transmit a batch of
> > > > grant references?  I assume that this requires the donor to pack
> > > > references into an index page, grant foreign access to the index and
> > > > transmit the index grant reference.  Does Linux provide any way to
> > > > do this, or are xenbus drivers expected to implement their own batch 
> > > > transmission?
> > >
> > > A bit of each. You would indeed want to setup a shared page and push
> > > the references into it, and Linux (/the Xen interface headers) provide
> > > some helpers for this sort of thing, but each driver largely sets
> > > things up themselves using a specific ring request format etc.
> > >
> > > As far as setup of the ring itself goes typically the frontend would
> > > allocate one of its pages, grant it to the backend and communicate
> > > that to the backend via xenstore. Most drivers use a little start of
> > > day synchronisation protocol based around the "state" keys in the
> > > front and backend xenstore dirs, working through the states in enum
> > > xenbus_state
> > > XenbusState* from xen/include/public/io/xenbus.h. It's assumed that
> > > this setup is infrequent (i.e. corresponds to plugging in a new disk
> > > etc)
> > >
> > > In Linux (for most drivers at least, yours may not fit this
> > > infrastructure) that state machine can be driven from the
> > > .otherend_changed callback in the struct xenbus_driver ops struct.
> > 
> > We have implemented front/backend drivers which perform this handshake
> > via xenstore state keys, and which share a single page allocated by
> > the frontend.
> > 
> > I think this gives us two options for grant reference batch transmission:
> > 
> > 1. Send the grant references via the ring buffer.
> > This doesn't require any additional allocations, but means that if the
> > number of grant references in the batch is greater than
> > O(sizeof(ringbuffer) / sizeof(grant_ref_t)), cycling through the ring
> > will be required.
> 
> Correct. In fact it's a bit worse because the ring pointers steal a bit
> of space out of the shared page. You might also find that in practice
> you want an id in the request which is echoed back in the response e.g.
> to handle out of order completion (depends on your use case though),
> which would increase sizeof(grant_ref_t) (which is really
> sizeof(my_req_t)).
> 
> What sorts of batch sizes are you expecting to see in your use case?
> 
> > 2. Allocate and share one or more "index" page(s) which hold the grant 
> > references.
> > This means that only a single grant_ref_t needs to be sent via the
> > ring, but at the cost of allocating additional memory for the index.
> > If multiple index pages are required, they could be chained together
> > by appending to index page N a grant reference pointing to page N+1.
> > 
> > AFAICS the existing drivers use approach #1; is there any precedent
> > for #2?
> 
> There have been patches for both net and blk at various points which
> implemented "multipage rings" in order to get around the limitation on
> the number of requests/responses which can git in a single page. I'm not
> sure what the status of those is (Roger, Wei?) but I would assume/hope
> that they also included some improvements to the common infrastructure
> to make it simpler to arrange for this.
> 

Search for "xenbus_client: extend interface to suppurt multi-page ring"
for the common infrastructure bits.

I think the latest public posting is 

  <1422008071-27643-1-git-send-email-bob.liu@xxxxxxxxxx>

The same thread also contains changes to blk drivers to make use of that
feature.

Wei.

> The other approach, which I believe is used by the blk protocol today,
> is "indirect descriptors" where the ring request contains a gref which
> references another page which can then be packed full of grant
> references. That has the downside of one more map call per request
> (although these can still be batched by the backend) but lets you pack a
> lot more "bandwidth" into a single ring.
> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel