Re: [Xen-devel] [Xen-users] Grant reference batch transmission

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Wed, 2015-03-11 at 10:12 +0000, Gareth Stockwell wrote:
> Hi Ian,
> 
> Thanks for your reply.
> 
> On Tue, Mar 10, 2015 at 16:22:40, Ian Campbell wrote:
> >
> > > What is the recommended way for the donor to transmit a batch of
> > > grant references?  I assume that this requires the donor to pack
> > > references into an index page, grant foreign access to the index and
> > > transmit the index grant reference.  Does Linux provide any way to
> > > do this, or are xenbus drivers expected to implement their own batch 
> > > transmission?
> >
> > A bit of each. You would indeed want to setup a shared page and push
> > the references into it, and Linux (/the Xen interface headers) provide
> > some helpers for this sort of thing, but each driver largely sets
> > things up themselves using a specific ring request format etc.
> >
> > As far as setup of the ring itself goes typically the frontend would
> > allocate one of its pages, grant it to the backend and communicate
> > that to the backend via xenstore. Most drivers use a little start of
> > day synchronisation protocol based around the "state" keys in the
> > front and backend xenstore dirs, working through the states in enum
> > xenbus_state
> > XenbusState* from xen/include/public/io/xenbus.h. It's assumed that
> > this setup is infrequent (i.e. corresponds to plugging in a new disk
> > etc)
> >
> > In Linux (for most drivers at least, yours may not fit this
> > infrastructure) that state machine can be driven from the
> > .otherend_changed callback in the struct xenbus_driver ops struct.
> 
> We have implemented front/backend drivers which perform this handshake
> via xenstore state keys, and which share a single page allocated by
> the frontend.
> 
> I think this gives us two options for grant reference batch transmission:
> 
> 1. Send the grant references via the ring buffer.
> This doesn't require any additional allocations, but means that if the
> number of grant references in the batch is greater than
> O(sizeof(ringbuffer) / sizeof(grant_ref_t)), cycling through the ring
> will be required.

Correct. In fact it's a bit worse because the ring pointers steal a bit
of space out of the shared page. You might also find that in practice
you want an id in the request which is echoed back in the response e.g.
to handle out of order completion (depends on your use case though),
which would increase sizeof(grant_ref_t) (which is really
sizeof(my_req_t)).

What sorts of batch sizes are you expecting to see in your use case?

> 2. Allocate and share one or more "index" page(s) which hold the grant 
> references.
> This means that only a single grant_ref_t needs to be sent via the
> ring, but at the cost of allocating additional memory for the index.
> If multiple index pages are required, they could be chained together
> by appending to index page N a grant reference pointing to page N+1.
> 
> AFAICS the existing drivers use approach #1; is there any precedent
> for #2?

There have been patches for both net and blk at various points which
implemented "multipage rings" in order to get around the limitation on
the number of requests/responses which can git in a single page. I'm not
sure what the status of those is (Roger, Wei?) but I would assume/hope
that they also included some improvements to the common infrastructure
to make it simpler to arrange for this.

The other approach, which I believe is used by the blk protocol today,
is "indirect descriptors" where the ring request contains a gref which
references another page which can then be packed full of grant
references. That has the downside of one more map call per request
(although these can still be batched by the backend) but lets you pack a
lot more "bandwidth" into a single ring.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel