Re: [Xen-devel] [Xen-users] Grant reference batch transmission

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

Hi Gareth,

I think this counts as a -devel question, so I've added -devel and moved
-users to bcc (-users is more for end users). I've done less quote
trimming than usual for the other folks on -devel.

On Tue, 2015-03-10 at 14:15 +0000, Gareth Stockwell wrote:
> What is the recommended way to transmit batches of grant references
> between Linux domains?

> I need to share large regions of memory between domains.  I understand
> that the grant table API can be used to share memory, with one grant
> reference being created per page (frame) to be shared:

>     gref = gnttab_grant_foreign_access(otherend_id, frame, readonly);

>     or

>     err = gnttab_alloc_grant_references(num_grefs, &gref_head);
>     gref = gnttab_claim_reference(&gref_head);
>     err = gnttab_grant_foreign_access_ref(gref, otherend_id, frame,
> readonly);
> 
>  
> 
> In order to share a large number of pages, it is desirable to minimise
> both the number of hypercalls required in each domain, and the number
> of messages (e.g. xenstore writes) required to transmit grant
> reference(s) from the donor to the recipient.
> 
>  
> 
> I see that gnttab_grant_foreign_access just updates grant table fields
> in a page which is mapped into the donor, and does not require a
> hypercall.  In the recipient domain, multiple grant references can be
> mapped by gnttab_map_refs using a single GNTTABOP_map_grant_ref
> hypercall (assuming that the target memory is not paged out).

Correct. Granting access to a page is just a case of writing to a local
page and the mapping interface is batched.

> What is the recommended way for the donor to transmit a batch of grant
> references?  I assume that this requires the donor to pack references
> into an index page, grant foreign access to the index and transmit the
> index grant reference.  Does Linux provide any way to do this, or are
> xenbus drivers expected to implement their own batch transmission?

A bit of each. You would indeed want to setup a shared page and push the
references into it, and Linux (/the Xen interface headers) provide some
helpers for this sort of thing, but each driver largely sets things up
themselves using a specific ring request format etc.

The actual ring structure helpers are in
xen.git/xen/include/public/io/ring.h you would define a request and
response pair (e.g. containing one or more grefs per request) and then
use the macros from ring.h to setup and use the shared ring data
structures.

(NB: xen.git/xen/include/public corresponds to
linux.git/include/xen/interface)

The ring macros include provision for batching and deferred
notifications, so you can balance the number of grefs per request vs.
multiple requests based on your needs.

As far as setup of the ring itself goes typically the frontend would
allocate one of its pages, grant it to the backend and communicate that
to the backend via xenstore. Most drivers use a little start of day
synchronisation protocol based around the "state" keys in the front and
backend xenstore dirs, working through the states in enum xenbus_state
XenbusState* from xen/include/public/io/xenbus.h. It's assumed that this
setup is infrequent (i.e. corresponds to plugging in a new disk etc)

xen/include/public/io/blkif.h has an example of how that works in the
case of the blk driver.

In Linux (for most drivers at least, yours may not fit this
infrastructure) that state machine can be driven from
the .otherend_changed callback in the struct xenbus_driver ops struct.

http://wiki.xen.org/wiki/XenBus covers some of this in the first 3rd,
but TBH it's not as helpful as it could be. I thought we had something
better somewhere (a whitepaper or something), but I can't find any sign
of such a thing, perhaps someone on the list has a reference to such a
thing.

Other than that there is the code in Linux. I think both net and blkback
put most of the initial setup xenbus stuff in their respective
drivers/{block,net}/xen-{blk,net}back/xenbus.c.

For the frontend (drivers/{block,net}/xen-{blk,net}front.c) it's in the
single file.

In both cases the .otherend_changed hook is probably the place to start.

I hope that helps.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel