Re: [Xen-devel] [Xen-users] Grant reference batch transmission

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Wed, 2015-03-11 at 13:30 +0000, Gareth Stockwell wrote:
> On Wed, Mar 11, 2015 at 10:54:25, Ian Campbell wrote:
> > > > What sorts of batch sizes are you expecting to see in your use case?
> > >
> > > We need to share hundreds of MB, so (assuming a 4kB guest page size)
> > > the batch size can be thousands of grant references.
> >
> > FWIW, the granularity of a gref is 4kB irrespective of the guest's
> > page size (so e.g. a 64Kb page would need 16 grefs to cover it), so
> > your calculation is correct everywhere.
> >
> > We have been considering allow "superpage" grants of higher order
> > mappings, but no work has been done on that yet. You such a feature be
> > useful or are your hundreds of MB scattered?
> 
> They will be contiguous in 2MB chunks - so higher order grants would be 
> useful.
> 
> What happens to the gref granularity if the stage-2 page size is larger than 
> the stage-1 page size?

I think it is unlikely that Xen would ever support anything other than
4K/2M/1G granules internally, i.e. in stage 2, there just too many
assumption that the basic page size is 4K in the ABI. superpages are
something we can cope with, but changing the base size not so much.

TBH my feeling is that for a hypervisor at least using 2M mappings where
possible is a better option than e.g. using 64K pages.

> 
> > I suppose the 100s of MB is changing reasonable often (e.g. not just a
> > start of day setup thing) otherwise the performance/efficiency of gref
> > communications wouldn't be such an issue.
> 
> It's not start of day (i.e. at driver probe time), but it's not
> particularly frequent either.  The memory being mapped into the remote
> domain contains data for a workload whose processing time is typically
> quite extended.  The time available for establishing the mapping
> should be at least in the tens of milliseconds.

From a back of the envelope calculation I think pushing 100MB worth of
grefs over a single page ring is going to take tens of full rings worth
of grefs, so pushing them individually is going to be pretty tedious and
probably blow the time budget.

Even multipage rings wouldn't cut it unless you went to pretty high
order rings. Some sort of indirect scheme seems like it would be best.
Given tens of ms to get it done I don't think the mapping of the
indirect pages will be to much of a factor if you batch them.

Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel