Re: [MirageOS-devel] copying packets

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

On 1 Nov 2015, at 11:12, Hannes Mehnert <hannes@xxxxxxxxxxx> wrote:
> 
> Hi (sorry if you already saw this),
> 
> recently Luke Gorrie explained whether packet copies are expensive or
> not -- https://github.com/SnabbCo/snabbswitch/issues/648 -- on modern
> CPUs.  Even if the packet is not page aligned, it seems to be fair
> enough to not have to think about copies of packets, and just do it ;)
> 
> (Downside for mirage is certainly that there's still no inline assembly
> in OCaml, thus we've to call out to a C function which seems to be
> slightly expensive.)

I think it's really important to distinguish between the mechanism uses
for copying, and the APIs that are exposed at the Mirage API level.
Some things to consider if the API bakes in copying:

- stacking abstractions gets linearly more expensive, whereas with
  a zero-copy API references to memory can just be passed around
  through multiple depths.  In Xen, grant references (which are handles
  to pages that contain the data) can be passed across multiple
  levels of backend/frontend drivers before the data itself is
  accessed.

- copying is fast on modern x86, but other architectures with
  different memory/CPU tradeoffs will suffer.  Those poor Cubieboards...

- on the same point, memory bandwidth may be in surplus now, but
  is certainly not an infinite resource.  On a modern NUMA system,
  you need to get "lines" of parallel bandwidth out of a multiqueue
  NIC, through the memory, via a CPU, and back out again.  If the
  scheduler decides to cross NUMA domains, latency will rise and
  throughput will drop quite suddenly.  I gave a talk about some
  measurement work we did on this at FOSDEM a few years ago:
  http://anil.recoil.org/talks/fosdem-io-2012.pdf
  http://fable.io (for raw data)

- some application domains like proxies really benefit from a 
  zero-copy API since they are passing through the bulk of the
  data untouched.

Note that with a zero-copy API, the low-level mechanism we use to
do the data transfer might still be a memory copy.  This is indeed
what Xen does at the device driver level, but it can always be shifted
back to a page-table flip at any point.

We're doing "ok" with zero-copy APIs in Mirage at the low-levels, but
it falls apart slightly in the higher level libraries when regexps
and other niceties are needed.  Cohttp for example drops backs to
strings, and it would be amazing to have it just take Cstruct vectors
instead and process them in-place so that higher-level apps would
all be able to do zero-copy instead.

cheers,
Anil

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel