Re: [MirageOS-devel] MPL compiler and bigarray

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

[ccing the mirageos-devel list]

On 29 Jul 2014, at 10:29, Goswin von Brederlow <goswin-v-b@xxxxxx> wrote:
> 
> I've looked at your melange paper [1] and it says that the MPL
> compiler will use a basis library for I/O and buffer management
> functions. You then show the data type for a light-weight packet
> environment record showing that the actual payload is stored in a
> string.
> 
> I want to use MPL for a high througput protocol that involves lots of
> disk I/O with the linux AIO libarry as well and I need to use page
> aligned bigarrays for that.
> 
> I'm wondering how flexible this is. Would it be easy to replace the
> string with a bigarray in the library or replace the library
> altogether? The advantage of a bigarray is that the data is allocated
> outside the ocaml heap and will never be moved by the GC. This allows
> using it in C stubs while the ocaml runtime lock is released or for
> DMA. Bigarrays can also be created to be page aligned or from an
> existing block of data. Which strings all of that requires copying the
> data.
> 
> The MPL generated code is already fast despite the extra copying the
> string requires. I wonder how much using bigarrays for true zero-copy
> networking and I/O would gain on top of that.

Sorry about the delay getting back to you on this.

While it would be easy to replace MPL with a Bigarray rather than a
string, it's probably not worth the effort to resurrect that tool.

We're leaning towards lightweight views over Bigarrays via Cstruct
these days, and using GADTs+metaprogramming to convert packet grammars
into parsers.   The GADT portion of ASN1 parsing for example:
http://openmirage.org/blog/introducing-asn1

What's not fully integrated yet is the use of Metacaml or ppx/camlp4
to perform the metaprogramming.  MPL's introduction of concrete syntax
can then disappear, and packets can be manipulated directly in the
host language as an EDSL.

For simpler protocols like IPv4 or TCP parsing, Cstruct provides a
camlp4 extension that lets you write things like:

cstruct ipv4 {
  uint8_t        hlen_version;
  uint8_t        tos;
  uint16_t       len;
  uint16_t       id;
  uint16_t       off;
  uint8_t        ttl;
  uint8_t        proto;
  uint16_t       csum;
  uint8_t        src[4];
  uint8_t        dst[4]
} as big_endian


and retrieve the values from a Bigarray representation using the
efficient compiler builtins (4.01.0+) for integer conversions.
See https://github.com/mirage/ocaml-cstruct for more on that.

-anil

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel