Mirari, the tool you've all been waiting for!

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

During Thomas' recent visit to Cambridge, we burnt through all the remaining 
TODOs to get a Mirage developer release out the door before ASPLOS in March. 
While all the libraries are in reasonable shape (just need documentation), the 
process of configuring and building Mirage applications is still difficult.  

The reason for this is an intrinsic part of the Mirage philosophy: the library 
OS model requires that libraries hold the mechanisms to do things, with the 
policy in the application itself.  In practise of course, this means hardcoding 
things like network configuration in your web server, or shell scripts to 
crunch up filesystems into static ML modules.

This needs cleaning up: in addition to the Xen backend, we also have a 
fledgling FreeBSD kernel module, a Javascript layer, an NS3 simulator, an HVM 
MiniOS which should run on KVM with a bit of hacking, and soon a uBoot-based 
Raspberry Pi port!  We are also beginning work on building real systems using 
Mirage, as well as working on distributed infrastructure to coordinate many Xen 
instances.

So Thomas and I have been hacking on Mirari, which takes a single configuration 
file and splits the lifecycle of a Mirage application into three distinct 
segments:

* configuration: it scans the config file, checks for any missing OPAM 
packages, and installs them if missing.  It also looks for any filesystem 
directives and calls mir-crunch to generate the static ML files.  All of this 
is glued together into an autogenerated main.ml which is the entry point for 
the application.

* build: it runs the OCaml build (via Vincent's obuild), and then issues any 
backend-specific commands (such as the Xen link, ocamlclean, or whatever else 
we dream up).

* run: this is yet to be implemented, and the reason for this mail.  Running a 
Mirage application is quite stateful: in the case of Xen, we want to monitor 
the kernel, attach a console, and so on.  Similarly for UNIX, one can imagine 
the socket version opening up a control channel for Mirari to listen on.  An in 
the kFreeBSD backend, this would be done via ioctls or other kernel/user 
interfaces.

So I'm going to extend Mirari to add `run` support which is stateful.  Each run 
will give the application a unique ID, stored locally, and enough information 
to poll the particular instance.  Every deployment has a different way to track 
it (Amazon EC2 vs XM vs Xenopsd are all completely different).

In the longer term, the configuration file format will almost certainly become 
a DSL to help control the policy a bit better.  We'll design that after we get 
more experience with building applications like this though.  I'm sure Raphael 
will chime in with "NO FRP!" at this point :-)

Thoughts welcome.  You'll need vincenthz/obuild and mirage/mirari installed, 
and the latest mirage/mirage-www works with it.  The website is *so* much 
simpler now without all the OASIS and shell script bits, so thank you to Thomas 
and Vincent for your respective tools!

https://github.com/mirage/mirage-www/tree/master/src
(www.conf for an example file)

-anil