Dear all,
One of the hardest problems to solve in Mirage over the years has been the incredible myriad of mechanisms to manage library compilations for all our different targets. This is, of course, a success disaster of sorts to have now — when we started the project, we had almost no libraries and just two hardware targets and this wasn’t a problem! Today, we have hundreds of packages built and many more targets, with an increasing number coming from the external community (since any pure OCaml library will do the trick). You can browse to http://docs.mirage.io to see some of them.
To manage this growth, we’ve contributed and built lots of core OCaml tooling along the way as well. Thomas Gazagnaire initially lead the construction of opam with MirageOS in mind, and we heavily integrated the opam package workflow into MirageOS 3, along with functoria by Gabriel Radanne to keep track of things. While packaging has been in good shape as a result of opam, build systems have historically not been well integrated in OCaml. Over the years in Mirage, we’ve used every combination from make (in the early days), to omake (in xen) to ocamlbuild (with an epic myocamlbuild) to oasis (that generated ocamlbuild files). All of these were steady improvements over the previous, but none were completely satisfactory.
Over the past eighteen months, we’ve been working on unifying the build infrastructure in Mirage using the dune build system [4]. Features we need for Mirage were patiently added to dune over the last year by Rudi Grinberg, and many of us have ported a ton of libraries to Dune over the past year, and recently Lucas Pluvinage has completed the very complex integration of the frontend CLI tool and gotten us to a significant milestone! Unikernels for multiple targets can now be simultaneously built and incrementally developed, with very consistent cross-compilation and flags tracking!
I wanted to round up why this is important for Mirage with a summary of the improvements:
- Speed: Dune builds are fast compared to oasis due to optimal build rules. As a rough metric, there is a 2-3x speed improvement over using oasis/ocamlbuild in our libraries.
- Usability: Dune does many things out of the box, such as generation and maintenance of merlin files.
- Monorepos: Dune supports assembling monorepos by placing packages into a subdirectory, and building it all in one go with a toplevel `dune build` [1]. I’ve got a simple prototype tool that assembles so-called ‘duniverses’ as a simple way to track all source code that goes into a unikernel in one git repo [3]. The workflow with this is amazing for revving mirage-types, since it can be done in a single go instead of having 10+ opam pins.
- Configuration: Dune has a ‘configurator’ library that is just another build target, so we can unify many of the hacks around pkg-config in one place
- Variants: The biggest addition to the build system has been to support the “cmi linking hack” as a first-class feature [2], so that we can track C stubs cross-compiled for different backends in a principled way. This opens the door for us to support third-party libraries having C stubs, which to date has been very difficult.
More broadly, a number of us are actively hacking on opam, dune, odoc, ocamlformat and other ocaml tools to continue to improve the Mirage developer experience. This is a good time to start moving towards the Mirage 4 release with some of these improvements exposed to the user. Some of the active PRs towards the release that does this (primarily opened by Lucas Pluvinage in his very productive internship at Tarides in recent months are):
If you have any questions, concerns or feature requests, now is a good time to bring them up. The next steps are even more exciting: with the consistent cross-compilation support now available, we should be able to upstream ESP32 (again, work by Lucas in his internship at Cambridge last year) which gives us support for interesting embedded microcontrollers and IoT devices. There is also the forthcoming RISC-V support for open hardware that KC and Malte are working on at IIT-M, thanks to Nicolas Ojeda Bar’s OCaml patch. All in all, this should make Mirage 4 a really fun release to hack around with!
While I’ve covered just tooling in this post, there are of course numerous other improvements in libraries and interfaces (thanks to Hannes Menhert here) and targets (Martin Lucina, Mindy Preston, Thomas Leonard and many others working on Solo5 and Xen here). Switching the tooling to write down some of our technical debt should make life a lot simpler for all this other work. My primary request is that if there is something wrong with the current tooling (especially opam or dune), that we make sure we collectively create an upstream issue for it. Several of the core Mirage developers such as Thomas Gazagnaire or myself are also maintainers of the upstream tooling, and I’d like to make sure that we remain as close to upstream as possible. This way, we can focus our efforts in Mirage on the unikernel end of things where we want to change the world :-)
regards, Anil
(PS: if I haven’t namechecked you, it’s not because I’ve forgotten you — I’m quite staggered and humbled by the number of contributions that have gone into this Mirage 3->4 release cycle all around. Thank you all!) | 
