Re: A unikernel for building unikernels

[Nick Betteridge <lists.nick.betteridge@xxxxxxxxx>]

Hi Hannes,

Thanks for your detailed reply - very useful and a great overview of how 
you and Robur handle the compilation chain.
Thanks for the correct ocamlnat link - I should have asked at the time!

I also naively overlooked the issue of the C compiler/linker being 
available in a mirage unikernel.
I'll take a good look at both the ocaml ocamlnat and Robur's platform 
binaries using docker/jails.
I guess the only alternative is to use something like Buildroot + 
ocaml-monorepo and run it virtually with kvm etc.
Anyhow, again, thanks for taking the trouble to give a detailed overview.

Best,

Nick


On 12/16/24 14:08, Hannes Mehnert wrote:
> Dear Nick,
>
> On 16/12/2024 11:42, Nick Betteridge wrote:
> > Does anyone know of any unikernels that can build unikernels for the 
> > local platform, using something like ocaml-monorepo and, perhaps, irmin?
> I'm afraid I'm not aware of such a unikernel - though that'd be great, 
> one that is able to compile itself (let's keep the C objects for the 
> OCaml runtime as given).
> > Hannes has suggested looking at something like ocamlnat [1], which would 
> What I meant was not that ocamlnat, but the one provided by the OCaml 
> compiler distribution -- on the 4.14 branch the toplevel/native 
> directory (https://github.com/ocaml/ocaml/tree/4.14/toplevel/native) 
> -- make ocamlnat
> > be one way of tackling the issue, but I was wondering if anyone had 
> > considered or implemented any other approaches to solve this  - I 
> > would really love to move away from using the relatively bulky Docker 
> > infrastructure and associated maintenance.
> What we use is actually docker (or FreeBSD jails):
> - we use the `mirage configure` generated opam file(s) as input (in 
> our unikernel-repo https://git.robur.coop/robur/unikernel-repo)
> - we then use builder (https://github.com/robur-coop/builder) as a 
> client/server architecture to schedule builds from the scheduler 
> (server) to the workers (that are polling for builds)
>   - here, we once say 'builder-client orb-build hello 
> mirage-unikernel-hello-hvt' (where "hello" is the job name, and 
> "mirage-unikernel-hello-hvt" the opam name), and then get a daily 
> build thereof
>   - manually we schedule a build with "builder-client execute hello"
>   - the builder-worker is running within a jail/docker container to 
> have a think layer of dependencies
> - the builder driver is orb (https://github.com/robur-coop/orb) that 
> basically uses the opam API to execute the build while preserving 
> information required for reproducing the build
> Now, another stepping stone is builder-web 
> (https://git.robur.coop/robur/builder-web/) where builder uploads the 
> builds and we have a nice web interface (https://builds.robur.coop) 
> with failed builds (https://builds.robur.coop/failed-builds), some 
> visualizations (binary size, dependencies), a database of the builds 
> (we can compare builds - i.e. 
> https://builds.robur.coop/compare/c43f6d82-9a28-4482-94e9-cf14e50816c0/920b68e8-1ed7-47ee-a5a4-eacef52e1f92 
> -- and if you look on the first build, it resulted in a 7.75MB hvt 
> image, the second only a 6.97MB -- while the difference is an update 
> of charrua that dropped sexplib dependencies), and finally also binary 
> package repositories (pkg.robur.coop / apt.robur.coop) for the 
> utilities just mentioned: orb. builder, builder-web, (and also 
> albatross, solo5).
> So, on our build machines we use these package repositories to "apt 
> update && apt upgrade". Bootstrapping may be a bit tedious - that's 
> why we provide the binaries for common platforms / OS (if yours is 
> missing, don't hesitate to complain).
>
> Back to the 'self-building MirageOS unikernel' - I guess apart from 
> the C compiler, and the linking step this could be achieved with a 
> reasonable effort.
>
> Hope that helped,
>
> Hannes