Re: mirage support for dynamically loaded modules

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

Great, let us know how you get along with Snowflake!

We do indeed have a toplevel almost working with Mirage.  The difficulties with this are getting a terminal emulator (now almost sorted by Dave, who went a bit further than expected and also has a VNC library).

For a native code toplevel, however, we need to use some JIT techniques to be able to compile code without having an assembler available.  Benedikt Maurer has already done some work on this [1], so his patches need to be investigated and adapted to Mirage (not difficult, but on the stack!)

[1] http://arxiv.org/pdf/1110.1029.pdf

-anil

On 10 Sep 2013, at 17:34, Andrew Lekar <sam.and.tetris@xxxxxxxxx> wrote:

Anil, this is just awesome! Snowflake is exactly the same I intended to do. It even has dynamically loaded modules, shell, filesystem support. Do you intend to implement shell or toplevel in mirage? It should ease os management very much.

2013/9/10 Anil Madhavapeddy <anil@xxxxxxxxxx>

Ah, I see.  One first step here might be to add support for HVM booting of normal guests.  The HVM mode is very similar to PV boot, except:

- you need to configure a hardware timer

- the boot process goes through normal 16-bit bootstrap instead of direct PV boot

- page table manipulation is direct instead of via hypercalls.

- you can interact with emulated device drivers for block/net.

The last point is most important -- it would be easiest to get the scaffolding for device drivers working by being able to evaluate the rest of the system against the PV equivalents.

Another way to go is to investigate Snowflock's hardware device drivers that were written in OCaml: https://github.com/jessicah/snowflake-svn/tree/master/kernel (the license should be clarified with the author first, however, as I didn't spot any LICENSE files in there).

There's another approach which doesn't involve Xen at all -- have you considered taking one of the Raspberry Pi ARM mini operating systems and porting the Mirage libraries to it?  They compile in ARM fine, and it should be possible to adapt Dave's VNC library to rasterise onto HDMI. The advantage of this is that there's a fairly small hardware HDL to port, vs the incredible number of x86 drivers that would need to be supported.  (this isn't relevant if you have a specific x86 hardware set to support, but in general there's no easy way out of x86 support except for re-using kernel drivers in interesting ways).

Finally, you should also check out rump kernels: it should be easy to re-use portions of the NetBSD rump infrastructure to interface with Mirage management -- http://www.netbsd.org/docs/rump/.

cheers

Anil

On 10 Sep 2013, at 11:22, Andrew Lekar <sam.and.tetris@xxxxxxxxx> wrote:

"Got it -- so you want to interface with hardware drivers via passthrough to a driver domain, I think."

No! I do not have any domains with drivers or kernels - only dom0 with mirage. So I have to implement hardware interaction all by myself with a thin layer of xen.

For now I've got working baremetal ocaml runtime and I teared off xenstore support from mirage to run it on dom0. You can think of it as a true baremetal mirage since xen does not offer anything but some debug io and timer support.

Therefore I'm interested in deploying hardware drivers separately from mirage kernel and I'd prefer to have in theese drivers as less C code as possible. Will you give me any suggestions about such mirage configuration?

2013/9/10 Anil Madhavapeddy <anil@xxxxxxxxxx>

Got it -- so you want to interface with hardware drivers via passthrough to a driver domain, I think.

The best way to think of Mirage is as a growing bunch of libraries, glued together by a minimal runtime for whatever platform is targetted and an increasingly sophisticated constraint-based package manager (OPAM).

For example, consider these backends:

- UNIX --- the compiler output is an ELF executable, and started/stopped via a helper binary that passes it file descriptors and performs privileged operations.  It can be passed raw packets via tuntap, or invoke kernel sockets, depending on how it's built.

- Xen -- the compiler output is an object file that is linked to a minimal Xen OS with a linker script to make it PV bootable.  All the drivers are normal OCaml libraries (including the shared memory ring handling).

- _javascript_ -- the compiler output is a JS blob which has externals that are satisfied via other _javascript_ functions made available, and the runtime used is the _javascript_ one.

- kFreeBSD -- the compiler output is linked with the OCaml GC and fixed-point floating point and kernel module scaffolding.  The internal kernel APIs are bound as OCaml libraries.

The final one is the most relevant to you...the easiest way to manage hardware in the first instance is to use FreeBSD as a driver domain (no userspace), and take advantage of existing Mirage libraries to manage the kernel lifecycle.

The most immediate thing to get working in the kernel guest is Xenstore.  This is exposed to a HVM guest via a special MSR which gives you the shared_info page, which in turns gives access to hypercalls and the Xenstore event channel.  Once this is set up, you can use the existing OCaml Xenstore library to handle all the communication to/from the driver domain, without having to worry about reimplementing the protocol in the kernel.  The same goes for other virtual devices such as Netfront, Blkfront, and (thanks to Dave) even a VNC framebuffer if you want a PV framebuffer.

On your other points about filesystems -- Thomas Gazagnaire is working on a Mirage filesystem at the moment (a branch-consistent one dubbed Irminsule), and Dave has a FAT32 implementation, and I wrote a simple static one (in mirage-fs).  We're finding that filesystems aren't often required by our current applications, however.

Hope this helps -- if you let us know more specific use-cases, I can guide you towards more specific solutions.  

cheers,

Anil

On 9 Sep 2013, at 18:21, Andrew Lekar <sam.and.tetris@xxxxxxxxx> wrote:

Thank you Anil.

I am working with baremetal port of mirage (well on top of xen in dom0 domain) and I'm expecting to have a lot of hardware drivers which I'd like to deploy and debug without rebuild of entire mirage image. Also I'd like to have a way of upgrading software and it is better done with in-system compiler or bytecode interpreter. As far as I understood your answer mirage ocaml does have bytecode interpreter but does not support filesystem or other modules sources, am I right?

Unikernel solution is good for me for now but it is way too long to debug anything with unikernel and I expect it to become worse in future.

2013/9/9 Anil Madhavapeddy <anil@xxxxxxxxxx>

On 8 Sep 2013, at 20:25, Andrew Lekar <sam.and.tetris@xxxxxxxxx> wrote:

> Hello everyone.
>
> Is it possible to support dynamically load ocaml modules in mirage? So that I do not rebuild entire mirage kernel but make it once and add additional drivers later. What are the steps required to make it possible? I guess to make it work I should implement filesystem support, bytecode interpreter or ocaml compiler in mirage.

Hi Andrew,

It depends which backend you're thinking of.  In Xen, we have some patches to further make the address space immutable post-boot (by dropping the ability to alter the guest page tables or switch CR3).  In that case, you can't do dynamic loading by design.

However, without this patch, and with a suitable filesystem, network or vchan source for the dynamic module, you could do dynamic loading.  The real question is why you'd want that at all, given it's easier to recompile a Mirage instance and redeploy it rather than go through all the trouble of dynamic loading.  In many ways, that's the whole point of the Xen backend ;-)

-anil