Re: [MirageOS-devel] Rust unikernels

[Thomas Leonard <talex5@xxxxxxxxx>]

On 24 July 2015 at 23:28, Geoffroy Couprie <contact@xxxxxxxxxxxxxxxxxxx> wrote:
> Hi Thomas,
>
> On Fri, Jul 24, 2015 at 3:02 PM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
>> On 24 July 2015 at 13:00, Geoffroy Couprie <contact@xxxxxxxxxxxxxxxxxxx> 
>> wrote:
>>> Hello!
>>>
>>> A chat on Twitter showed some interest in unikernels developed in
>>> Rust, and I was pointed here for further discussion.
>>>
>>> First, I want to acknowledge the amazing work done on MirageOS by
>>> everybody here. I have followed it from afar for some time, and there
>>> has been great progress to make it available and usable.
>>>
>>> Second, I know making a unikernel system means years of work and a
>>> team of developers and testers. I am not planning to attempt it alone.
>>> What I want is to sort out the skills needed, the big pitfalls, the
>>> important milestones, and know about the unknown unknowns. Then push
>>> for that project in the Rust community.
>>>
>>> From the beginning, people experimented with OS development in Rust
>>> (cf https://github.com/ryanra/RustOS
>>> https://github.com/thepowersgang/rust-barebones-kernel or
>>> https://github.com/charliesome/rustboot ). There is also a very active
>>> IRC channel, #rust-osdev on irc.mozilla.org
>>>
>>> From what I understand, the bare minimum to experiment would be an OS
>>> booting and communicating with a network card, scheduling and task
>>> switching, remote debugging, a network stack. Also, tools to help in
>>> building, deploying and testing applications.
>>> The ryanra/RustOS project seems well advanced, but it could use a network 
>>> stack.
>>
>> Hi Geoffroy,
>>
>> Have you thought about running Mirage on top of a minimal Rust OS
>> kernel to start with? Then you could implement new drivers in Rust
>> alongside existing Mirage code. As a garbage-collected language, OCaml
>> can't be used to implement the very lowest levels of Mirage (e.g.
>> malloc) and we currently use (my branch of) Mini-OS [1] for running as
>> a unikernel on Xen. A common question is whether we could replace this
>> C code with Rust. I'd be interested to help out if so - I've been
>> meaning to take another look at Rust now it's stable.
>
> This is interesting, someone else proposed me that today too. This
> would be a good strategy to get something up and running quickly
> enough. Basically, Rust can build to a static library which can be
> linked to C code, but until there's a memory allocator available,
> things can be tricky.
> This is how RustOS does it: a small part written in C, which links to
> a Rust library that will handle the rest.

What does the C code do? Is this for connecting to code in other
languages, or something that Rust can't do by itself?

> The core parts of MiniOS could probably not be reimplemented right
> away, but drivers would be doable. From what I see, the first step is
> to communicate with the Xenbus and the Xenstore, right? Then have some
> ring buffers and a way for xen to signal that some data is available
> (I'm currently looking at netfront.c).

The only driver needed for Mirage is the console driver, for early
boot messages. Mirage has its own OCaml implementations of Xenbus and
Xenstore, so replacing them isn't needed to get rid of the C. Much of
Mini-OS is optional and unused (and it would be better to package the
optional bits separately really).

[...]
>>> - the system can be monotask at first, but preemtive threading should
>>> happen at some point
>>
>> If you have promises (and make all your low-level APIs non-blocking)
>> then you can get a very long way without preemptive threading,
>> although it would be good to have it eventually.
>
> Could you tell me more about this? This approach looks interesting.

Every function in Mirage that might need to wait instead immediately
returns a promise. e.g. the read function is defined as:

val read: flow -> [`Ok of buffer | `Eof | `Error of error ] io

(https://github.com/mirage/mirage/blob/d2d4502be0e53de16c915ac72dee008dcaf62991/types/V1.mli#L127)

The type "X io" means "promise of an X" or "light-weight thread that
will produce an X". Normally Mirage is used with the Lwt library, in
which case "io" is defined to be "Lwt.t":

https://github.com/mirage/mirage/blob/d2d4502be0e53de16c915ac72dee008dcaf62991/types/V1_LWT.mli#L25

Functions processing promises can be chained together in a similar way
to non-promise functions, using the >>= operator. It's very useful,
and most languages have something similar, though the names vary. A
quick Google turned up this (incomplete-looking) Rust library:
https://github.com/lucidd/rust-promise

See https://mirage.io/wiki/tutorial-lwt for more details.

>>> - a part of Rust standard library is already available, I do not know
>>> how much is needed to support the use case of a HTTP server
>>> - building monitoring tools inside the kernel
>>
>> When running under Xen, we share a trace buffer with dom0. This allows
>> you to see what the system is doing, even if it is hung or crashed.
>> See: https://mirage.io/wiki/profiling
>
> That visualization tool looks useful! Do you have remote debuggers
> too? With a serial port or something similar.

Mirage can also be compiled to a Unix process and debugged with gdb
and similar in most cases. I believe Xen also provides for debugging
of guests from dom0, but I've never used that.

> I guess the first thing I could do is to integrate some Rust code in
> MiniOS. Is there a part of the code you would recommend? Something
> well contained, without too many dependencies?

I'd probably start with the page allocator:

https://github.com/talex5/mini-os/blob/master/mm.c

/*
 * Initialise allocator, placing addresses [@min,@max] in free pool.
 * @min and @max are PHYSICAL addresses.
 */
static void init_page_allocator(unsigned long min, unsigned long max)

This gets called early during boot with all the free RAM as the
argument. It provides alloc_pages and free_pages to allocate from this
range.


Another possible starting place is the console driver:

https://github.com/talex5/mini-os/blob/master/console/console.c

That's a bit harder though because you're writing to memory shared
with dom0, so you need to be careful about memory barriers and atomic
operations. I assume Rust can handle this nicely? It would be fun to
see how to use Rust's ownership types to ensure that other bits of
code can't write to parts of a shared buffer that dom0 may be reading,
etc.


-- 
Dr Thomas Leonard        http://roscidus.com/blog/
GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel