Re: mutable store on mirage?

[David Scott <scott.dj@xxxxxxxxx>]

Hi Chris.

On Wed, Oct 16, 2013 at 6:55 PM, Christopher Greenhalgh <Chris.Greenhalgh@xxxxxxxxxxxxxxxx> wrote:

is there currently a sensible option for a mutable datastore in mirage?
i'd like to make an interactive web site/service and need at least the equivalent of a persistent mutable key-value store with user-defined keys (e.g. to take a minimal example, key = user ID, value = current user profile information).

a) I guess i could port baardskeerder - has that been done already?

I made an attempt a while ago, back here:

https://github.com/djs55/mirage/tree/baardskeerder

I didn't hook it up to blkif because the blkif wasn't very stable back then. It's worth another look now though.

b) 've had a look at irminsule, but
b.1) at the moment i don't actually get how you would use it in this kind of scenario, e.g. does my user defined key (e.g. user login) map to a tag and a value (e.g. their current profile information) to a value? or does the whole current key/value state (i.e. user ID/value for all current users) map to a single value and the 'current state' to a single tag? or something else??
b.2) is there currently a mirage (non-unix) backend for it?

c) in a previous incarnation of mirage i started to write a mutable (strictly append only) key/value over that version of blkif, but ran out of time/energy; is this obviously a waste of time/duplication of effort or would it be useful/sensible?

d.1) is there any reason for/against persisting directly against blkif in mirage? i,e, with not filesystem abstraction?
d.2) as far as i can see the only up-to-date filesystem in mirage is the crunch RO fs; the fat fs is mostly there but need porting to the latest blkif interface, and there aren't any others at the moment - is that right?!

I think blkif is stable enough to use now, although the interface will probably need to evolve a bit. TBH we'll only really know what the interface should be when we have a bunch of code trying to use it. I've got a branch of mirage-www which replaces the 'crunch' fs with data read from blkif in .tar format:

https://github.com/mirage/mirage-www/pull/38

 

e) the mirage-unix OS has a skeleton blkif (over file) implementation but this is currently unimplemented; is it worth implementing and using this for posix or do we expect to use something like dave's ocaml xen-block-driver and xen-disk to emulate block devices from files instead even on posix targets?

I'd like to see unix blkif implemented (without using the xen implementation). I've got some patches lying around which open files with O_DIRECT and perform unbuffered sector-aligned I/O. I was thinking of making a 'unix-block-driver' with this code in it, mirroring the xen implementation. It's mostly a thin veneer over Lwt_bytes, replacing Bigarray.t with Cstruct.t. In future I was thinking the mirage-platform repo could become the minimal 'boot' code plus module types for Blkif, Netif etc; and all the concrete implementations could be spun out into other repos.

What do people think?

Cheers,

Dave