Re: [MirageOS-devel] Irmin roadmap

[Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx>]

Hi,

I've spent the last week working on the new version of Irmin. This is still not 
finished but it is making good progress. A preliminary ocamldoc page is 
available online[1]. It is not complete and polished yet, but most parts are 
documented.

The main changes are:

- Improve compile time and portability: no more core_kernel and camlp4, now use 
nocrypto and mirage-tc

- No more -pack. All the internal modules are prefixed by Ir_ and are not 
available to the users (the cmi are not installed) and the public module (with 
the public documentation) is Irmin

- All the operations in a store now take a **mandatory** `origin` argument, 
even the read operation [2] (before, only the merge operations where requiring 
it). The origin contains the process/user name, the date, some (maybe 
structured) message, ... (this can be configured by the user). That origin 
value comes from the high-level call ("I want to add that value to my Irmin 
store") and are passed through all the low-level calls ("I need to create a new 
block in the block store, I need to update the head tag to point to the new 
head, etc) so you can very easily track every read and write operations in your 
store. My hope is that we can use that origin value to control the store 
encryption: you can store some keys in the origin and use them to encrypt on 
write and decrypt on read. Need more though on that, but I think we have 
everything needed now.

- The high-level "branch-consistant" signature is simpler [3]. The internals of 
the contents, node, commit and tag stores are not exposed anymore.

- The synchronisation functions are much more efficient and looks quite similar 
to what Git does, although we don't compress the data (yet) (but the slice[4] 
type is abstract, so we could do that later if needed, this involves tweaking 
[5]).

What needs to be done: porting the backends, the http server and checking that 
everything still works fine. Hopefully this will be done early next week.

Best,
Thomas

[1]: http://samoht.github.io/irmin/
[2]: http://samoht.github.io/irmin/Irmin.Store.RO.html
[3]: http://samoht.github.io/irmin/Irmin.Store.BC.html
[4]: http://samoht.github.io/irmin/Irmin.Store.BC.html#TYPEslice
[5]: 
https://github.com/samoht/irmin/blob/ab20999e43a9aa4a3bdb1733a55288080ff2c565/lib/ir_bc.ml#L488

> On 1 Oct 2014, at 21:38, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
> 
> I'm very late on all of these points (although I almost have 1/ done, needs 
> more polishing) but I've generated some docs which might be useful to people 
> starting using the library:
> 
> http://samoht.github.io/irmin/
> 
> Thomas
> 
> On 9 Sep 2014, at 16:57, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
> 
>> Hi all,
>> 
>> Lots of people starting asked me about the status of Irmin. As you might 
>> have notice, I'vent had much time to spend on Irmin last month, but I plan 
>> to fix that. My current priorities are:
>> 
>> 1/ make Irmin compiles on the Xen backend of Mirage.  
>> https://github.com/mirage/irmin/issues/81
>> This means dropping the dependency to core_kernel (for now on). The external 
>> API will not change.
>> 
>> 2/ improve the high-level JSON API. This is related to 
>> https://github.com/mirage/irmin/issues/80 
>> Currently, we only have a JSON API for the low-level stores, and the very 
>> partial implementation of the higher-level calls. Needs to complete and 
>> document that.
>> 
>> 3/ implement a pure-ocaml Git server to be able to at least `git pull` from 
>> it. https://github.com/mirage/ocaml-git/issues/15 
>> Together with 1/ this will make possible to query the state of a running 
>> Mirage unikernel with a simple `git pull <vm-address>`
>> 
>> 4/ implement a simple distributed log server 
>> .https://github.com/mirage/irmin/issues/82 
>> Could use Benjamin's rope implementation, that I first need to release 
>> properly.
>> 
>> 5/ try to come up with a solution to the "unlimited" memory/storage usage 
>> issues https://github.com/mirage/irmin/issues/83
>> This is the most uncertain part of the short-term roadmap. I'll need to 
>> check again what are the limitation of Git shallow copies and see how we can 
>> use them to limit the history size to remember.
>> 
>> There are also few open embarrassing bugs that I want to fix as well 
>> (regarding fd leaks ...). 
>> 
>> If anyone is interested to help me on any of these topics, please feel free 
>> to comment on the related issues on Github -- I'll gladly share the 
>> workload. Also, feel free to reply to that email if you think there is any 
>> important Imrin features that you think are missing now.
>> 
>> Best,
>> Thomas
> 


_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel