Re: [MirageOS-devel] Irmin roadmap

[Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx>]

Hi all,

after 3 months of refactoring (!), 11,5k lines added and 8,8k lines removed, I 
am very happy to announce that I've merged my working patches into the master 
branch of Irmin. I'll pass the remaining of the week fixing the last quirks 
(mainly documentation related, but also around the command-line tool usability) 
before announcing the 0.9.0 release.

One of the main change is the new public API with a proper documentation: 
http://samoht.github.io/irmin/ (although it needs more code examples[1])

Summary of the changes:
* Expose a cleaner public API
* Expose a cleaner API for the Unix backends
* Expose a high-level REST API over HTTP (#80)
* Rename `Origin` into `Task` and use it pervasively through the API
* Add `remove_rec` to remove directories (#74, #85)
* More efficient synchronization protocol between Irmin stores (#11)
* Improve the efficiency of the Git backend
* Remove dependency to `core_kernel` (#22, #81)
* Remove dependency to `cryptokit` and `ocaml-sha` and use `nocrypto` instead
* Fix writing contents at the root of the store (#73)
* Fix the Git backend to stop constantly overwrite `.git/HEAD` (#76)
* Add a limit on concurrently open files (#93, #75)

As usual, feedback is welcome.

The initial goal for the 1.0.0 release was to integrate Mirage and Irmin 
properly. That goal is very close now. What is missing:
- Support the Git synchronisation protocol in Mirage. This means implementing 
Git.Sync.IO [2]
- Add the right combinators in the mirage eDSL
And that should be it. 

Best,
Thomas

[1] https://github.com/mirage/irmin/issues/99
[2] https://github.com/mirage/ocaml-git/blob/master/lib/sync.mli#L67

> On 21 Nov 2014, at 16:18, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
> 
> Hi,
> 
> I've spent the last week working on the new version of Irmin. This is still 
> not finished but it is making good progress. A preliminary ocamldoc page is 
> available online[1]. It is not complete and polished yet, but most parts are 
> documented.
> 
> The main changes are:
> 
> - Improve compile time and portability: no more core_kernel and camlp4, now 
> use nocrypto and mirage-tc
> 
> - No more -pack. All the internal modules are prefixed by Ir_ and are not 
> available to the users (the cmi are not installed) and the public module 
> (with the public documentation) is Irmin
> 
> - All the operations in a store now take a **mandatory** `origin` argument, 
> even the read operation [2] (before, only the merge operations where 
> requiring it). The origin contains the process/user name, the date, some 
> (maybe structured) message, ... (this can be configured by the user). That 
> origin value comes from the high-level call ("I want to add that value to my 
> Irmin store") and are passed through all the low-level calls ("I need to 
> create a new block in the block store, I need to update the head tag to point 
> to the new head, etc) so you can very easily track every read and write 
> operations in your store. My hope is that we can use that origin value to 
> control the store encryption: you can store some keys in the origin and use 
> them to encrypt on write and decrypt on read. Need more though on that, but I 
> think we have everything needed now.
> 
> - The high-level "branch-consistant" signature is simpler [3]. The internals 
> of the contents, node, commit and tag stores are not exposed anymore.
> 
> - The synchronisation functions are much more efficient and looks quite 
> similar to what Git does, although we don't compress the data (yet) (but the 
> slice[4] type is abstract, so we could do that later if needed, this involves 
> tweaking [5]).
> 
> What needs to be done: porting the backends, the http server and checking 
> that everything still works fine. Hopefully this will be done early next week.
> 
> Best,
> Thomas
> 
> [1]: http://samoht.github.io/irmin/
> [2]: http://samoht.github.io/irmin/Irmin.Store.RO.html
> [3]: http://samoht.github.io/irmin/Irmin.Store.BC.html
> [4]: http://samoht.github.io/irmin/Irmin.Store.BC.html#TYPEslice
> [5]: 
> https://github.com/samoht/irmin/blob/ab20999e43a9aa4a3bdb1733a55288080ff2c565/lib/ir_bc.ml#L488
> 
>> On 1 Oct 2014, at 21:38, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
>> 
>> I'm very late on all of these points (although I almost have 1/ done, needs 
>> more polishing) but I've generated some docs which might be useful to people 
>> starting using the library:
>> 
>> http://samoht.github.io/irmin/
>> 
>> Thomas
>> 
>> On 9 Sep 2014, at 16:57, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
>> 
>>> Hi all,
>>> 
>>> Lots of people starting asked me about the status of Irmin. As you might 
>>> have notice, I'vent had much time to spend on Irmin last month, but I plan 
>>> to fix that. My current priorities are:
>>> 
>>> 1/ make Irmin compiles on the Xen backend of Mirage.  
>>> https://github.com/mirage/irmin/issues/81
>>> This means dropping the dependency to core_kernel (for now on). The 
>>> external API will not change.
>>> 
>>> 2/ improve the high-level JSON API. This is related to 
>>> https://github.com/mirage/irmin/issues/80 
>>> Currently, we only have a JSON API for the low-level stores, and the very 
>>> partial implementation of the higher-level calls. Needs to complete and 
>>> document that.
>>> 
>>> 3/ implement a pure-ocaml Git server to be able to at least `git pull` from 
>>> it. https://github.com/mirage/ocaml-git/issues/15 
>>> Together with 1/ this will make possible to query the state of a running 
>>> Mirage unikernel with a simple `git pull <vm-address>`
>>> 
>>> 4/ implement a simple distributed log server 
>>> .https://github.com/mirage/irmin/issues/82 
>>> Could use Benjamin's rope implementation, that I first need to release 
>>> properly.
>>> 
>>> 5/ try to come up with a solution to the "unlimited" memory/storage usage 
>>> issues https://github.com/mirage/irmin/issues/83
>>> This is the most uncertain part of the short-term roadmap. I'll need to 
>>> check again what are the limitation of Git shallow copies and see how we 
>>> can use them to limit the history size to remember.
>>> 
>>> There are also few open embarrassing bugs that I want to fix as well 
>>> (regarding fd leaks ...). 
>>> 
>>> If anyone is interested to help me on any of these topics, please feel free 
>>> to comment on the related issues on Github -- I'll gladly share the 
>>> workload. Also, feel free to reply to that email if you think there is any 
>>> important Imrin features that you think are missing now.
>>> 
>>> Best,
>>> Thomas
>> 
> 


_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel