Re: [MirageOS-devel] Irmin roadmap

[Jon Crowcroft <jac22@xxxxxxxxxxxx>]

wow max re factor!
cool!

In missive <B03E3D4A-6600-4168-BA62-A751961C5EE5@xxxxxxxxxxxxxx>, Thomas 
Gazagnaire
 typed:

 >>Hi all,
 >>
 >>after 3 months of refactoring (!), 11,5k lines added and 8,8k lines removed, 
 >>I am very happy to announce that I've merged my working patches into the 
 >>master branch of Irmin. I'll pass the remaining of the week fixing the last 
 >>quirks (mainly documentation related, but also around the command-line tool 
 >>usability) before announcing the 0.9.0 release.
 >>
 >>One of the main change is the new public API with a proper documentation: 
 >>http://samoht.github.io/irmin/ (although it needs more code examples[1])
 >>
 >>Summary of the changes:
 >>* Expose a cleaner public API
 >>* Expose a cleaner API for the Unix backends
 >>* Expose a high-level REST API over HTTP (#80)
 >>* Rename `Origin` into `Task` and use it pervasively through the API
 >>* Add `remove_rec` to remove directories (#74, #85)
 >>* More efficient synchronization protocol between Irmin stores (#11)
 >>* Improve the efficiency of the Git backend
 >>* Remove dependency to `core_kernel` (#22, #81)
 >>* Remove dependency to `cryptokit` and `ocaml-sha` and use `nocrypto` instead
 >>* Fix writing contents at the root of the store (#73)
 >>* Fix the Git backend to stop constantly overwrite `.git/HEAD` (#76)
 >>* Add a limit on concurrently open files (#93, #75)
 >>
 >>As usual, feedback is welcome.
 >>
 >>The initial goal for the 1.0.0 release was to integrate Mirage and Irmin 
 >>properly. That goal is very close now. What is missing:
 >>- Support the Git synchronisation protocol in Mirage. This means 
 >>implementing Git.Sync.IO [2]
 >>- Add the right combinators in the mirage eDSL
 >>And that should be it. 
 >>
 >>Best,
 >>Thomas
 >>
 >>[1] https://github.com/mirage/irmin/issues/99
 >>[2] https://github.com/mirage/ocaml-git/blob/master/lib/sync.mli#L67
 >>
 >>> On 21 Nov 2014, at 16:18, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
 >>> 
 >>> Hi,
 >>> 
 >>> I've spent the last week working on the new version of Irmin. This is 
 >>> still not finished but it is making good progress. A preliminary ocamldoc 
 >>> page is available online[1]. It is not complete and polished yet, but most 
 >>> parts are documented.
 >>> 
 >>> The main changes are:
 >>> 
 >>> - Improve compile time and portability: no more core_kernel and camlp4, 
 >>> now use nocrypto and mirage-tc
 >>> 
 >>> - No more -pack. All the internal modules are prefixed by Ir_ and are not 
 >>> available to the users (the cmi are not installed) and the public module 
 >>> (with the public documentation) is Irmin
 >>> 
 >>> - All the operations in a store now take a **mandatory** `origin` 
 >>> argument, even the read operation [2] (before, only the merge operations 
 >>> where requiring it). The origin contains the process/user name, the date, 
 >>> some (maybe structured) message, ... (this can be configured by the user). 
 >>> That origin value comes from the high-level call ("I want to add that 
 >>> value to my Irmin store") and are passed through all the low-level calls 
 >>> ("I need to create a new block in the block store, I need to update the 
 >>> head tag to point to the new head, etc) so you can very easily track every 
 >>> read and write operations in your store. My hope is that we can use that 
 >>> origin value to control the store encryption: you can store some keys in 
 >>> the origin and use them to encrypt on write and decrypt on read. Need more 
 >>> though on that, but I think we have everything needed now.
 >>> 
 >>> - The high-level "branch-consistant" signature is simpler [3]. The 
 >>> internals of the contents, node, commit and tag stores are not exposed 
 >>> anymore.
 >>> 
 >>> - The synchronisation functions are much more efficient and looks quite 
 >>> similar to what Git does, although we don't compress the data (yet) (but 
 >>> the slice[4] type is abstract, so we could do that later if needed, this 
 >>> involves tweaking [5]).
 >>> 
 >>> What needs to be done: porting the backends, the http server and checking 
 >>> that everything still works fine. Hopefully this will be done early next 
 >>> week.
 >>> 
 >>> Best,
 >>> Thomas
 >>> 
 >>> [1]: http://samoht.github.io/irmin/
 >>> [2]: http://samoht.github.io/irmin/Irmin.Store.RO.html
 >>> [3]: http://samoht.github.io/irmin/Irmin.Store.BC.html
 >>> [4]: http://samoht.github.io/irmin/Irmin.Store.BC.html#TYPEslice
 >>> [5]: 
 >>> https://github.com/samoht/irmin/blob/ab20999e43a9aa4a3bdb1733a55288080ff2c565/lib/ir_bc.ml#L488
 >>> 
 >>>> On 1 Oct 2014, at 21:38, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
 >>>> 
 >>>> I'm very late on all of these points (although I almost have 1/ done, 
 >>>> needs more polishing) but I've generated some docs which might be useful 
 >>>> to people starting using the library:
 >>>> 
 >>>> http://samoht.github.io/irmin/
 >>>> 
 >>>> Thomas
 >>>> 
 >>>> On 9 Sep 2014, at 16:57, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
 >>>> 
 >>>>> Hi all,
 >>>>> 
 >>>>> Lots of people starting asked me about the status of Irmin. As you might 
 >>>>> have notice, I'vent had much time to spend on Irmin last month, but I 
 >>>>> plan to fix that. My current priorities are:
 >>>>> 
 >>>>> 1/ make Irmin compiles on the Xen backend of Mirage.  
 >>>>> https://github.com/mirage/irmin/issues/81
 >>>>> This means dropping the dependency to core_kernel (for now on). The 
 >>>>> external API will not change.
 >>>>> 
 >>>>> 2/ improve the high-level JSON API. This is related to 
 >>>>> https://github.com/mirage/irmin/issues/80 
 >>>>> Currently, we only have a JSON API for the low-level stores, and the 
 >>>>> very partial implementation of the higher-level calls. Needs to complete 
 >>>>> and document that.
 >>>>> 
 >>>>> 3/ implement a pure-ocaml Git server to be able to at least `git pull` 
 >>>>> from it. https://github.com/mirage/ocaml-git/issues/15 
 >>>>> Together with 1/ this will make possible to query the state of a running 
 >>>>> Mirage unikernel with a simple `git pull <vm-address>`
 >>>>> 
 >>>>> 4/ implement a simple distributed log server 
 >>>>> .https://github.com/mirage/irmin/issues/82 
 >>>>> Could use Benjamin's rope implementation, that I first need to release 
 >>>>> properly.
 >>>>> 
 >>>>> 5/ try to come up with a solution to the "unlimited" memory/storage 
 >>>>> usage issues https://github.com/mirage/irmin/issues/83
 >>>>> This is the most uncertain part of the short-term roadmap. I'll need to 
 >>>>> check again what are the limitation of Git shallow copies and see how we 
 >>>>> can use them to limit the history size to remember.
 >>>>> 
 >>>>> There are also few open embarrassing bugs that I want to fix as well 
 >>>>> (regarding fd leaks ...). 
 >>>>> 
 >>>>> If anyone is interested to help me on any of these topics, please feel 
 >>>>> free to comment on the related issues on Github -- I'll gladly share the 
 >>>>> workload. Also, feel free to reply to that email if you think there is 
 >>>>> any important Imrin features that you think are missing now.
 >>>>> 
 >>>>> Best,
 >>>>> Thomas
 >>>> 
 >>> 
 >>
 >>
 >>_______________________________________________
 >>MirageOS-devel mailing list
 >>MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
 >>http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel

 cheers

   jon


_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel