Re: [MirageOS-devel] [ANNOUNCE] Mirage storage project

[Gabriel de Perthuis <g2p.code@xxxxxxxxx>]

Le 14/12/2016 à 14:42, Anil Madhavapeddy a écrit :
> On 14 Dec 2016, at 13:37, Gabriel de Perthuis <g2p.code@xxxxxxxxx> wrote:
>>
>> Hello,
> 
> Dear Gabriel,
> 
>> I'm announcing I intend to work on the Mirage storage stack.
>>
>> There is a clear need for something that sits on top of the BLOCK API
>> and provides higher-level capabilities.  I'm planning to write a
>> storage library that provides a key-value store (with fixed-size keys)
>> and an Irmin backend.  A more generic key-value store with variable
>> size keys could also be implemented on top.
>>
>> I'm envisioning a data layout that's suitable for SSDs, meaning it
>> will be log-structured, fit large erase blocks and never rewrite
>> in-place.  Unused blocks can be discarded in batches.
> 
> Welcome aboard, and glad to have you working on this very important
> component!  When you are ready to share, it would be good to expand
> on the scope of the initial version of the filesystem: for example how
> many concurrent readers and writers are supported and so on.  The
> first instance should be just fine with a single reader and writer, since our
> primary usecase is to serialise an Irmin Git store to disk (and Irmin itself
> will handle the concurrent writers and output a single stream of changes
> to the filesystem).
> 
> Others on the list may comment about other use-cases they may have
> as well for future iterations.

Here's the concurrency story.

The library has exclusive ownership of its storage backend, like most
filesystems.  As a concession to inspectability, and as discussed on
IRC, GC (discarding unused blocks) will be under user control so that
stale data may be read by another reader of the raw device.

The first iteration will probably expect access from a single thread,
but that may be relaxed to a single writer thread for many reader
threads.  Reader threads do write to the page cache so the relaxed
option does require keeping the cache consistent.

For a distributed store, the application needs to use Irmin for
reconciliation.  Etcd-style raft consensus would be an interesting
application.


_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel