Re: [MirageOS-devel] Current persistent storage situation

[Sven Anderson <sven@xxxxxxxxxxx>]

Am 22.11.2017 um 18:21 schrieb Tom Ridge <tom.j.ridge@xxxxxxxxxxxxxx
<mailto:tom.j.ridge@xxxxxxxxxxxxxx>>:

> But if you are happy to have the storage remote from the mirage machine

Actually, since we need multi-core support and in general multi-node
support for the application, it would be of advantage if the design is
based on a remote and concurrency-safe solution from the beginning. All
local storage solutions would bring the problem of simultaneous storage
device access I guess, because Lwt has no multicore support yet, and I
would have to run an instance on each core, IIUC. 

So: I’m happy if it is remote, if too much performance impact can be
avoided with caching and similar things.

> , and running an arbitrary OS, then you have many options to choose
> from eg just send key-value pair via tcp to remote server running an
> ocaml listener, then do what you want from there.

Could you give an example how to send an ocaml KV pair over TCP? Or are
you talking about implementing a custom protocol with serialization and
everything?

> The consistency/caching aspect of your query really depends on what
> properties you want (if you are prepared to cache, then you are
> prepared to receive possibly-stale data presumably... so you have to
> be a bit more precise about what properties you want).

The application stores private cryptographic keys, that means the keys
are basically write-once-read-often objects. So caching with a kind of
„dirty“ notification could be very efficient. Using stale data is
probably not super critical. A maximum time of propagation would be
sufficient I guess.


Thanks a lot for the input!


Cheers,

Sven


>
> On 21 November 2017 at 17:42, Tom Ridge <tom.j.ridge@xxxxxxxxxxxxxx
> <mailto:tom.j.ridge@xxxxxxxxxxxxxx>> wrote:
>
>     https://github.com/tomjridge/tjr_btree
>     <https://github.com/tomjridge/tjr_btree> ? It is not hooked into
>     mirage block device, so that might require some hacking.
>
>     On 14 November 2017 at 17:14, Gabriel de Perthuis
>     <g2p.code@xxxxxxxxx <mailto:g2p.code@xxxxxxxxx>> wrote:
>
>         Hello,
>
>         I'm the developer of Wodan.  Wodan itself is restricted to
>         fixed-sized
>         keys and bounded-size values which might be too constraining
>         for you.  I
>         am currently working on an Irmin layer on top of it.  It's
>         currently in
>         a private branch until all tests pass; the chunking approach
>         seems to
>         break the existing tests.
>
>         Le 14/11/2017 à 15:39, Sven Anderson a écrit :
>         > Hi,
>         >
>         > TL;DR: what current persistent storage options are out there
>         _now_?
>         >
>         > for our project (secure private key storage,
>         > https://github.com/keyfender) we need a persistent storage
>         for our
>         > unikernel application, persistent in the meaning of
>         surviving reboots of
>         > the unikernel. It doesn't have to be necessarily "local" in
>         a sense of
>         > writing it to a storage device that the mirage unikernel has
>         direct
>         > access to, it could also be another VM, that runs a network
>         daemon that
>         > the unikernel is connecting to. The latter would have the
>         advantage of
>         > giving us concurrency for multi-core or multi-node for free
>         which we
>         > plan to support anyway.
>         >
>         > The data we need to store has a simple KV structure, for
>         configuration
>         > and payload data, the keys are strings, the values are
>         arbitrary ocaml
>         > objects, they could be serialized into json though, if
>         necessary. The
>         > data amount is rather small, write rarely, read often (but
>         should be
>         > cached locally, because of the pretty constant nature of the
>         data).
>         > Consistency between different threads/nodes would be more
>         important than
>         > IO performance.
>         >
>         > Now the question is: what options do we have, if any?
>         >
>         > Irmin seems to be a pretty good fit, giving us a distributed
>         storage
>         > structure, and IIUC Irmin within MirageOS doesn't store
>         locally but
>         > could use a Git server or Non-Mirage-Irmin-Server on another
>         node as
>         > persistent storage, is this correct?
>         >
>         > What other options are out there? the Wodan project
>         > (https://github.com/g2p/wodan) is ready for "production"?
>         >
>         > Are there backends for standard SQL databases, like
>         MariaDB/PostgreSQL?
>         >
>         > Thanks in advance for any pointers and hints!
>         >
>         >
>         > Cheers
>         >
>         > Sven
>         >
>         > _______________________________________________
>         > MirageOS-devel mailing list
>         > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
>         <mailto:MirageOS-devel@xxxxxxxxxxxxxxxxxxxx>
>         >
>         https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
>         <https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel>
>         >
>
>
>         _______________________________________________
>         MirageOS-devel mailing list
>         MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
>         <mailto:MirageOS-devel@xxxxxxxxxxxxxxxxxxxx>
>         https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
>         <https://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel>
>
>
>

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/mirageos-devel