Re: I am confused by Mirage_kv.S.batch

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

Looking at littlefs, you could probably get close to the transactional 
semantics by writing all the file data out first in the batch operation, and 
then hooking them into the main kv store by updating the metadata last. In the 
case of a crash, the files written out would be orphaned and so a separate 
"fsck" operation would be required to cleanup, but at least metadata would 
remain consistent.

I say "pretty close" since I think littlefs doesn't provide atomic directory 
renames, only atomic file renames, so if you have a tmp directory full of files 
there would be a small race condition as each file is moved into the main 
metadata store.

Now I'm really curious about what hardware you're running littlefs on, Mindy :-)

cheers,
Anil

> On 3 Jan 2022, at 16:41, Thomas Gazagnaire <thomas@xxxxxxxxxxxxxx> wrote:
> 
> Hi Mindy :-)
> 
> The semantics intend to match the “commit” operation of Irmin, e.g. a commit 
> will ever read what is in the current checkout (and/or what have been written 
> in that commit since the beginning of the transaction). The important bits is 
> that it won’t be able to read writes done by concurrent writers.
> 
> It’s obviously not super easy to implement if you do not have access to a 
> transactional file-system. In that case, I won’t worry too much about it; the 
> only important thing is durability, e.g. the data should be written (and 
> flushed) to disk once the operation has completed. We could probably edit 
> that docstring to remove the sentence about “reading inside a batch”: I don’t 
> think any user of Mirage_kv is relying on this behaviour at the moment.
> 
> Best,
> Thomas
> 
>> On 2 Jan 2022, at 23:13, Mindy <mindy@xxxxxxxxxxxxxxxxxxx> wrote:
>> 
>> Hi all,
>> 
>> I've been working on a block-backed Mirage_kv.S implementation inspired by 
>> (and largely interoperable with) littlefs[1] . I'm approaching readiness for 
>> release, but there remains one large unimplemented piece of Mirage_kv.S : 
>> `batch`. The relevant excerpt of mirage_kv.mli is reproduced below:
>> 
>> ```
>> 
>>  val batch: t -> ?retries:int -> (t -> 'a Lwt.t) -> 'a Lwt.t
>>  (** [batch t f] run [f] in batch. Ensure the durability of
>>     operations.
>> 
>>      Since a batch is applied at once, the readings inside a batch
>>     will return the state before the entire batch. Concurrent
>>     operations will not affect other ones executed during the batch.
>> 
>>      Batch applications can fail to apply if other operations are
>>     happening concurrently. In case of failure, [f] will run again
>>     with the most recent version of [t]. The result is
>>     [Error `Too_many_retries] if [f] is run for more then [retries] attemps
>>     (default is [13]). *)
>> ```
>> 
>> I confess to being uncertain how to implement the "readings inside a batch" 
>> portion of this. Given an arbitrary function `t -> 'a Lwt.t`, how is the 
>> implementer expected to differentiate between operations within such a 
>> function, possibly implemented as a series of sequential binds?
>> 
>> Currently my implementation simply runs the given function against the 
>> filesystem, but this is clearly semantically wrong given the explanation in 
>> the docstring.
>> 
>> I see `wodan` implements this by differentiating between a `t` used for read 
>> operations and one used for write operations[2] , but I lack the 
>> understanding of `wodan` necessary to understand how this can work for a `t 
>> -> 'a Lwt.t` that mixes read and write operations, which I think the 
>> docstring implies to be an expected use of `batch`.
>> 
>> Thanks for your help,
>> 
>> Mindy
>> 
>> [1] https://github.com/littlefs-project/littlefs
>> 
>> [2] 
>> https://github.com/mirage/wodan/blob/main/src/wodan-irmin/wodan_irmin.ml#L345
>> 
>> 
> 
>