From mirageos-devel-bounces@lists.xenproject.org Sun Jan 02 22:13:29 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Sun, 02 Jan 2022 22:13:29 +0000 Received: from list by lists.xenproject.org with outflank-mailman.252810.433788 (Exim 4.92) (envelope-from ) id 1n496M-0006HD-TH; Sun, 02 Jan 2022 22:13:18 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 252810.433788; Sun, 02 Jan 2022 22:13:18 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n496M-0006H6-Q3; Sun, 02 Jan 2022 22:13:18 +0000 Received: by outflank-mailman (input) for mailman id 252810; Sun, 02 Jan 2022 22:13:17 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n496L-0006H0-Sm for mirageos-devel@lists.xenproject.org; Sun, 02 Jan 2022 22:13:17 +0000 Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 2e6a9b42-6c19-11ec-81c0-a30af7de8005; Sun, 02 Jan 2022 23:13:16 +0100 (CET) Received: (Authenticated sender: guybrush@somerandomidiot.com) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 2D5E720003 for ; Sun, 2 Jan 2022 22:13:13 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 2e6a9b42-6c19-11ec-81c0-a30af7de8005 Message-ID: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> Date: Sun, 2 Jan 2022 16:13:11 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Content-Language: en-US From: Mindy Subject: I am confused by Mirage_kv.S.batch To: mirageos-devel@lists.xenproject.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi all, I've been working on a block-backed Mirage_kv.S implementation inspired by (and largely interoperable with) littlefs[1] . I'm approaching readiness for release, but there remains one large unimplemented piece of Mirage_kv.S : `batch`. The relevant excerpt of mirage_kv.mli is reproduced below: ```   val batch: t -> ?retries:int -> (t -> 'a Lwt.t) -> 'a Lwt.t   (** [batch t f] run [f] in batch. Ensure the durability of      operations.       Since a batch is applied at once, the readings inside a batch      will return the state before the entire batch. Concurrent      operations will not affect other ones executed during the batch.       Batch applications can fail to apply if other operations are      happening concurrently. In case of failure, [f] will run again      with the most recent version of [t]. The result is      [Error `Too_many_retries] if [f] is run for more then [retries] attemps      (default is [13]). *) ``` I confess to being uncertain how to implement the "readings inside a batch" portion of this. Given an arbitrary function `t -> 'a Lwt.t`, how is the implementer expected to differentiate between operations within such a function, possibly implemented as a series of sequential binds? Currently my implementation simply runs the given function against the filesystem, but this is clearly semantically wrong given the explanation in the docstring. I see `wodan` implements this by differentiating between a `t` used for read operations and one used for write operations[2] , but I lack the understanding of `wodan` necessary to understand how this can work for a `t -> 'a Lwt.t` that mixes read and write operations, which I think the docstring implies to be an expected use of `batch`. Thanks for your help, Mindy [1] https://github.com/littlefs-project/littlefs [2] https://github.com/mirage/wodan/blob/main/src/wodan-irmin/wodan_irmin.ml#L345 From mirageos-devel-bounces@lists.xenproject.org Sun Jan 02 22:18:00 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Sun, 02 Jan 2022 22:18:00 +0000 Received: from list by lists.xenproject.org with outflank-mailman.252816.433793 (Exim 4.92) (envelope-from ) id 1n49At-0006Wx-A2; Sun, 02 Jan 2022 22:17:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 252816.433793; Sun, 02 Jan 2022 22:17:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n49At-0006Wq-71; Sun, 02 Jan 2022 22:17:59 +0000 Received: by outflank-mailman (input) for mailman id 252816; Sun, 02 Jan 2022 22:17:58 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n49As-0006Wk-1V for mirageos-devel@lists.xenproject.org; Sun, 02 Jan 2022 22:17:58 +0000 Received: from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net [217.70.183.194]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id d5a38432-6c19-11ec-9ce5-af14b9085ebd; Sun, 02 Jan 2022 23:17:56 +0100 (CET) Received: (Authenticated sender: guybrush@somerandomidiot.com) by relay2-d.mail.gandi.net (Postfix) with ESMTPSA id 72EE740003 for ; Sun, 2 Jan 2022 22:17:54 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: d5a38432-6c19-11ec-9ce5-af14b9085ebd Message-ID: <8342501f-7bfc-a2ba-39bb-31d8abdf3661@somerandomidiot.com> Date: Sun, 2 Jan 2022 16:17:51 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Subject: Re: I am confused by Mirage_kv.S.batch Content-Language: en-US To: mirageos-devel@lists.xenproject.org References: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> From: Mindy In-Reply-To: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Er, s/Mirage_kv\.S/Mirage_kv.RW/g below. :sweat_smile: On 1/2/22 16:13, Mindy wrote: > Hi all, > > I've been working on a block-backed Mirage_kv.S implementation > inspired by (and largely interoperable with) littlefs[1] . I'm > approaching readiness for release, but there remains one large > unimplemented piece of Mirage_kv.S : `batch`. The relevant excerpt of > mirage_kv.mli is reproduced below: > > ``` > >   val batch: t -> ?retries:int -> (t -> 'a Lwt.t) -> 'a Lwt.t >   (** [batch t f] run [f] in batch. Ensure the durability of >      operations. > >       Since a batch is applied at once, the readings inside a batch >      will return the state before the entire batch. Concurrent >      operations will not affect other ones executed during the batch. > >       Batch applications can fail to apply if other operations are >      happening concurrently. In case of failure, [f] will run again >      with the most recent version of [t]. The result is >      [Error `Too_many_retries] if [f] is run for more then [retries] > attemps >      (default is [13]). *) > ``` > > I confess to being uncertain how to implement the "readings inside a > batch" portion of this. Given an arbitrary function `t -> 'a Lwt.t`, > how is the implementer expected to differentiate between operations > within such a function, possibly implemented as a series of sequential > binds? > > Currently my implementation simply runs the given function against the > filesystem, but this is clearly semantically wrong given the > explanation in the docstring. > > I see `wodan` implements this by differentiating between a `t` used > for read operations and one used for write operations[2] , but I lack > the understanding of `wodan` necessary to understand how this can work > for a `t -> 'a Lwt.t` that mixes read and write operations, which I > think the docstring implies to be an expected use of `batch`. > > Thanks for your help, > > Mindy > > [1] https://github.com/littlefs-project/littlefs > > [2] > https://github.com/mirage/wodan/blob/main/src/wodan-irmin/wodan_irmin.ml#L345 > > From mirageos-devel-bounces@lists.xenproject.org Mon Jan 03 14:41:23 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 03 Jan 2022 14:41:23 +0000 Received: from list by lists.xenproject.org with outflank-mailman.252965.433919 (Exim 4.92) (envelope-from ) id 1n4OWP-0005QF-UR; Mon, 03 Jan 2022 14:41:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 252965.433919; Mon, 03 Jan 2022 14:41:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4OWP-0005Q8-RI; Mon, 03 Jan 2022 14:41:13 +0000 Received: by outflank-mailman (input) for mailman id 252965; Mon, 03 Jan 2022 14:41:11 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4OWN-0005Q2-Qr for mirageos-devel@lists.xenproject.org; Mon, 03 Jan 2022 14:41:11 +0000 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 3045da9c-6ca3-11ec-9ce5-af14b9085ebd; Mon, 03 Jan 2022 15:41:09 +0100 (CET) Received: (Authenticated sender: thomas@gazagnaire.org) by relay12.mail.gandi.net (Postfix) with ESMTPSA id CBCD5200005; Mon, 3 Jan 2022 14:41:07 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 3045da9c-6ca3-11ec-9ce5-af14b9085ebd Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: I am confused by Mirage_kv.S.batch From: Thomas Gazagnaire In-Reply-To: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> Date: Mon, 3 Jan 2022 15:41:07 +0100 Cc: mirageos-devel@lists.xenproject.org Content-Transfer-Encoding: quoted-printable Message-Id: <0ED82698-D7AA-4949-88DE-1791748085C9@gazagnaire.org> References: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> To: Mindy X-Mailer: Apple Mail (2.3693.20.0.1.32) Hi Mindy :-) The semantics intend to match the =E2=80=9Ccommit=E2=80=9D operation of = Irmin, e.g. a commit will ever read what is in the current checkout = (and/or what have been written in that commit since the beginning of the = transaction). The important bits is that it won=E2=80=99t be able to = read writes done by concurrent writers. It=E2=80=99s obviously not super easy to implement if you do not have = access to a transactional file-system. In that case, I won=E2=80=99t = worry too much about it; the only important thing is durability, e.g. = the data should be written (and flushed) to disk once the operation has = completed. We could probably edit that docstring to remove the sentence = about =E2=80=9Creading inside a batch=E2=80=9D: I don=E2=80=99t think = any user of Mirage_kv is relying on this behaviour at the moment. Best, Thomas > On 2 Jan 2022, at 23:13, Mindy wrote: >=20 > Hi all, >=20 > I've been working on a block-backed Mirage_kv.S implementation = inspired by (and largely interoperable with) littlefs[1] . I'm = approaching readiness for release, but there remains one large = unimplemented piece of Mirage_kv.S : `batch`. The relevant excerpt of = mirage_kv.mli is reproduced below: >=20 > ``` >=20 > val batch: t -> ?retries:int -> (t -> 'a Lwt.t) -> 'a Lwt.t > (** [batch t f] run [f] in batch. Ensure the durability of > operations. >=20 > Since a batch is applied at once, the readings inside a batch > will return the state before the entire batch. Concurrent > operations will not affect other ones executed during the batch. >=20 > Batch applications can fail to apply if other operations are > happening concurrently. In case of failure, [f] will run again > with the most recent version of [t]. The result is > [Error `Too_many_retries] if [f] is run for more then [retries] = attemps > (default is [13]). *) > ``` >=20 > I confess to being uncertain how to implement the "readings inside a = batch" portion of this. Given an arbitrary function `t -> 'a Lwt.t`, how = is the implementer expected to differentiate between operations within = such a function, possibly implemented as a series of sequential binds? >=20 > Currently my implementation simply runs the given function against the = filesystem, but this is clearly semantically wrong given the explanation = in the docstring. >=20 > I see `wodan` implements this by differentiating between a `t` used = for read operations and one used for write operations[2] , but I lack = the understanding of `wodan` necessary to understand how this can work = for a `t -> 'a Lwt.t` that mixes read and write operations, which I = think the docstring implies to be an expected use of `batch`. >=20 > Thanks for your help, >=20 > Mindy >=20 > [1] https://github.com/littlefs-project/littlefs >=20 > [2] = https://github.com/mirage/wodan/blob/main/src/wodan-irmin/wodan_irmin.ml#L= 345 >=20 >=20 From mirageos-devel-bounces@lists.xenproject.org Mon Jan 03 15:35:53 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 03 Jan 2022 15:35:53 +0000 Received: from list by lists.xenproject.org with outflank-mailman.252973.433924 (Exim 4.92) (envelope-from ) id 1n4PND-0001fN-UX; Mon, 03 Jan 2022 15:35:47 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 252973.433924; Mon, 03 Jan 2022 15:35:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4PND-0001fG-RR; Mon, 03 Jan 2022 15:35:47 +0000 Received: by outflank-mailman (input) for mailman id 252973; Mon, 03 Jan 2022 15:35:46 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4PNC-0001fA-Bi for mirageos-devel@lists.xenproject.org; Mon, 03 Jan 2022 15:35:46 +0000 Received: from honk.recoil.org (honk.recoil.org [147.75.102.218]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id cf84ed31-6caa-11ec-81c0-a30af7de8005; Mon, 03 Jan 2022 16:35:42 +0100 (CET) Received: from smtpclient.apple (185-11-210-92.s1networks.fi [185.11.210.92]) by honk.recoil.org (Postfix) with ESMTPSA id 8BA9E3C008C; Mon, 3 Jan 2022 15:35:41 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: cf84ed31-6caa-11ec-81c0-a30af7de8005 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=recoil.org; s=selector1; t=1641224141; bh=XSt7EFQohJHaA2Xn3OOCYkxQJ5igq+VSQ/KDSR8iE6I=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From; b=g0GN8FbN13y4W49hwNr1TBgdXrpfztU2ylh+r+Ee15DbNjCG7K0T7BcO3sAqXD9aY NBjo2MqSbjDg2w5yBhOS1+bpd4g8PftVQS2QJMeYcRa/pKBIlWjH8OxxL9i3oFQnaT jr5NeKGRAaVpj7G8PhmMiwbtPrHLN0EBDxUosakQ= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: I am confused by Mirage_kv.S.batch From: Anil Madhavapeddy In-Reply-To: <0ED82698-D7AA-4949-88DE-1791748085C9@gazagnaire.org> Date: Mon, 3 Jan 2022 17:34:40 +0200 Cc: Mindy Preston , mirageos-devel@lists.xenproject.org Content-Transfer-Encoding: quoted-printable Message-Id: <65052059-5FD8-4492-A2BA-EC632D6D8A9F@recoil.org> References: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> <0ED82698-D7AA-4949-88DE-1791748085C9@gazagnaire.org> To: Thomas Gazagnaire X-Mailer: Apple Mail (2.3693.20.0.1.32) Looking at littlefs, you could probably get close to the transactional = semantics by writing all the file data out first in the batch operation, = and then hooking them into the main kv store by updating the metadata = last. In the case of a crash, the files written out would be orphaned = and so a separate "fsck" operation would be required to cleanup, but at = least metadata would remain consistent. I say "pretty close" since I think littlefs doesn't provide atomic = directory renames, only atomic file renames, so if you have a tmp = directory full of files there would be a small race condition as each = file is moved into the main metadata store. Now I'm really curious about what hardware you're running littlefs on, = Mindy :-) cheers, Anil > On 3 Jan 2022, at 16:41, Thomas Gazagnaire = wrote: >=20 > Hi Mindy :-) >=20 > The semantics intend to match the =E2=80=9Ccommit=E2=80=9D operation = of Irmin, e.g. a commit will ever read what is in the current checkout = (and/or what have been written in that commit since the beginning of the = transaction). The important bits is that it won=E2=80=99t be able to = read writes done by concurrent writers. >=20 > It=E2=80=99s obviously not super easy to implement if you do not have = access to a transactional file-system. In that case, I won=E2=80=99t = worry too much about it; the only important thing is durability, e.g. = the data should be written (and flushed) to disk once the operation has = completed. We could probably edit that docstring to remove the sentence = about =E2=80=9Creading inside a batch=E2=80=9D: I don=E2=80=99t think = any user of Mirage_kv is relying on this behaviour at the moment. >=20 > Best, > Thomas >=20 >> On 2 Jan 2022, at 23:13, Mindy wrote: >>=20 >> Hi all, >>=20 >> I've been working on a block-backed Mirage_kv.S implementation = inspired by (and largely interoperable with) littlefs[1] . I'm = approaching readiness for release, but there remains one large = unimplemented piece of Mirage_kv.S : `batch`. The relevant excerpt of = mirage_kv.mli is reproduced below: >>=20 >> ``` >>=20 >> val batch: t -> ?retries:int -> (t -> 'a Lwt.t) -> 'a Lwt.t >> (** [batch t f] run [f] in batch. Ensure the durability of >> operations. >>=20 >> Since a batch is applied at once, the readings inside a batch >> will return the state before the entire batch. Concurrent >> operations will not affect other ones executed during the batch. >>=20 >> Batch applications can fail to apply if other operations are >> happening concurrently. In case of failure, [f] will run again >> with the most recent version of [t]. The result is >> [Error `Too_many_retries] if [f] is run for more then [retries] = attemps >> (default is [13]). *) >> ``` >>=20 >> I confess to being uncertain how to implement the "readings inside a = batch" portion of this. Given an arbitrary function `t -> 'a Lwt.t`, how = is the implementer expected to differentiate between operations within = such a function, possibly implemented as a series of sequential binds? >>=20 >> Currently my implementation simply runs the given function against = the filesystem, but this is clearly semantically wrong given the = explanation in the docstring. >>=20 >> I see `wodan` implements this by differentiating between a `t` used = for read operations and one used for write operations[2] , but I lack = the understanding of `wodan` necessary to understand how this can work = for a `t -> 'a Lwt.t` that mixes read and write operations, which I = think the docstring implies to be an expected use of `batch`. >>=20 >> Thanks for your help, >>=20 >> Mindy >>=20 >> [1] https://github.com/littlefs-project/littlefs >>=20 >> [2] = https://github.com/mirage/wodan/blob/main/src/wodan-irmin/wodan_irmin.ml#L= 345 >>=20 >>=20 >=20 >=20 From mirageos-devel-bounces@lists.xenproject.org Mon Jan 03 22:41:54 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 03 Jan 2022 22:41:54 +0000 Received: from list by lists.xenproject.org with outflank-mailman.252982.433929 (Exim 4.92) (envelope-from ) id 1n4W1O-0006QF-HV; Mon, 03 Jan 2022 22:41:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 252982.433929; Mon, 03 Jan 2022 22:41:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4W1O-0006Q8-Dy; Mon, 03 Jan 2022 22:41:42 +0000 Received: by outflank-mailman (input) for mailman id 252982; Mon, 03 Jan 2022 22:41:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4W1N-0006Q2-Uy for mirageos-devel@lists.xenproject.org; Mon, 03 Jan 2022 22:41:41 +0000 Received: from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net [217.70.183.194]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 504d639d-6ce6-11ec-81c0-a30af7de8005; Mon, 03 Jan 2022 23:41:40 +0100 (CET) Received: (Authenticated sender: guybrush@somerandomidiot.com) by relay2-d.mail.gandi.net (Postfix) with ESMTPSA id F0D3940004 for ; Mon, 3 Jan 2022 22:41:37 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 504d639d-6ce6-11ec-81c0-a30af7de8005 Message-ID: <54e7a82f-b4be-98db-2689-77163f52c12e@somerandomidiot.com> Date: Mon, 3 Jan 2022 16:41:34 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Subject: Re: I am confused by Mirage_kv.S.batch Content-Language: en-US To: mirageos-devel@lists.xenproject.org References: <8a72ad55-0c04-eb96-15c1-4940b60e9482@somerandomidiot.com> <0ED82698-D7AA-4949-88DE-1791748085C9@gazagnaire.org> From: Mindy In-Reply-To: <0ED82698-D7AA-4949-88DE-1791748085C9@gazagnaire.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 1/3/22 08:41, Thomas Gazagnaire wrote: > Hi Mindy :-) > > The semantics intend to match the “commit” operation of Irmin, e.g. a commit will ever read what is in the current checkout (and/or what have been written in that commit since the beginning of the transaction). The important bits is that it won’t be able to read writes done by concurrent writers. Good news - this implementation trivially fulfills this requirement by not supporting concurrent writes ;) > It’s obviously not super easy to implement if you do not have access to a transactional file-system. In that case, I won’t worry too much about it; the only important thing is durability, e.g. the data should be written (and flushed) to disk once the operation has completed. We could probably edit that docstring to remove the sentence about “reading inside a batch”: I don’t think any user of Mirage_kv is relying on this behaviour at the moment. Thanks for the reply!  I got a few suggestions for how I might implement this, but to be honest they don't seem worth the increased complexity to me.  I think any users who want to use this functionality should definitely be using `irmin` on top of the storage solution of their choice, rather than my Mirage_kv.RW implementation. If the docstring is changed as you describe, I'll implement `batch` by just executing the given function; if not, I'll have all batch operations error out immediately, so nobody gets the wrong idea. Thanks! -Mindy From mirageos-devel-bounces@lists.xenproject.org Tue Jan 04 21:14:57 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Tue, 04 Jan 2022 21:14:57 +0000 Received: from list by lists.xenproject.org with outflank-mailman.253396.434536 (Exim 4.92) (envelope-from ) id 1n4r8o-0004GP-4s; Tue, 04 Jan 2022 21:14:46 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 253396.434536; Tue, 04 Jan 2022 21:14:46 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4r8o-0004GI-1t; Tue, 04 Jan 2022 21:14:46 +0000 Received: by outflank-mailman (input) for mailman id 253396; Tue, 04 Jan 2022 21:14:45 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n4r8n-0004GC-EQ for mirageos-devel@lists.xenproject.org; Tue, 04 Jan 2022 21:14:45 +0000 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [2a00:1450:4864:20::52f]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 565ade69-6da3-11ec-81c0-a30af7de8005; Tue, 04 Jan 2022 22:14:44 +0100 (CET) Received: by mail-ed1-x52f.google.com with SMTP id b13so153436038edd.8 for ; Tue, 04 Jan 2022 13:14:43 -0800 (PST) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 565ade69-6da3-11ec-81c0-a30af7de8005 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=RrKtU3hQ/z3ifYzrJCdTspZFPdTCuPn84Lt9hYECTW8=; b=UIZM4xxYpq6KjMcD13bJ1DO7tki2kApgiRvRDyznW2OsbjJ5sUEtmMwp2DTHAXWo18 FD+o3pN9xwCMjMU2xlrf4UOhveak8KHHAiVbfGp3dYBUZMefs/prY0uCJ8oxqUIwRKuO JLAxUOW/KYAQMjywJ+zSQ8AifUIxhlqrI87kRefpoNffdN9cUVRmmUc9B/xPAB7UraGP ZiplalQnsA0xda+CQGcDQHB811vmwKLHO9wnfZeBg/667rTe/mQZ5C8HQcjShitD3+we QV4U37dzhFZPQj2fTZSqu7i/vAJ5G591cIMRnI3zMHj39/rPq4kB5SYaShUhGgtInasq xzcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=RrKtU3hQ/z3ifYzrJCdTspZFPdTCuPn84Lt9hYECTW8=; b=vYFuuR6/62kAMZ968YLFOeWGT3gOTJl2ljO/bp6nkmOvuZ0OtzP5sZY0br1mv1XGDx 0IRie2xxzvZi6rXh5o1nk4oUxPZJxKQoInAHrjkYlBrLVtBWTFvtgWyfN9B3TX0KUcEu FavMjTpBpkJZwFZi96cnadBPQS7wSxm5ZFxRlpJ9GgJeBkbFwd8Kd+/xDkSIAVtWEe3Z 2XuYh9hydaJAVNnU4wqkK7HMohZpjaCyHIzXk3LKraKQlHAr797s4RXBCyzfHRQojZC0 Ibkb/YqSG5XW4x10ZA/9vCWft3I85D0j6G2GphPK9xqWXr7mJYgqteCbOqBxwwE8ylxX QfjQ== X-Gm-Message-State: AOAM531xyIZL4NTOL5R7C0hNR4BOGjZvFuFYKjyzithK/ii0IWoYQ1jH YDeFVykIq19NCxeAOLDHY1EuN6p7HUd35yEkTMjcP/9eyfA= X-Google-Smtp-Source: ABdhPJwkJxK1kKdqj3DNltjeuzyrjAhIB7osjpfW4vt5qxeOpOx5UzCmi1C9XjvPr9AvGuq/WKDQVZ4L3tIu3jf4shM= X-Received: by 2002:a05:6402:34d2:: with SMTP id w18mr50212719edc.152.1641330883297; Tue, 04 Jan 2022 13:14:43 -0800 (PST) MIME-Version: 1.0 From: Nobody Gone Date: Tue, 4 Jan 2022 16:14:32 -0500 Message-ID: Subject: Ideas for: Networking / RPC, Cross Language Support, and Provisioning and deployment. To: "mirageos-devel@lists.xenproject.org" Content-Type: multipart/alternative; boundary="0000000000004203cb05d4c82058" --0000000000004203cb05d4c82058 Content-Type: text/plain; charset="UTF-8" So I've been a big lurker on mirage since 2018. Overtime I've had a few ideas on how to improve the project and gain more adoption. *Cross Lang Support.* So I'm not an expert on compiler tools and transpilation, but a lot of people in the communities do, but I think we can use WASM and WASI to port from other languages and export access to our underlying runtimes, WE GET A FREE BROWSER TARGET, and we can use it as a sort of intermediate image we're able to covert into multiple others. My next suggestions are in an area I'm more familiar with deployment, rpc, and provisioning. *Scheduling / Deployments: * Making a side car service that can start and stop unikernels, list it's supported targets, running unikernels, and host different metrics. Make a secure image registry, an auth service, and a acl service built on TUF. Make it a target for the popular schedulers. Build a custom scheduler that is compatible with the omega and mesos models so backends are pluggable. *RPC / Inter Networking: * Build a Simple multiplexed pathrouted stream named* MS* using yamux and a multistream, so you have multiple virtual sessions on your connection and you can run different protocols on each session on the stream. A more Complex Session Protocol call *MSP* with Path, Headers, Payload similar to grpc, ubers TChannel, twitters mux, thrift, etc.. , that can carry metadata such as encoding used, tracing data, metrics, names of function called, that runs on top of *MS*. Add a simple RPC protocol named *RPC *that allows you to select your encoding and what functions you want to serve, lists them, and can call them built on *MSP*. A *METRICS SERVICE *for *MSP *that you can instrument. *A CLIENT SIDE LOAD BALANCING LIBRARY * *A SERVICE DISCOVERY LIBRARY * *LIBS/SERVICES FOR OCAML TO ATTRACT DYST SYS NERDS . * A service that can answer to Go's serf built on cyclon + vicinity and ringcast/poldercast, with the advantage of isolating virtual views that you can select as a means of forming a topology given by an algorithm of your choosing we should allow users to be able to contribute their own algorithms. This means you can like create service groups that use ring cast, and then apply lifegaurds failure detector on that constructed view. You can create a group that contains instances of a service and run a failure detector algorithm on it. This will allow us *WE NEED TO MAKE PIMP ASS IRMIN STORES* *A CASPAXOS *library Esposes, Get, Set, Seal, and Snapshot Operations. Takes user supplied function to change the state of named register. Takes user supplied view. Build on it with pluggable leader election, failure detectors, autoscaling policies. I have plenty of more ideas but this is too long already I'm already working on a few just hit me up if you think I'm on to somethig. --0000000000004203cb05d4c82058 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So I've been a big lurker on mirag= e since 2018.

Overtime I've had a few ideas on how t= o improve the project and gain more adoption.

Cross L= ang Support.

So I'm not an expert on compil= er tools and transpilation, but a lot of people in the communities do, but = I think we can use WASM and WASI to port from other languages and export ac= cess to our underlying runtimes, WE GET A FREE BROWSER TARGET, and=C2=A0 we= can use it as a sort of intermediate image we're able to covert into m= ultiple others.


My next suggestions are in an area I= 'm more familiar with deployment, rpc, and provisioning.


Scheduling / Deployments:
Making = a side car service that can start and stop unikernels, list it's suppor= ted targets,=C2=A0 running unikernels, and host different metrics.

<= /div>
Make a secure image registry, an auth service, and a acl service = built on TUF.
Make it a target for the popular schedulers.
<= br>

Build a custom scheduler that is compatible wi= th the omega and mesos models so backends are pluggable.



RPC / Inter Networking:
Build a S= imple multiplexed pathrouted stream named MS using yamux and a multi= stream, so you have multiple virtual sessions on your connection and you ca= n run different protocols on each session on the stream.

A more Comp= lex Session Protocol call MSP with Path, Headers, Payload=C2=A0 simi= lar to grpc, ubers TChannel, twitters mux, thrift, etc.. , that can carry m= etadata such as encoding used, tracing data, metrics, names of function cal= led, that runs on top of MS.


=
Add a simple RPC protocol named RPC=C2=A0 that allows you to se= lect your encoding and what functions you want to serve, lists them, and ca= n call them built on MSP.

A METRICS = SERVICE for MSP that you can instrument.


=

A=C2=A0 CLIENT SIDE LOAD BALANCING LIBRARY
A SERVICE DISCOVERY LIBRARY


LIBS/SERVICES FOR OCAML TO ATTRACT DYST SYS NERDS .=

A service that can answer to Go's serf bu= ilt on cyclon=C2=A0+=C2=A0 vicinity and ringcast/poldercast, with the advan= tage of isolating virtual views that you can select as a means of forming a= topology given by an algorithm of your choosing we should allow users to b= e able to contribute their own algorithms.

This means yo= u can like create service groups that use ring cast, and then apply lifegau= rds failure detector on that constructed view.=C2=A0

You= can create a group that contains instances of a service and run a failure = detector algorithm on it.

This wil= l allow us

WE NEED TO MAKE PIMP ASS IRMIN = STORES

A CASPAXOS library
=C2= =A0=C2=A0
=C2=A0=C2=A0 Esposes, Get, Set, Seal, and Snapshot= Operations.
=C2=A0=C2=A0 Takes user supplied function to ch= ange the state of named register.
=C2=A0=C2=A0 Takes user su= pplied view.
=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0= =C2=A0=C2=A0 Build on it with pluggable leader election,=C2=A0 failure dete= ctors, autoscaling policies.

I have plenty of= more ideas but this is too long already I'm already working on a few j= ust hit me up if you think I'm on to somethig.
--0000000000004203cb05d4c82058-- From mirageos-devel-bounces@lists.xenproject.org Mon Jan 10 20:33:35 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Mon, 10 Jan 2022 20:33:35 +0000 Received: from list by lists.xenproject.org with outflank-mailman.255548.437936 (Exim 4.92) (envelope-from ) id 1n71Lz-0000rz-EX; Mon, 10 Jan 2022 20:33:19 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 255548.437936; Mon, 10 Jan 2022 20:33:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n71Lz-0000rs-BN; Mon, 10 Jan 2022 20:33:19 +0000 Received: by outflank-mailman (input) for mailman id 255548; Mon, 10 Jan 2022 20:33:18 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1n71Ly-0000rm-JZ for mirageos-devel@lists.xenproject.org; Mon, 10 Jan 2022 20:33:18 +0000 Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 8a0391f0-7254-11ec-9ce5-af14b9085ebd; Mon, 10 Jan 2022 21:33:16 +0100 (CET) Received: (Authenticated sender: guybrush@somerandomidiot.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 19F8BC0006 for ; Mon, 10 Jan 2022 20:33:14 +0000 (UTC) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 8a0391f0-7254-11ec-9ce5-af14b9085ebd Message-ID: Date: Mon, 10 Jan 2022 14:33:12 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Content-Language: en-US To: mirageos-devel@lists.xenproject.org From: Mindy Subject: performance metrics for network stack? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi all, Is anyone aware of recent performance metrics for the network stack (specifically, solo5/ethernet/ipv4/tcp, for someone who's interested in running an http server)? I've been asked about this off-list and haven't kept my ear close enough to the ground to know offhand. Thanks, yomimono