From mirageos-devel-bounces@lists.xenproject.org Sun Dec 04 09:56:27 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Sun, 04 Dec 2022 09:56:27 +0000 Received: from list by lists.xenproject.org with outflank-mailman.452940.710711 (Exim 4.92) (envelope-from ) id 1p1ljJ-0003Xa-7k; Sun, 04 Dec 2022 09:56:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 452940.710711; Sun, 04 Dec 2022 09:56:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p1ljJ-0003XT-4J; Sun, 04 Dec 2022 09:56:13 +0000 Received: by outflank-mailman (input) for mailman id 452940; Sun, 04 Dec 2022 09:56:11 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p1ljH-0003XN-EI for mirageos-devel@lists.xenproject.org; Sun, 04 Dec 2022 09:56:11 +0000 Received: from mx.upb.ro (mail-sender251.upb.ro [141.85.13.251]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id decf2f08-73b9-11ed-91b6-6bf2151ebd3b; Sun, 04 Dec 2022 10:56:08 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mx.upb.ro (Postfix) with ESMTP id 35628B560415 for ; Sun, 4 Dec 2022 11:38:59 +0200 (EET) Received: from mx.upb.ro ([127.0.0.1]) by localhost (mx.upb.ro [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id B5W7UHYdEwrJ for ; Sun, 4 Dec 2022 11:38:56 +0200 (EET) Received: from localhost (localhost [127.0.0.1]) by mx.upb.ro (Postfix) with ESMTP id 8B3AAB5608A2 for ; Sun, 4 Dec 2022 11:38:56 +0200 (EET) Received: from mx.upb.ro ([127.0.0.1]) by localhost (mx.upb.ro [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Vxb6sR1OnCCG for ; Sun, 4 Dec 2022 11:38:56 +0200 (EET) Received: from localhost (unknown [141.85.225.204]) by mx.upb.ro (Postfix) with ESMTPSA id 67E92B560415 for ; Sun, 4 Dec 2022 11:38:56 +0200 (EET) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: decf2f08-73b9-11ed-91b6-6bf2151ebd3b DKIM-Filter: OpenDKIM Filter v2.10.3 mx.upb.ro 8B3AAB5608A2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=upb.ro; s=96342B8A-77E4-11E5-BA93-D93D0963A2DF; t=1670146736; bh=pSEXYnWD4WxmOMW6HXaBOk+s30UrSeAQKZ6PGmSIeZ0=; h=From:To:Date:Message-ID:MIME-Version; b=uVGTjz4XBNkxqS/wnhs7nZRlOBYKhqJY5I39JT1/dEkhlb7DXdAHNVq+3jG7WwLF1 ieSBctdnyRbJba+cAtdFcr2X/dNZPI7cTeULnMhkf+0ICWeQ96lDtcfQAyDaJKEPsw bJx6nmFqINbZ2GOGq0wZShZD95RH9VcnVr5A7hA8= X-Virus-Scanned: amavisd-new at upb.ro From: Razvan Deaconescu To: MirageOS Devel Subject: [FOSDEM'23] Submit talk for the "Microkernel and Component-based OS" devroom Date: Sun, 04 Dec 2022 11:38:56 +0200 Message-ID: <86a643y0cv.fsf@drone.cs.pub.ro> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Hi. I am Razvan Deaconescu, I'm from University POLITEHNICA of Bucharest (working on Unikraft unikernel) and I'm managing the "Microkernel and Component-based OS" devroom[2] at FOSDEM'2023[1]. The devroom will take place in person, at FOSDEM'23, in Brussels, on Sunday, February 5, 2023. If you have an interesting topic to talk about it would be great if you made a submission. This is something easy to do, takes about 15-20 minutes on Pentabarf[3]. The deadline for submission is Wednesday, December 7, 2022. [1] https://fosdem.org/2023/ [2] https://fosdem.org/2023/schedule/track/microkernel_and_component_based_os/ [3] https://penta.fosdem.org/submission/ Razvan From mirageos-devel-bounces@lists.xenproject.org Wed Dec 07 14:39:43 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 07 Dec 2022 14:39:43 +0000 Received: from list by lists.xenproject.org with outflank-mailman.456375.714158 (Exim 4.92) (envelope-from ) id 1p2va7-0005DL-0A; Wed, 07 Dec 2022 14:39:31 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 456375.714158; Wed, 07 Dec 2022 14:39:30 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p2va6-0005DE-Tc; Wed, 07 Dec 2022 14:39:30 +0000 Received: by outflank-mailman (input) for mailman id 456375; Wed, 07 Dec 2022 14:39:29 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p2va5-0005D8-PM for mirageos-devel@lists.xenproject.org; Wed, 07 Dec 2022 14:39:29 +0000 Received: from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com [2001:4860:4864:20::2a]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f3456288-763c-11ed-8fd2-01056ac49cbb; Wed, 07 Dec 2022 15:39:28 +0100 (CET) Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-1443a16b71cso18003327fac.13 for ; Wed, 07 Dec 2022 06:39:27 -0800 (PST) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: f3456288-763c-11ed-8fd2-01056ac49cbb DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=aFmPNwvH1yu2e1FI7gdj8MnjPQP4bu1vIc/N8w6AeVM=; b=mcpXQP9HQMDZERCXGxYmVBtQpa7rASjst+tE1C1ZriVRA77jtFlrpNBS2+yKRJcznJ Iw6rUJRlhdiEN/q0QkTayrn/4EGh0OJldol+syswn365Vf7FN4fG6/Ki9aBZ51dH79lS 48iG+so6FAb7ycRNtLjYRqBjid59aHdJqur0JXmZKt92RaiOVXPVZgYD7/jz2HCKVqwf zZ4MPPvIGHke1PD+Xe6HtOfeW9v9uS9U+hOywflsExjM7kJkDjbf4Zh/vLTj1aqNbrCY uvu1OnYst1xxGyOPFS12hKoeyf8+f3dDx5sL4JO1HdVbKLnOQDHTGmDUVYeNoEPNIeKI OyYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aFmPNwvH1yu2e1FI7gdj8MnjPQP4bu1vIc/N8w6AeVM=; b=hbOJOgE77y3MgXVKhpo3rMH+iHRqJQR4245rbcHf70KPp1hmtkdje39DcbNXBd6oY0 ZGSVt1KCyr2QhKgGPEacq0aFLEWOFz8tmUX7K2yXnacS7k2NyN4cCuTj/Nek4Ov9z9tU 5j7n9PvbiMxLYoAY+cWTNHCWY3OmePFDvfrk51yjo+YfElnhamEivXRAtKqvbd61Gqby 9HoKhhPuRAqFFKXPKicPwSoA9z/J/gvnwb12AWgexhG9GNTZxt3X6B0cUd2sHX7CBzCS wvbmrDLU6HGrbbbXOIygMe476dN8ZTie/2tUJHzaHNCCNnwFIGZkcbGr6PKve+nw1Neg Pmow== X-Gm-Message-State: ANoB5pkoOV3cDnAAwbO0h1ZjT0Bzrw4LOppeo3tOM36kCf8lM70uGAOT CG8ZcFOXSeml92+OShhMFHARAMOe3V/us0Dx2TDNwbM5pl+bPw== X-Google-Smtp-Source: AA0mqf6FyD1Ccf1CCnjkrcE3ziGzxuZJnbwZFmnM04I5XyF3dJWm6EIWz7SH0C+j16PezC2HkPOFX5VJj5uPoG8u8Bo= X-Received: by 2002:a05:6870:eaa6:b0:13b:cff2:6387 with SMTP id s38-20020a056870eaa600b0013bcff26387mr48859728oap.35.1670423966445; Wed, 07 Dec 2022 06:39:26 -0800 (PST) MIME-Version: 1.0 References: <630516db-486c-4638-3000-a4dca32a1a91@gmail.com> <2d6b491d-6ed2-e110-776c-94731e5d1bf9@mehnert.org> <14355FA8-5DF1-4EC1-B402-F654CAACC4C3@cl.cam.ac.uk> In-Reply-To: <14355FA8-5DF1-4EC1-B402-F654CAACC4C3@cl.cam.ac.uk> From: Romain Calascibetta Date: Wed, 7 Dec 2022 15:39:14 +0100 Message-ID: Subject: Re: MirageOS meeting 2022-11-30 To: Anil Madhavapeddy Cc: Hannes Mehnert , mirageos-devel@lists.xenproject.org Content-Type: multipart/alternative; boundary="00000000000024f3cc05ef3de30d" --00000000000024f3cc05ef3de30d Content-Type: text/plain; charset="UTF-8" Hi, It's not a real `mmap` but more a `read()` without `Lwt`. In the case when we limit the access to the block (only for reading), it's fine to provide a `read()` without a scheduling idea mainly because whatever what we do with the block, it's a read-only block and data will be the same all the time. The idea behind that is to unlock the ability to create a read-only file-system and where the access of datas will not be determined by a underlying scheduler. A new signature like: ```ocaml module type Mirage_block.RD = sig type t val read : offset:int64 -> Cstruct.t end ``` Will allow us to make a `Mirage_kv.RO` and be able to compose such layout (the file-system layout) with something else (like `ccm_block`). Best, On Wed, Nov 30, 2022 at 6:01 PM Anil Madhavapeddy wrote: > On 30 Nov 2022, at 15:13, Hannes Mehnert wrote: > > > ## `mmap` available on `Mirage_block.S` (dinosaure, > https://github.com/mirage/mirage-block/issues/53) > - dinosaure has an implementation to get a part of the block (similar to > mmap), without being in the Lwt monad > - at the moment, read is in Lwt.t, i.e. does not block, but returns the > filled page(s) > - dinosaure needs a blocking function that returns the data > - the solo5 interface is already blocking (and synchronous), > mirage-block-solo5 adds the asynchronous stuff > - christiano mentions that it could be done with locking > - maybe develop a block read-only interface with a synchronous read > > > In general, having "automatic" scheduling via mmap is a bad idea for > anything non-trivial, since you slow to a crawl when under memory pressure > and having a lot of page faults. There's no way a caller can determine > whether or not a set of accesses will result in a blocking fetch or not. > > It may be workable for a read-only mmap, but... why do you want it? To get > out of Lwt allocations? > > Anil > -- Romain Calascibetta - http://din.osau.re/ --00000000000024f3cc05ef3de30d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

It's not a real `mmap` bu= t more a `read()` without `Lwt`. In the case when we limit the access to th= e block (only for reading), it's fine to provide a `read()` without a s= cheduling idea mainly because whatever what we do with the block, it's = a read-only block and data will be the same all the time.
The idea = behind that is to unlock the ability to create a read-only file-system and = where the access of datas will not be determined by a underlying scheduler.= A new signature like:
```ocaml
module type Mirage_bloc= k.RD =3D sig
=C2=A0 type t

=C2=A0 val re= ad : offset:int64 -> Cstruct.t
end
```
Will allow us to make a `Mirage_kv.RO` and be able to compose s= uch layout (the file-system layout) with something else (like `ccm_block`).=

Best,

On Wed, Nov 30, 2022 at 6:01 PM An= il Madhavapeddy <avsm2@cl.cam.ac.u= k> wrote:
On 30 Nov 2022, at 15:13, Hannes Mehnert <hannes@mehnert.org> wrote:
<= blockquote type=3D"cite">
## `mmap` available on `Mirage_block.S` (dinosaure,=C2=A0<= /span>https://github.com/mirage/mirage-block/issues/53)
- dinosaure has an implementation to get a part of the b= lock (similar to mmap), without being in the Lwt monad
- at the moment, read is in= Lwt.t, i.e. does not block, but returns the filled page(s)
- dinosaure needs a bl= ocking function that returns the data
- the solo5 interface is already blocking (a= nd synchronous), mirage-block-solo5 adds the asynchronous stuff
- christiano menti= ons that it could be done with locking
- maybe develop a block read-only interfa= ce with a synchronous read
=
In general, having "automatic" scheduling via mmap is a = bad idea for anything non-trivial, since you slow to a crawl when under mem= ory pressure and having a lot of page faults. There's no way a caller c= an determine whether or not a set of accesses will result in a blocking fet= ch or not.

It may be workable for a read-only mmap= , but... why do you want it? To get out of Lwt allocations?

<= /div>
Anil


--
<= div dir=3D"ltr" class=3D"gmail_signature">Romain Calascibetta - http://din.osau.re/ --00000000000024f3cc05ef3de30d-- From mirageos-devel-bounces@lists.xenproject.org Wed Dec 07 14:53:34 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 07 Dec 2022 14:53:34 +0000 Received: from list by lists.xenproject.org with outflank-mailman.456381.714163 (Exim 4.92) (envelope-from ) id 1p2vnd-00086o-W4; Wed, 07 Dec 2022 14:53:29 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 456381.714163; Wed, 07 Dec 2022 14:53:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p2vnd-00086h-Sm; Wed, 07 Dec 2022 14:53:29 +0000 Received: by outflank-mailman (input) for mailman id 456381; Wed, 07 Dec 2022 14:53:28 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p2vna-00086X-HW for mirageos-devel@lists.xenproject.org; Wed, 07 Dec 2022 14:53:28 +0000 Received: from mta3.cl.cam.ac.uk (mta3.cl.cam.ac.uk [2001:630:212:200::25:3]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id e5ece03e-763e-11ed-8fd2-01056ac49cbb; Wed, 07 Dec 2022 15:53:23 +0100 (CET) Received: from avsm2-manual-fw2.pem.private.cam.ac.uk ([172.28.255.232]:54331 helo=smtpclient.apple) (dnseec=no) by mta3.cl.cam.ac.uk:587 [128.232.25.23] with esmtpsa auth as avsm2+recoil (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) id 1p2vnX-00Gox9-1D (envelope-from ); Wed, 07 Dec 2022 14:53:23 +0000 X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: e5ece03e-763e-11ed-8fd2-01056ac49cbb DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cl.cam.ac.uk; s=mta3; h=References:To:Cc:In-Reply-To:Date:Subject: Mime-Version:Content-Type:Message-Id:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=JXjauf0m4yaKoPULEWFsUsoj1nF71PopjK7yUn5BYnk=; t=1670424803; x=1671288803; b=ldMht46SMzd/aceqeF8N2SupbCki40YKdXrjsV8ReLQtAJt1pV5ff2+w8LmxvwoC0SnlJXDTWWU 1opfZk3a4qFW7UZxQD3UELp0ZcHm6fYCI53crdJAA6cecdjSzTWQFpgzHtcTeiXkDKBRFtxmCxlwa oNFEdoTDok8AYL8t5SQq4Qmej2A5Eozf5cchSk3r5LfruaZyykRarj14DgSW57BuuOjzxn1THGqdi Ko8rssSuZceoRUBs/KiH7EcI1T4rys6XQ0Cr3nkCs2//Hrq8gYeYSoDMMN8+HZeSuCNJTliK55vO3 9N/Xql89Tqse62RK+9g0+j7COsLbA25g7YGQ==; From: Anil Madhavapeddy Message-Id: <8A8BA811-B554-4B29-879E-94DE23CF8218@cl.cam.ac.uk> Content-Type: multipart/alternative; boundary="Apple-Mail=_D3C92DD2-FEAD-40EA-9A6D-8370F322F10A" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: Re: MirageOS meeting 2022-11-30 Date: Wed, 7 Dec 2022 14:53:09 +0000 In-Reply-To: Cc: Hannes Mehnert , mirageos-devel@lists.xenproject.org To: Romain Calascibetta References: <630516db-486c-4638-3000-a4dca32a1a91@gmail.com> <2d6b491d-6ed2-e110-776c-94731e5d1bf9@mehnert.org> <14355FA8-5DF1-4EC1-B402-F654CAACC4C3@cl.cam.ac.uk> X-Mailer: Apple Mail (2.3731.200.110.1.12) Received-SPF: neutral client-ip=172.28.255.232; envelope-from=avsm2@cl.cam.ac.uk; helo=smtpclient.apple --Apple-Mail=_D3C92DD2-FEAD-40EA-9A6D-8370F322F10A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I'd be much more comfortable doing this once direct-style IO lands in = Mirage. mmap is a _really_ POSIX-oriented abstraction, and exposing it = in all Mirage_block.S will add a fair bit of complexity to any = implementations. It's true that it's simpler if you just restrict it to = read-only, but there is still the problem of when to unmap something. Note that access to the data is still determined by an underlying = scheduler -- but instead of Lwt or the OCaml runtime, you are now = depending on the page table subsystem to do this decision making for = you. Do you have an example of a usecase you want to solve here -- is it for = fairly small pieces of encrypted data?=20 Anil On 7 Dec 2022, at 14:39, Romain Calascibetta = wrote: >=20 > Hi, >=20 > It's not a real `mmap` but more a `read()` without `Lwt`. In the case = when we limit the access to the block (only for reading), it's fine to = provide a `read()` without a scheduling idea mainly because whatever = what we do with the block, it's a read-only block and data will be the = same all the time. > The idea behind that is to unlock the ability to create a read-only = file-system and where the access of datas will not be determined by a = underlying scheduler. A new signature like: > ```ocaml > module type Mirage_block.RD =3D sig > type t >=20 > val read : offset:int64 -> Cstruct.t > end > ``` >=20 > Will allow us to make a `Mirage_kv.RO` and be able to compose such = layout (the file-system layout) with something else (like `ccm_block`). >=20 > Best, >=20 > On Wed, Nov 30, 2022 at 6:01 PM Anil Madhavapeddy > wrote: >> On 30 Nov 2022, at 15:13, Hannes Mehnert > wrote: >>>=20 >>> ## `mmap` available on `Mirage_block.S` (dinosaure, = https://github.com/mirage/mirage-block/issues/53) >>> - dinosaure has an implementation to get a part of the block = (similar to mmap), without being in the Lwt monad >>> - at the moment, read is in Lwt.t, i.e. does not block, but returns = the filled page(s) >>> - dinosaure needs a blocking function that returns the data >>> - the solo5 interface is already blocking (and synchronous), = mirage-block-solo5 adds the asynchronous stuff >>> - christiano mentions that it could be done with locking >>> - maybe develop a block read-only interface with a synchronous read >>=20 >> In general, having "automatic" scheduling via mmap is a bad idea for = anything non-trivial, since you slow to a crawl when under memory = pressure and having a lot of page faults. There's no way a caller can = determine whether or not a set of accesses will result in a blocking = fetch or not. >>=20 >> It may be workable for a read-only mmap, but... why do you want it? = To get out of Lwt allocations? >>=20 >> Anil >=20 >=20 > --=20 > Romain Calascibetta - http://din.osau.re/ --Apple-Mail=_D3C92DD2-FEAD-40EA-9A6D-8370F322F10A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
I'd be = much more comfortable doing this once direct-style IO lands in Mirage. =  mmap is a _really_ POSIX-oriented abstraction, and exposing it in = all Mirage_block.S will add a fair bit of complexity to any = implementations.  It's true that it's simpler if you just restrict = it to read-only, but there is still the problem of when to unmap = something.

Note that access to the data is = still determined by an underlying scheduler -- but instead of Lwt or the = OCaml runtime, you are now depending on the page table subsystem to do = this decision making for you.

Do you have an = example of a usecase you want to solve here -- is it for fairly small = pieces of encrypted = data? 

Anil

On 7 Dec = 2022, at 14:39, Romain Calascibetta = <romain.calascibetta@gmail.com> wrote:

Hi,

It's not a real `mmap` but more = a `read()` without `Lwt`. In the case when we limit the access to the = block (only for reading), it's fine to provide a `read()` without a = scheduling idea mainly because whatever what we do with the block, it's = a read-only block and data will be the same all the time.
The = idea behind that is to unlock the ability to create a read-only = file-system and where the access of datas will not be determined by a = underlying scheduler. A new signature = like:
```ocaml
module type Mirage_block.RD =3D = sig
  type t

  val read : = offset:int64 -> = Cstruct.t
end
```

Will = allow us to make a `Mirage_kv.RO` and be able to compose such layout = (the file-system layout) with something else (like = `ccm_block`).

Best,

On Wed, Nov = 30, 2022 at 6:01 PM Anil Madhavapeddy <avsm2@cl.cam.ac.uk> = wrote:
On 30 Nov 2022, at 15:13, Hannes = Mehnert <hannes@mehnert.org> wrote:

## `mmap` available on = `Mirage_block.S` (dinosaure, https://github.com/mirage/mirage-block/issues/53)
- dinosaure has an = implementation to get a part of the block (similar to mmap), without = being in the Lwt monad
- at the moment, read is in = Lwt.t, i.e. does not block, but returns the filled page(s)
- dinosaure needs a blocking = function that returns the data
- the solo5 interface is = already blocking (and synchronous), mirage-block-solo5 adds the = asynchronous stuff
- christiano mentions that it = could be done with locking
- maybe develop a block = read-only interface with a synchronous read

In general, having = "automatic" scheduling via mmap is a bad idea for anything non-trivial, = since you slow to a crawl when under memory pressure and having a lot of = page faults. There's no way a caller can determine whether or not a set = of accesses will result in a blocking fetch or = not.

It may be workable for a read-only mmap, = but... why do you want it? To get out of Lwt = allocations?

Anil
<= br clear=3D"all">
--
Romain Calascibetta - http://din.osau.re/

= --Apple-Mail=_D3C92DD2-FEAD-40EA-9A6D-8370F322F10A-- From mirageos-devel-bounces@lists.xenproject.org Wed Dec 14 09:43:54 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 14 Dec 2022 09:43:54 +0000 Received: from list by lists.xenproject.org with outflank-mailman.461936.720110 (Exim 4.92) (envelope-from ) id 1p5OIf-000749-Jy; Wed, 14 Dec 2022 09:43:41 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 461936.720110; Wed, 14 Dec 2022 09:43:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5OIf-000742-Gx; Wed, 14 Dec 2022 09:43:41 +0000 Received: by outflank-mailman (input) for mailman id 461936; Wed, 14 Dec 2022 09:43:40 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5OIe-00071i-4o for mirageos-devel@lists.xenproject.org; Wed, 14 Dec 2022 09:43:40 +0000 Received: from mail.mehnert.org (mail.mehnert.org [213.73.89.200]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id c87c4fc0-7b93-11ed-8fd2-01056ac49cbb; Wed, 14 Dec 2022 10:43:38 +0100 (CET) Received: from [192.168.42.80] (dslb-094-223-123-080.094.223.pools.vodafone-ip.de [94.223.123.80]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "hannes@mehnert.org", Issuer "mehnert root CA" (not verified)) by mail.mehnert.org (Postfix) with ESMTPS id DE57D4F1D for ; Wed, 14 Dec 2022 10:43:35 +0100 (CET) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: c87c4fc0-7b93-11ed-8fd2-01056ac49cbb Message-ID: <24fc9ecb-ea44-aa43-b9ab-e20f4de106f1@mehnert.org> Date: Wed, 14 Dec 2022 10:43:34 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 From: Hannes Mehnert Subject: Reminder: MirageOS meeting today 14:00 CET To: mirageos-devel Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hello everyone, as a reminder we'll have later today at 14:00 CET (in 3 hours) the next MirageOS meeting at https://whereby.com/ocamllabs. Please find and edit the agenda at https://pad.data.coop/RszC9z57QIi4ovLXUvVWIw# See you soon, Hannes From mirageos-devel-bounces@lists.xenproject.org Wed Dec 14 13:51:44 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 14 Dec 2022 13:51:44 +0000 Received: from list by lists.xenproject.org with outflank-mailman.462267.720435 (Exim 4.92) (envelope-from ) id 1p5SAV-0004on-6z; Wed, 14 Dec 2022 13:51:31 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 462267.720435; Wed, 14 Dec 2022 13:51:31 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5SAV-0004og-4L; Wed, 14 Dec 2022 13:51:31 +0000 Received: by outflank-mailman (input) for mailman id 462267; Wed, 14 Dec 2022 13:51:30 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5SAT-0004oa-VK for mirageos-devel@lists.xenproject.org; Wed, 14 Dec 2022 13:51:29 +0000 Received: from mail.mehnert.org (mail.mehnert.org [213.73.89.200]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 682acfdb-7bb6-11ed-8fd2-01056ac49cbb; Wed, 14 Dec 2022 14:51:28 +0100 (CET) Received: from [192.168.42.80] (dslb-094-223-123-080.094.223.pools.vodafone-ip.de [94.223.123.80]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "hannes@mehnert.org", Issuer "mehnert root CA" (not verified)) by mail.mehnert.org (Postfix) with ESMTPS id F01FC6250 for ; Wed, 14 Dec 2022 14:51:26 +0100 (CET) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 682acfdb-7bb6-11ed-8fd2-01056ac49cbb Message-ID: Date: Wed, 14 Dec 2022 14:51:26 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Content-Language: en-US To: mirageos-devel From: Hannes Mehnert Subject: MSA 03: xen with solo5 >= 0.6.6 & < 0.7.5, qubes-mirage-firewall < 0.8.4 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ## MirageOS Security Advisory 03 - infinite loop in console output on xen - Module: solo5 - Announced: 2022-12-07 - Credits: Krzysztof Burghardt, Pierre Alain, Thomas Leonard, Hannes Mehnert - Affects: solo5 >= 0.6.6 & < 0.7.5, qubes-mirage-firewall >= 0.8.0 & < 0.8.4 - Corrected: 2022-12-07: solo5 0.7.5, 2022-12-07: qubes-mirage-firewall 0.8.4 - CVE: CVE-2022-46770 For general information regarding MirageOS Security Advisories, please visit [https://mirage.io/security](https://mirage.io/security). ### Background MirageOS is a library operating system using cooperative multitasking, which can be executed as a guest of the Xen hypervisor. Output on the console is performed via the Xen console protocol. ### Problem Description Since MirageOS moved from PV mode to PVH, and thus replacing Mini-OS with solo5, there was an issue in the solo5 code which failed to properly account the already written bytes on the console. This only occurs if the output to be performed does not fit in a single output buffer (2048 bytes on Xen). The code in question set the number of bytes written to the last written count (written = output_some(buf)), instead of increasing the written count (written += output_some(buf)). ### Impact Console output may lead to an infinite loop, endlessly printing data onto the console. A prominent unikernel is the Qubes MirageOS firewall, which prints some input packets onto the console. This can lead to a remote denial of service vulnerability, since any client could send a malformed and sufficiently big network packet. ### Workaround No workaround is available. ### Solution The solution is to fix the console output code in solo5, as done in https://github.com/Solo5/solo5/pull/538/commits/099be86f0a17a619fcadbb970bb9e511d28d3cd8 For the qubes-mirage-firewall, update to a solo5 release (0.7.5) which has the issue fixed. This has been done in the release 0.8.4 of qubes-mirage-firewall. The recommended way to upgrade is: ```bash opam update opam upgrade solo5 ``` ### Correction details The following PRs were part of the fix: - [solo5/pull/538](https://github.com/Solo5/solo5/pull/538) - xen console: update the "to be written" count - [qubes-mirage-firewall/pull/167](https://github.com/mirage/qubes-mirage-firewall/pull/167) - update opam repository commit ### Timeline - 2022-12-04: initial report by Krzysztof Burghardt https://github.com/mirage/qubes-mirage-firewall/issues/166 - 2022-12-04: investigation by Hannes Mehnert and Pierre Alain - 2022-12-05: initial fix by Pierre Alain https://github.com/Solo5/solo5/pull/538 - 2022-12-05: review of fix by Thomas Leonard - 2022-12-07: release of fixed packages and security advisory ### References You can find the latest version of this advisory online at [https://mirage.io/blog/MSA03](https://mirage.io/blog/MSA03). This advisory is signed using OpenPGP, you can verify the signature by downloading our public key from a keyserver (`gpg --recv-key 4A732D757C0EDA74`), downloading the raw markdown source of this advisory from [GitHub](https://raw.githubusercontent.com/mirage/mirage-www/master/tmpl/advisories/03.txt.asc) and executing `gpg --verify 03.txt.asc`. From mirageos-devel-bounces@lists.xenproject.org Wed Dec 14 13:52:40 2022 Return-path: Envelope-to: archives@lists.xenproject.org Delivery-date: Wed, 14 Dec 2022 13:52:40 +0000 Received: from list by lists.xenproject.org with outflank-mailman.462272.720440 (Exim 4.92) (envelope-from ) id 1p5SBb-0004ze-CU; Wed, 14 Dec 2022 13:52:39 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 462272.720440; Wed, 14 Dec 2022 13:52:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5SBb-0004zX-9e; Wed, 14 Dec 2022 13:52:39 +0000 Received: by outflank-mailman (input) for mailman id 462272; Wed, 14 Dec 2022 13:52:38 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1p5SBa-0004z0-1m for mirageos-devel@lists.xenproject.org; Wed, 14 Dec 2022 13:52:38 +0000 Received: from mail.mehnert.org (mail.mehnert.org [213.73.89.200]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 902ee5ef-7bb6-11ed-91b6-6bf2151ebd3b; Wed, 14 Dec 2022 14:52:36 +0100 (CET) Received: from [192.168.42.80] (dslb-094-223-123-080.094.223.pools.vodafone-ip.de [94.223.123.80]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "hannes@mehnert.org", Issuer "mehnert root CA" (not verified)) by mail.mehnert.org (Postfix) with ESMTPS id 8DC27486E for ; Wed, 14 Dec 2022 14:52:35 +0100 (CET) X-BeenThere: mirageos-devel@lists.xenproject.org List-Id: Developer list for MirageOS List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: mirageos-devel-bounces@lists.xenproject.org Precedence: list Sender: "MirageOS-devel" X-Inumbo-ID: 902ee5ef-7bb6-11ed-91b6-6bf2151ebd3b Message-ID: <886d9b9e-b7f6-aac5-b939-d0052c51bc1a@mehnert.org> Date: Wed, 14 Dec 2022 14:52:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: Reminder: MirageOS meeting today 14:00 CET Content-Language: en-US From: Hannes Mehnert To: mirageos-devel References: <24fc9ecb-ea44-aa43-b9ab-e20f4de106f1@mehnert.org> In-Reply-To: <24fc9ecb-ea44-aa43-b9ab-e20f4de106f1@mehnert.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Notes from the meeting, next one in three weeks (January 4th). Happy holidays! # Mirage 2022-12-14 participants: christiano, talex5, palainp, reynir, hannes ## qubes-mirage-firewall security issue - two years after going from Mini-OS (PV) to solo5 (PVH), there was a denial of service for console output - an infinite loop in console output when the output was too big (not fitting into a single write on xen) - the change was changing "=" to "+=" - lead to a security advisory, CVE 2022 46770, MSA03 ## mirage.4.3.2 released ## mirage-kv 6.0.0 - optint - allocate function to reserve space. Use case is e.g. partial writes when downloading files over http - use less memory. ## Hillingar: MirageOS Unikernels on NixOS - https://ryan.freumh.org/blog/hillingar/ ## Mirage on OCaml 5 - PR for OCaml-solo5 https://github.com/mirage/ocaml-solo5/pull/122 - still some segfaults, needs further work - christiano solo5 and multicore still of interest ## TCP benchmark for the qubes firewall - christiano: maybe reordering? - nobody had time to work on it ## next meeting: January 4h 2023 14:00 CET On 14/12/2022 10:43, Hannes Mehnert wrote: > Hello everyone, > > as a reminder we'll have later today at 14:00 CET (in 3 hours) the next > MirageOS meeting at https://whereby.com/ocamllabs. > > Please find and edit the agenda at > https://pad.data.coop/RszC9z57QIi4ovLXUvVWIw# > > > See you soon, > > Hannes >