From anil@recoil.org Tue May 08 11:31:20 2012 Received: from ppsw-52.csi.cam.ac.uk ([131.111.8.152]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SRhhQ-0001Et-Mn (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 08 May 2012 11:31:20 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:13342 helo=dark.recoil.org) by ppsw-52.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with smtp id 1SRhhP-0006El-FJ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 08 May 2012 11:31:20 +0100 Received: (qmail 2072 invoked by uid 634); 8 May 2012 10:31:19 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from freak-0.srg.cl.cam.ac.uk (HELO [192.168.2.15]) (128.232.32.220) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 08 May 2012 11:31:19 +0100 From: Anil Madhavapeddy Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Mirage release milestone Date: Tue, 8 May 2012 11:31:17 +0100 Message-Id: <436B4BB1-8F7F-4007-A669-21D277137C04@recoil.org> To: "cl-mirage@lists.cam.ac.uk Mailing List" Mime-Version: 1.0 (Apple Message framework v1257) X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on dark.recoil.org X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2012 10:31:20 -0000 X-Keywords: X-UID: 258 Status: O Content-Length: 1500 Lines: 45 We have made great progress in benchmarking and understanding the = performance characteristics of Mirage kernels in the past few months. It's now time = to cut a reasonably stable alpha release to act as a base for future work. = Luckily, not a huge amount of work is needed for this! I've created a 0.2-alpha milestone in Github: https://github.com/avsm/mirage/issues?milestone=3D3&state=3Dopen The three main issues are: Convert to Bigarray I/O exclusively: https://github.com/avsm/mirage/issues/93 Disaggregate the giant monolithic repository: https://github.com/avsm/mirage/issues/95 Convert Bitstring into a C-like 'struct' camlp4 extension: https://github.com/avsm/mirage/issues/94 I can't figure out how to CC Github issue traffic to this list (or even if we should), but I'll update those issues with progress as I work = through them. None of these are particularly chunky bits of work, but they touch = the whole repository and so should be complete before we release any code. Comments (or other things anyone would like to do before the 0.2 alpha) = are welcome. In particular, Amir's Jenkins setup should be integrated, so = that we have performance regression tests, so help on that front (both = writing tests and the Jenkins integration) would be welcome.=20 Also Thomas: your thoughts on #95 and OPAM integration Dave: figuring out the OS.Xen/OS.Unix portion of #95 (with respect to = the=20 user-space blkback ring patches, configure scripts, and all that stuff) -anil= From anil@recoil.org Thu May 10 16:42:55 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSVW3-00042R-Cn (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:42:55 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:26584 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1SSVW2-0006cq-SB (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:42:55 +0100 Received: (qmail 13415 invoked by uid 634); 10 May 2012 15:39:32 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from freak-0.srg.cl.cam.ac.uk (HELO [192.168.2.15]) (128.232.32.220) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 10 May 2012 16:39:32 +0100 From: Anil Madhavapeddy Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: pa_cstruct first cut available Date: Thu, 10 May 2012 16:42:52 +0100 Message-Id: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> To: "cl-mirage@lists.cam.ac.uk Mailing List" Mime-Version: 1.0 (Apple Message framework v1257) X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on dark.recoil.org X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 15:42:55 -0000 X-Keywords: X-UID: 259 Status: O Content-Length: 856 Lines: 21 I've knocked together the C-struct syntax extension, and it's actually = quite simple and effective. https://github.com/avsm/ocaml-cstruct/blob/master/lib_test/pcap.ml is a pcap parser that cracks open pcap/ethernet/ipv4/tcp and displays = the payloads. The 'cstruct' declaration defines all the accessors as you = might expect, and the underlying buffers are all reference counted (as = in MPL) and so can be passed around. The base-pointer can also be = retrieved, so you can do comparisons of the underlying Io_page (e.g. to = get a Xen grant onto it). If noone has any objections, I'll add unit tests for the write path, and = merge this into Mirage in my os-bigarray branch. I dont think we = actually need bitstring at all anymore. Mort: reckon you can easily port the DNS code to this? It should speed = up the responses an awful lot. -anil= From Richard.Mortier@nottingham.ac.uk Thu May 10 16:48:39 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSVbb-0004I2-CO (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:48:39 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from thb-mta-19-tx.emailfiltering.com ([194.116.199.150]:45169 helo=thb-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SSVbW-0001DO-ph (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:48:39 +0100 Received: from smtp1.nottingham.ac.uk ([128.243.44.4]) by thb-mta-19.emailfiltering.com with emfmta (version 4.8.5.86) by TLS id 2414389570 ;7b10b2a523f51b26; Thu, 10 May 2012 16:48:13 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp1.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1SSVbB-0004PQ-F3 for cl-mirage@lists.cam.ac.uk; Thu, 10 May 2012 16:48:13 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::7962:f868:e6ee:6267]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Thu, 10 May 2012 16:48:05 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Thu, 10 May 2012 16:48:11 +0100 Subject: Re: pa_cstruct first cut available Thread-Topic: pa_cstruct first cut available Thread-Index: Ac0uxEmPTyuqK4gQRla6NXmclP7HNA== Message-ID: References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> In-Reply-To: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 15:48:39 -0000 X-Keywords: X-UID: 260 Status: O Content-Length: 1411 Lines: 39 On 10 May 2012, at 16:42, Anil Madhavapeddy wrote: > If noone has any objections, I'll add unit tests for the write path, and = merge this into Mirage in my os-bigarray branch. I dont think we actually = need bitstring at all anymore. sweet :) none from me! by way of a request for status, what are the branches we should currently b= e tracking for mirage, or just wait until you announce the results of the b= ig split/merge? > Mort: reckon you can easily port the DNS code to this? It should speed up= the responses an awful lot. i'll take a look. and, as i threatened, pyrt too :) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Thu May 10 16:55:18 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSVi2-0004f9-Ns (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:55:18 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:44149 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1SSVi2-0000pl-QS (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 16:55:18 +0100 Received: (qmail 8326 invoked by uid 634); 10 May 2012 15:51:59 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from freak-0.srg.cl.cam.ac.uk (HELO [192.168.2.15]) (128.232.32.220) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 10 May 2012 16:51:59 +0100 Subject: Re: pa_cstruct first cut available Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Anil Madhavapeddy In-Reply-To: Date: Thu, 10 May 2012 16:55:16 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <8AC69365-1E03-457B-971D-751575D69856@recoil.org> References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> To: Richard Mortier X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 15:55:18 -0000 X-Keywords: X-UID: 261 Status: O Content-Length: 1656 Lines: 42 On 10 May 2012, at 16:48, Richard Mortier wrote: >=20 > On 10 May 2012, at 16:42, Anil Madhavapeddy wrote: >=20 >> If noone has any objections, I'll add unit tests for the write path, = and merge this into Mirage in my os-bigarray branch. I dont think we = actually need bitstring at all anymore. >=20 > sweet :) none from me! >=20 > by way of a request for status, what are the branches we should = currently be tracking for mirage, or just wait until you announce the = results of the big split/merge? I'm working in os-bigarray in my tree, and that's going to touch pretty = much every file in OS and Net. So I'm just going to work through it as = quickly as possible and get it over with. Once that's done, we can = benchmark using Balraj's TCP tests again, and verify that the write path = is up to full speed. >=20 >> Mort: reckon you can easily port the DNS code to this? It should = speed up the responses an awful lot. >=20 > i'll take a look. and, as i threatened, pyrt too :) Great, if you want to do this in parallel, you could make avsm/ocaml-dns = use pa_cstruct. You can get the packets in Bigarray in vanilla UNIX/Lwt = via Lwt_bytes.recv_msg, which is the same underlying type. Would be = good to run a quick perf comparison before/after to make sure it is = indeed faster and allocates less... Incidentally, the pa_cstruct design makes it quite easy to eliminate = bounds checks, by adding an 'attach' operation which verifies the = overall bound of a buffer, and uses unsafe accesses within the generated = code. However, I'll only do this if performance becomes an issue, since = I prefer no unsafe accesses at all. -anil= From anil@recoil.org Thu May 10 17:13:42 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSVzq-0005ZE-Qu (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 17:13:42 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:40985 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1SSVzp-0004fw-Rd (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 17:13:42 +0100 Received: (qmail 23536 invoked by uid 634); 10 May 2012 16:10:28 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from freak-0.srg.cl.cam.ac.uk (HELO [192.168.2.15]) (128.232.32.220) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 10 May 2012 17:10:28 +0100 Subject: Re: pa_cstruct first cut available Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Anil Madhavapeddy In-Reply-To: <5B96DBFE-7579-41C6-A7A8-9BB3C9D68127@gmail.com> Date: Thu, 10 May 2012 17:13:39 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> <5B96DBFE-7579-41C6-A7A8-9BB3C9D68127@gmail.com> To: Thomas Gazagnaire X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 16:13:42 -0000 X-Keywords: X-UID: 262 Status: O Content-Length: 2723 Lines: 66 No manual free: the Bigarray buffers are internally reference counted, = so as soon as all of the views onto a buffer are no longer used, the Gc = will free the underlying resource. The runtime generates a = 'caml_ba_proxy' which is a shared reference count on the underlying = data. One thing I might need to extend the bigarray stubs for is that you = can't register a *custom* finaliser. Each bigarray can either be = CAML_BA_MANAGED (i.e. the runtime allocated the bigarray and should free = it), or CAML_BA_MAPPED_FILE (mmap, so needs an munmap), or = CAML_BA_EXTERNAL (something else, take no action). Right now, the runtime takes no action for CAML_BA_EXTERNAL, but we need = to be able to register a custom finaliser, so that we can do something = interesting with the underlying resource in Xen (mostly putting it back = into the Io_page pool for re-use). This would also be useful in UNIX to = similarly pool pages rather than going to expensive malloc and free all = the time. Are you familiar with that code? Something upstreamable for ocaml-4.0 = (or indeed, if there's another way to do what I want) would be helpful = so we don't need to maintain a custom patch... Regarding your section question, the get_* functions just make a copy = into the local heap (but in most cases, it's a cheap unboxed integer in = the minor heap), and get_buffer calls are references and not copies. = It's all pretty cheap and comparable to Bitstring, which allocates a = fair few tuples. -anil On 10 May 2012, at 17:04, Thomas Gazagnaire wrote: > Neat! >=20 > So you need to manually free the objects you don't use anymore ? > Aslo, do the get_* functions makes copy into the ocaml heap or there = is a clever trick there as well ? >=20 > Thomas >=20 > On May 10, 2012, at 5:42 PM, Anil Madhavapeddy wrote: >=20 >> I've knocked together the C-struct syntax extension, and it's = actually quite simple and effective. >>=20 >> https://github.com/avsm/ocaml-cstruct/blob/master/lib_test/pcap.ml >>=20 >> is a pcap parser that cracks open pcap/ethernet/ipv4/tcp and displays = the payloads. The 'cstruct' declaration defines all the accessors as you = might expect, and the underlying buffers are all reference counted (as = in MPL) and so can be passed around. The base-pointer can also be = retrieved, so you can do comparisons of the underlying Io_page (e.g. to = get a Xen grant onto it). >>=20 >> If noone has any objections, I'll add unit tests for the write path, = and merge this into Mirage in my os-bigarray branch. I dont think we = actually need bitstring at all anymore. >>=20 >> Mort: reckon you can easily port the DNS code to this? It should = speed up the responses an awful lot. >>=20 >> -anil >=20 From thomas.gazagnaire@gmail.com Thu May 10 17:35:39 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSWL5-0006KM-Ro (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 17:35:39 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.9 from SpamAssassin-3.3.2-1293136 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.212.179 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from * author's domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-wi0-f179.google.com ([209.85.212.179]:46730) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SSWL4-0002XN-sC (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 17:35:39 +0100 Received: by wibhr2 with SMTP id hr2so645730wib.2 for ; Thu, 10 May 2012 09:35:38 -0700 (PDT) Received: by 10.180.105.194 with SMTP id go2mr10577536wib.22.1336667738159; Thu, 10 May 2012 09:35:38 -0700 (PDT) Received: from [192.168.0.14] (gou06-3-88-170-165-56.fbx.proxad.net. [88.170.165.56]) by mx.google.com with ESMTPS id 6sm3982730wiz.1.2012.05.10.09.35.35 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 10 May 2012 09:35:36 -0700 (PDT) Subject: Re: pa_cstruct first cut available Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: Date: Thu, 10 May 2012 18:33:34 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <91A91C80-0CCD-4159-BE95-61C0D6AC57D1@gmail.com> References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> <5B96DBFE-7579-41C6-A7A8-9BB3C9D68127@gmail.com> To: Anil Madhavapeddy X-Mailer: Apple Mail (2.1084) Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 16:35:39 -0000 X-Keywords: X-UID: 263 Status: O Content-Length: 3099 Lines: 76 > No manual free: the Bigarray buffers are internally reference counted, = so as soon as all of the views onto a buffer are no longer used, the Gc = will free the underlying resource. The runtime generates a = 'caml_ba_proxy' which is a shared reference count on the underlying = data. nice > One thing I might need to extend the bigarray stubs for is that you = can't register a *custom* finaliser. Each bigarray can either be = CAML_BA_MANAGED (i.e. the runtime allocated the bigarray and should free = it), or CAML_BA_MAPPED_FILE (mmap, so needs an munmap), or = CAML_BA_EXTERNAL (something else, take no action). >=20 > Right now, the runtime takes no action for CAML_BA_EXTERNAL, but we = need to be able to register a custom finaliser, so that we can do = something interesting with the underlying resource in Xen (mostly = putting it back into the Io_page pool for re-use). This would also be = useful in UNIX to similarly pool pages rather than going to expensive = malloc and free all the time. >=20 > Are you familiar with that code? Something upstreamable for ocaml-4.0 = (or indeed, if there's another way to do what I want) would be helpful = so we don't need to maintain a custom patch... yes, I've hacked few things there for numcaml one year ago, so I can try = to remember how it is working :-) Regarding the upstreamable patch, I = guess that's just adding few line in caml_ba_finalize. But I really = can't say if Xavier will accept it ... we can still try to publish the = patch on Mantis and I'll ask Fabrice to convince Xavier ... > Regarding your section question, the get_* functions just make a copy = into the local heap (but in most cases, it's a cheap unboxed integer in = the minor heap), and get_buffer calls are references and not copies. = It's all pretty cheap and comparable to Bitstring, which allocates a = fair few tuples. ok, seems fine to me! -- Thomas >> Neat! >>=20 >> So you need to manually free the objects you don't use anymore ? >> Aslo, do the get_* functions makes copy into the ocaml heap or there = is a clever trick there as well ? >>=20 >> Thomas >>=20 >> On May 10, 2012, at 5:42 PM, Anil Madhavapeddy wrote: >>=20 >>> I've knocked together the C-struct syntax extension, and it's = actually quite simple and effective. >>>=20 >>> https://github.com/avsm/ocaml-cstruct/blob/master/lib_test/pcap.ml >>>=20 >>> is a pcap parser that cracks open pcap/ethernet/ipv4/tcp and = displays the payloads. The 'cstruct' declaration defines all the = accessors as you might expect, and the underlying buffers are all = reference counted (as in MPL) and so can be passed around. The = base-pointer can also be retrieved, so you can do comparisons of the = underlying Io_page (e.g. to get a Xen grant onto it). >>>=20 >>> If noone has any objections, I'll add unit tests for the write path, = and merge this into Mirage in my os-bigarray branch. I dont think we = actually need bitstring at all anymore. >>>=20 >>> Mort: reckon you can easily port the DNS code to this? It should = speed up the responses an awful lot. >>>=20 >>> -anil >>=20 >=20 From Richard.Mortier@nottingham.ac.uk Thu May 10 19:07:18 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSXlm-00001D-UK (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 19:07:18 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:47389 helo=ixe-mta-19.emailfiltering.com) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1SSXlh-0002wG-RW (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 10 May 2012 19:07:18 +0100 Received: from smtp1.nottingham.ac.uk ([128.243.44.4]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.86) by TLS id 2236349912 ;d9de90e408b4fd27; Thu, 10 May 2012 19:06:53 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp1.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1SSXkf-0001sD-Ij for cl-mirage@lists.cam.ac.uk; Thu, 10 May 2012 19:06:09 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::7962:f868:e6ee:6267]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Thu, 10 May 2012 19:05:54 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Thu, 10 May 2012 19:06:02 +0100 Subject: Re: pa_cstruct first cut available Thread-Topic: pa_cstruct first cut available Thread-Index: Ac0u14p7YrcluMseQfOgzkwoU4hoWw== Message-ID: <6FFAC11C-6D2A-4C20-8E6D-10D663B73B3E@nottingham.ac.uk> References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> <8AC69365-1E03-457B-971D-751575D69856@recoil.org> In-Reply-To: <8AC69365-1E03-457B-971D-751575D69856@recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 18:07:19 -0000 X-Keywords: X-UID: 264 Status: O Content-Length: 2662 Lines: 70 On 10 May 2012, at 16:55, Anil Madhavapeddy wrote: >=20 > On 10 May 2012, at 16:48, Richard Mortier wrote: >=20 >>=20 >> On 10 May 2012, at 16:42, Anil Madhavapeddy wrote: >>=20 >>> If noone has any objections, I'll add unit tests for the write path, an= d merge this into Mirage in my os-bigarray branch. I dont think we actuall= y need bitstring at all anymore. >>=20 >> sweet :) none from me! >>=20 >> by way of a request for status, what are the branches we should currentl= y be tracking for mirage, or just wait until you announce the results of th= e big split/merge? >=20 > I'm working in os-bigarray in my tree, and that's going to touch pretty m= uch every file in OS and Net. So I'm just going to work through it as quic= kly as possible and get it over with. Once that's done, we can benchmark us= ing Balraj's TCP tests again, and verify that the write path is up to full = speed. ok great; i'll ignore mirage tree for now and work off ocaml-cstruct and oc= aml-dns. >>> Mort: reckon you can easily port the DNS code to this? It should speed = up the responses an awful lot. >>=20 >> i'll take a look. and, as i threatened, pyrt too :) >=20 > Great, if you want to do this in parallel, you could make avsm/ocaml-dns = use pa_cstruct. You can get the packets in Bigarray in vanilla UNIX/Lwt via= Lwt_bytes.recv_msg, which is the same underlying type. Would be good to r= un a quick perf comparison before/after to make sure it is indeed faster an= d allocates less... >=20 > Incidentally, the pa_cstruct design makes it quite easy to eliminate boun= ds checks, by adding an 'attach' operation which verifies the overall bound= of a buffer, and uses unsafe accesses within the generated code. However, = I'll only do this if performance becomes an issue, since I prefer no unsafe= accesses at all. safety ftw! :) (agree) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Fri May 11 14:59:58 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SSqNy-0003Nu-Tz (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 11 May 2012 14:59:58 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1293136 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:40733 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1SSqNy-0002Zr-Xh (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 11 May 2012 14:59:58 +0100 Received: (qmail 1601 invoked by uid 634); 11 May 2012 13:59:58 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from freak-0.srg.cl.cam.ac.uk (HELO [192.168.2.15]) (128.232.32.220) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Fri, 11 May 2012 14:59:57 +0100 Subject: Re: pa_cstruct first cut available Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Anil Madhavapeddy In-Reply-To: <91A91C80-0CCD-4159-BE95-61C0D6AC57D1@gmail.com> Date: Fri, 11 May 2012 14:59:58 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <223C3A4F-2D4B-4B79-BC30-3CD74B78F346@recoil.org> <5B96DBFE-7579-41C6-A7A8-9BB3C9D68127@gmail.com> <91A91C80-0CCD-4159-BE95-61C0D6AC57D1@gmail.com> To: Thomas Gazagnaire X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2012 13:59:59 -0000 X-Keywords: X-UID: 265 Status: O Content-Length: 1716 Lines: 36 On 10 May 2012, at 17:33, Thomas Gazagnaire wrote: >=20 >> One thing I might need to extend the bigarray stubs for is that you = can't register a *custom* finaliser. Each bigarray can either be = CAML_BA_MANAGED (i.e. the runtime allocated the bigarray and should free = it), or CAML_BA_MAPPED_FILE (mmap, so needs an munmap), or = CAML_BA_EXTERNAL (something else, take no action). >>=20 >> Right now, the runtime takes no action for CAML_BA_EXTERNAL, but we = need to be able to register a custom finaliser, so that we can do = something interesting with the underlying resource in Xen (mostly = putting it back into the Io_page pool for re-use). This would also be = useful in UNIX to similarly pool pages rather than going to expensive = malloc and free all the time. >>=20 >> Are you familiar with that code? Something upstreamable for ocaml-4.0 = (or indeed, if there's another way to do what I want) would be helpful = so we don't need to maintain a custom patch... >=20 > yes, I've hacked few things there for numcaml one year ago, so I can = try to remember how it is working :-) Regarding the upstreamable patch, = I guess that's just adding few line in caml_ba_finalize.. Hm, the problem is that the caml_ba_proxy isn't an OCaml value, and so = an OCaml finaliser can't be directly attached to it.=20 What we really want is a shared finaliser that's only called when the = underlying caml_ba_proxy hits zero. We *could* stash a finaliser value = in the proxy as a new field (and register it as a generational root?), = and invoke it in caml_ba_finalize. This requires forking Bigarray, so I'll do that do that for now (call it = Extbuffer or something) and then generate a patch for Mantis. -anil=