From anil@recoil.org Fri Aug 03 18:24:40 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxLc8-0003fk-3K (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 03 Aug 2012 18:24:40 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:4810 helo=dark.recoil.org) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with smtp id 1SxLc7-0006vn-mh (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 03 Aug 2012 18:24:40 +0100 Received: (qmail 18607 invoked by uid 634); 3 Aug 2012 17:24:38 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc2-belf1-0-0-cust23.2-1.cable.virginmedia.com (HELO [10.0.1.16]) (82.29.224.24) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Fri, 03 Aug 2012 18:24:38 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Mirage/kFreeBSD on GitHub From: Anil Madhavapeddy In-Reply-To: <20120731133034.GB7254@caesar.elte.hu> Date: Fri, 3 Aug 2012 18:24:35 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> References: <20120731133034.GB7254@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2012 17:24:40 -0000 X-Keywords: X-UID: 435 Status: O Content-Length: 1885 Lines: 56 On 31 Jul 2012, at 14:30, PALI Gabor Janos wrote: > Hi there, >=20 > I have just added a repository with an initial version of the = Mirage/kFreeBSD > port to GitHub: >=20 > https://github.com/pgj/mirage-kfreebsd >=20 > It does not do too much at the moment, however it is possible to build = and run > a kernel module with OCaml inside. I am hoping to continue the = development > there, any feedback is appreciated. Great stuff! I have a FreeBSD 9 VM that I am installing all this on at = the moment. I ran into this after cloning the repo fresh: ~/mirage-kfreebsd/packages/mirage-stdlib/src$ make gmake -f GNUMakefile PREFIX=3D/usr/local all ocamlrun ocamlc -strict-sequence -g -warn-error A -nostdlib `./Compflags = pervasives.cmo` -c pervasives.ml Fatal error: the file ocamlc is not a bytecode executable file gmake: *** [pervasives.cmo] Error 2 Why are you running "ocamlrun ocamlc"? The bytecode binaries are compiled in with a copy of the bytecode interpreter, so you can just invoke them directly as "ocamlc". Better still, use the native version of "ocamlc.opt" which will give you much faster compilation. I applied this to continue the compilation, but I guess this is because you have a custom version of ocaml installed, I'm guessing? diff --git a/packages/mirage-stdlib/src/Makefile.shared = b/packages/mirage-stdlib/src/Makefile.shared index f947f8d..bd89701 100644 --- a/packages/mirage-stdlib/src/Makefile.shared +++ b/packages/mirage-stdlib/src/Makefile.shared @@ -89,10 +89,10 @@ MKMAINDLL=3Dcc -shared =20 RUNTIME=3Docamlrun COMPILER=3Docamlc -CAMLC=3D$(RUNTIME) $(COMPILER) +CAMLC=3D$(COMPILER) COMPFLAGS=3D-strict-sequence -g -warn-error A -nostdlib OPTCOMPILER=3Docamlopt -CAMLOPT=3D$(RUNTIME) $(OPTCOMPILER) +CAMLOPT=3D$(OPTCOMPILER) OPTCOMPFLAGS=3D-warn-error A -nostdlib -g -nodynlink -fno-PIC CAMLDEP=3Docamldep =20 -anil= From anil@recoil.org Fri Aug 03 18:30:21 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxLhd-0003iN-Bt (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 03 Aug 2012 18:30:21 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:1091 helo=dark.recoil.org) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with smtp id 1SxLhc-0008LH-oV (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 03 Aug 2012 18:30:21 +0100 Received: (qmail 24724 invoked by uid 634); 3 Aug 2012 17:30:20 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc2-belf1-0-0-cust23.2-1.cable.virginmedia.com (HELO [10.0.1.16]) (82.29.224.24) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Fri, 03 Aug 2012 18:30:19 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Mirage/kFreeBSD on GitHub From: Anil Madhavapeddy In-Reply-To: <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> Date: Fri, 3 Aug 2012 18:30:16 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2012 17:30:21 -0000 X-Keywords: X-UID: 436 Status: O Content-Length: 3043 Lines: 71 On 3 Aug 2012, at 18:24, Anil Madhavapeddy wrote: > On 31 Jul 2012, at 14:30, PALI Gabor Janos wrote: >=20 >> Hi there, >>=20 >> I have just added a repository with an initial version of the = Mirage/kFreeBSD >> port to GitHub: >>=20 >> https://github.com/pgj/mirage-kfreebsd >>=20 >> It does not do too much at the moment, however it is possible to = build and run >> a kernel module with OCaml inside. I am hoping to continue the = development >> there, any feedback is appreciated. >=20 > Great stuff! I have a FreeBSD 9 VM that I am installing all this on at = the > moment. I ran into this after cloning the repo fresh: >=20 And everything works great after that with your test! Hurrah! I'll = play around with it more over the weekend. [MIRAGE] Kernel module is about to load. --> mirage_kthread_init() <-- mirage_kthread_init() --> mirage_kthread_launch() <-- mirage_kthread_launch() --> mirage_kthread_body() mirage: kernel thread actived, caml run-time starts This message comes somewhere from the body. Current date and time: 2012.08.03. Fri 10:02:01. (216) mirage: caml run-time finished, searching the main function mirage: main function found, kicking off the main loop Sorry, no in-kernel Mirage today! And there is a second message... Finally a third one. Let's do some Fibonacci! [n =3D 10][n =3D 8][n =3D 6][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D = 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 5][n =3D 3][n =3D 1][n =3D = 2][n =3D 0][n =3D 1][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D = 1][n =3D 2][n =3D 0][n =3D 1][n =3D 7][n =3D 5][n =3D 3][n =3D 1][n =3D = 2][n =3D 0][n =3D 1][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D = 1][n =3D 2][n =3D 0][n =3D 1][n =3D 6][n =3D 4][n =3D 2][n =3D 0][n =3D = 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 5][n =3D 3][n =3D = 1][n =3D 2][n =3D 0][n =3D 1][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D = 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 9][n =3D 7][n =3D 5][n =3D = 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 4][n =3D 2][n =3D 0][n =3D = 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 6][n =3D 4][n =3D = 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D = 5][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 4][n =3D 2][n =3D = 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D 8][n =3D = 6][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D = 0][n =3D 1][n =3D 5][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D = 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D = 1][n =3D 7][n =3D 5][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D 1][n =3D = 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D = 1][n =3D 6][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D = 2][n =3D 0][n =3D 1][n =3D 5][n =3D 3][n =3D 1][n =3D 2][n =3D 0][n =3D = 1][n =3D 4][n =3D 2][n =3D 0][n =3D 1][n =3D 3][n =3D 1][n =3D 2][n =3D = 0][n =3D 1] mirage: main loop exited =3D (1,0) mirage: kernel thread exiting From pgj@caesar.elte.hu Sat Aug 04 11:46:26 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxbsI-0003en-Jp (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 11:46:26 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:52270) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SxbsH-0001DS-sK (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 11:46:26 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1SxbsF-0007Oz-Bv from ; Sat, 04 Aug 2012 12:46:25 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SxbsE-0000oy-LM; Sat, 04 Aug 2012 12:46:22 +0200 Date: Sat, 4 Aug 2012 12:46:22 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120804104622.GA3387@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=AWL, BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 AWL AWL: From: address is in the auto white-list Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 10:46:26 -0000 X-Keywords: X-UID: 437 Status: O Content-Length: 210 Lines: 6 On Fri, Aug 03, 2012 at 06:24:35PM +0100, Anil Madhavapeddy wrote: > Why are you running "ocamlrun ocamlc"? I think this is just a left over from importing the stdlib sources as a package -- not intentional. From pgj@caesar.elte.hu Sat Aug 04 12:02:22 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sxc7i-0003kK-CE (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 12:02:22 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:55256) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1Sxc7h-00041y-re (Exim 4.72) (return-path ); Sat, 04 Aug 2012 12:02:22 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1Sxc7e-0001Sd-3E from ; Sat, 04 Aug 2012 13:02:20 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1Sxc7d-0001FJ-DA; Sat, 04 Aug 2012 13:02:17 +0200 Date: Sat, 4 Aug 2012 13:02:17 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy , Robert Watson Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120804110217.GB3387@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0006] Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 11:02:22 -0000 X-Keywords: X-UID: 438 Status: O Content-Length: 1013 Lines: 21 On Fri, Aug 03, 2012 at 06:30:16PM +0100, Anil Madhavapeddy wrote: > And everything works great after that with your test! Hurrah! I'll play around > with it more over the weekend. Excellent! :-) You must have seen that I also started to import the regression tests from the earlier Mirage repository to test the code better. They can be found under mirage-test/regress, and each of them can be compiled as separate kernel module. And that is where perf/alloc and perf/gcperf grind the machine to a halt: after the first few size tests, the system becomes unresponsive and one can only switch between the virtual terminals, but nothing more. I am also asking Robert: do you have any ideas on how to debug this? In addition, please note that because of the exclusion of the floating-point code, time is measured in microseconds instead of seconds (as reals). This may cause some incompatibility with other platforms. It is also the cause why I could not import the test cases like nbody and mandelbrot. From robert.watson@cl.cam.ac.uk Sat Aug 04 13:17:56 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxdIq-0004G1-0N (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 13:17:56 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:14315) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SxdIo-00063D-Yt (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 13:17:55 +0100 Received: from cilantro.home (pool-96-255-23-153.washdc.fios.verizon.net [96.255.23.153]) by cyrus.watson.org (Postfix) with ESMTPSA id 24B9346B0A; Sat, 4 Aug 2012 08:17:50 -0400 (EDT) Subject: Re: Mirage/kFreeBSD on GitHub Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20120804110217.GB3387@caesar.elte.hu> Date: Sat, 4 Aug 2012 08:17:49 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 12:17:56 -0000 X-Keywords: X-UID: 439 Status: O Content-Length: 2027 Lines: 48 On 4 Aug 2012, at 07:02, PALI Gabor Janos wrote: > And that is where perf/alloc and perf/gcperf grind the machine to a = halt: after > the first few size tests, the system becomes unresponsive and one can = only switch > between the virtual terminals, but nothing more. >=20 > I am also asking Robert: do you have any ideas on how to debug this? Hi Gabor: Sounds like great progress! On debugging: is this a multi-CPU system or a single-CPU system? Are you = able to break into the debugger using the break sequence? Being able switch VTYs means that interrupt delivery is enabled, and = that the kernel thread scheduler is working. Two scenarios come to mind: 1. This is a single-CPU system and something in the kernel (OCaml?) is = running in a loop, preventing userspace from running. If you use a = multi-CPU system you might find userspace is able to run on another CPU, = and a tool like top (using -SH or similar) spots the problem. 2. Something in the kernel is holding a lock and not releasing it, due = to deadlock, a lock leak, etc, and other parts (such as userspace) have = ended up stacked up behind it. This is best debugged using the kernel debugger, especially if there's = a locking issue. Break into the debugger, if possible (likely, as you = can switch VTYs) and use WITNESS's "show alllocks" to see if there's an = obvious candidate for 2. It occurs to me that there is synchronisation around the sysctl syscall = in order to stabilise the sysctl tree while a sysctl is in use. There = are a number of long-running sysctls (e.g., those used in bgfsck) and so = I wouldn't have thought that there were any notable locks that could = stack up behind it. However, it would be something to check. Also, if you're kicking off your code with sysctl, is there some mutual = exclusion or other mechanism to deal with the possibility that more than = once instance of the sysctl might get started at once? (e.g., return = EBUSY in thread two if thread one is already running the OCaml code). Robert= From pgj@caesar.elte.hu Sat Aug 04 18:29:28 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxiAK-0005vo-Gl (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 18:29:28 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:58190) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1SxiAJ-0007fs-ma (Exim 4.72) (return-path ); Sat, 04 Aug 2012 18:29:28 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1SxiAE-0006i8-J8 from ; Sat, 04 Aug 2012 19:29:23 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SxiAD-0003VA-VV; Sat, 04 Aug 2012 19:29:21 +0200 Date: Sat, 4 Aug 2012 19:29:21 +0200 From: PALI Gabor Janos To: "Robert N. M. Watson" Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120804172921.GA3137@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: 0.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=0.0 required=5.9 tests=none autolearn=no SpamAssassin version=3.3.1 Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 17:29:28 -0000 X-Keywords: X-UID: 440 Status: O Content-Length: 2153 Lines: 48 On Sat, Aug 04, 2012 at 08:17:49AM -0400, Robert N. M. Watson wrote: > On debugging: is this a multi-CPU system or a single-CPU system? This is a single-CPU virtual machine. But I can reconfigure it to have 2 processors. > Are you able to break into the debugger using the break sequence? No, I have not tried it yet, but I will. > 1. This is a single-CPU system and something in the kernel (OCaml?) is running in a loop, preventing userspace from running. I tried to solve this problem by launching the OCaml execution on a separate kernel thread. You can find my attempt to do this here: https://github.com/pgj/mirage-kfreebsd/blob/master/packages/mirage-platform/runtime/kernel/kmod.c I set the lowest priority possible to this thread to enable the userspace to continue running while the OCaml program may not finished its execution. And this seems to work well so far. > If you use a multi-CPU system you might find userspace is able to run on another CPU, and a tool like top (using -SH or similar) spots the problem. No, I could not launch top(1) on the other terminal with even 2 processors. However, the situation has improved a little bit as I was at least able to type something and have it processed by the shell. > Break into the debugger, [..] and use WITNESS's "show alllocks" to see if there's an obvious candidate for 2. Okay, thanks for the hint, I will give it a try. > Also, if you're kicking off your code with sysctl, is there some mutual exclusion or other mechanism to deal with the possibility that more than once instance of the sysctl might get started at once? (e.g., return EBUSY in thread two if thread one is already running the OCaml code). Yes, there is a simple variable (mirage_kthread_done) in the file above that flags the thread responsible for running the OCaml execution still working. Note that I have successfully run OCaml programs taking longer time, e.g. calculating the 42nd element of the Fibonacci series, while the rest of the system remain responsive. Thus I am willing to think that this effect is related to the memory allocation and/or the operation of the garbage collector. From pgj@caesar.elte.hu Sat Aug 04 23:48:01 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sxn8b-0007VT-AG (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 04 Aug 2012 23:48:01 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:34557) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1Sxn8a-0003YO-rA (Exim 4.72) (return-path ); Sat, 04 Aug 2012 23:48:01 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1Sxn8X-0003zX-Qa from ; Sun, 05 Aug 2012 00:47:59 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1Sxn8X-0003sU-27; Sun, 05 Aug 2012 00:47:57 +0200 Date: Sun, 5 Aug 2012 00:47:57 +0200 From: PALI Gabor Janos To: "Robert N. M. Watson" Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120804224757.GA11328@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120804172921.GA3137@caesar.elte.hu> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=AWL, BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 AWL AWL: From: address is in the auto white-list Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 22:48:01 -0000 X-Keywords: X-UID: 441 Status: O Content-Length: 2111 Lines: 55 On Sat, Aug 04, 2012 at 07:29:21PM +0200, PALI Gabor Janos wrote: > On Sat, Aug 04, 2012 at 08:17:49AM -0400, Robert N. M. Watson wrote: > > Are you able to break into the debugger using the break sequence? > > No, I have not tried it yet, but I will. Okay, I have managed to break into the debugger and it works. However, the "show alllock" command does not seem to give any useful clue: db> show alllock Process 688 (cron) thread 0xfffffe0005274480 (100050) shared sx vm map (user) (vm map (user)) r = 0 (0xfffffe00052119a8) locked @ /usr/src/sys/vm/vm_fault.c:823 Process 12 (intr) thread 0xfffffe00022d3000 (100015) exclusive sleep mutex Giant (Giant) r = 1 (0xffffffff8126b740) locked @ /usr/src/sys/dev/syscons/syscons.c:723 I found the mirage kernel thread in the meanwhile, though. db> x mirage_kthread mirage_kthread: 5273000 This helped me to find the thread with this address in the output of the "show threads" command (which has become 100096 in my case). Here is the corresponding backtrace for that thread: db> bt 100096 Tracing pid 0 tid 100096 td 0xfffffe0005273000 sched_switch() at sched_switch+0x19f mi_switch() at mi_switch+0x208 sleepq_switch() at sleepq_switch+0xfc sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x3fc kmem_back() at kmem_back+0x1a3 kmem_malloc() at kmem_malloc+0x1f8 uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0x15d caml_aligned_malloc_for_major() at caml_aligned_malloc_for_major+0x2f caml_alloc_for_heap() at caml_alloc_for_heap+0x1e caml_alloc_shr() at caml_alloc_shr+0x10e caml_alloc_string() at caml_alloc_string+0xbd caml_c_call() at caml_c_call+0x20 Perhaps the rest of the kernel is just simply too busy with allocating the requested memory? Note that this test tries to allocate a lot of memory: 1 million instances of strings with the given sizes: first 1, then 500, and 2500 -- that is where the test hangs after around 372,000 allocations. But I suppose the garbage collector should be dropping the unused instances (since they are just allocted, not even used) continuously, freeing up the memory on the other side. From pgj@caesar.elte.hu Sun Aug 05 00:31:03 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxnoF-00009q-J1 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 05 Aug 2012 00:31:03 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:41820) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SxnoE-0004wy-Yg (Exim 4.72) (return-path ); Sun, 05 Aug 2012 00:31:03 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1SxnoC-0006w6-47 from ; Sun, 05 Aug 2012 01:31:01 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SxnoB-00054i-0v; Sun, 05 Aug 2012 01:30:59 +0200 Date: Sun, 5 Aug 2012 01:30:59 +0200 From: PALI Gabor Janos To: "Robert N. M. Watson" Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120804233058.GA18542@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120804224757.GA11328@caesar.elte.hu> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 23:31:03 -0000 X-Keywords: X-UID: 442 Status: O Content-Length: 507 Lines: 16 On Sun, Aug 05, 2012 at 12:47:57AM +0200, PALI Gabor Janos wrote: > Perhaps the rest of the kernel is just simply too busy with allocating the > requested memory? Yeah, technically all the physical memory is consumed by the module at the point of hanging. db> show malloc Type InUse MemUse Requests mirage 967 923430K 976 ... (The machine has 1 GB RAM.) Anil: what should be the expected behavior of the garbage collector? From robert.watson@cl.cam.ac.uk Sun Aug 05 00:44:36 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sxo1M-0000Dh-CR (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 05 Aug 2012 00:44:36 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:34108) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1Sxo1L-0006j2-Yp (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 05 Aug 2012 00:44:36 +0100 Received: from cilantro.home (pool-96-255-23-153.washdc.fios.verizon.net [96.255.23.153]) by cyrus.watson.org (Postfix) with ESMTPSA id D636E46B06; Sat, 4 Aug 2012 19:44:34 -0400 (EDT) Subject: Re: Mirage/kFreeBSD on GitHub Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20120804224757.GA11328@caesar.elte.hu> Date: Sat, 4 Aug 2012 19:44:33 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 23:44:36 -0000 X-Keywords: X-UID: 443 Status: O Content-Length: 1573 Lines: 50 On 4 Aug 2012, at 18:47, PALI Gabor Janos wrote: > This helped me to find the thread with this address in the output of = the > "show threads" command (which has become 100096 in my case). Here is = the > corresponding backtrace for that thread: >=20 > db> bt 100096 > Tracing pid 0 tid 100096 td 0xfffffe0005273000 > sched_switch() at sched_switch+0x19f > mi_switch() at mi_switch+0x208 > sleepq_switch() at sleepq_switch+0xfc > sleepq_wait() at sleepq_wait+0x4d > _sleep() at _sleep+0x3fc > kmem_back() at kmem_back+0x1a3 > kmem_malloc() at kmem_malloc+0x1f8 > uma_large_malloc() at uma_large_malloc+0x4a > malloc() at malloc+0x15d > caml_aligned_malloc_for_major() at caml_aligned_malloc_for_major+0x2f > caml_alloc_for_heap() at caml_alloc_for_heap+0x1e > caml_alloc_shr() at caml_alloc_shr+0x10e > caml_alloc_string() at caml_alloc_string+0xbd > caml_c_call() at caml_c_call+0x20 >=20 >=20 > Perhaps the rest of the kernel is just simply too busy with allocating = the > requested memory? >=20 > Note that this test tries to allocate a lot of memory: 1 million = instances > of strings with the given sizes: first 1, then 500, and 2500 -- that = is > where the test hangs after around 372,000 allocations. But I suppose = the > garbage collector should be dropping the unused instances (since they = are just > allocted, not even used) continuously, freeing up the memory on the = other > side. You can use "show uma" and "show malloc" to show kernel UMA and malloc = statistics, which include both allocation counts and net outstanding = allocations. Robert= From robert.watson@cl.cam.ac.uk Sun Aug 05 00:56:02 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SxoCQ-0000JI-3s (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 05 Aug 2012 00:56:02 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:53365) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SxoCP-0008Pj-Xx (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 05 Aug 2012 00:56:02 +0100 Received: from cilantro.home (pool-96-255-23-153.washdc.fios.verizon.net [96.255.23.153]) by cyrus.watson.org (Postfix) with ESMTPSA id 04A3A46B09; Sat, 4 Aug 2012 19:56:00 -0400 (EDT) Subject: Re: Mirage/kFreeBSD on GitHub Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> Date: Sat, 4 Aug 2012 19:56:00 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2012 23:56:02 -0000 X-Keywords: X-UID: 444 Status: O Content-Length: 645 Lines: 16 On 4 Aug 2012, at 19:44, Robert N. M. Watson wrote: > You can use "show uma" and "show malloc" to show kernel UMA and malloc = statistics, which include both allocation counts and net outstanding = allocations. BTW, I'm not sure what tolerance OCaml has for allocation failures at = lower levels, but you might consider using M_NOWAIT instead of M_WAITOK = and simply returning a failure when memory is exhausted. This doesn't = necessarily help you -- if you are out of memory, not a lot of work can = be done in the system, but at least you could print a message, panic(), = or similar, giving an easier to identify failure mode. Robert= From pgj@caesar.elte.hu Mon Aug 06 16:31:14 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPH0-0004GM-Kl (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:31:14 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:57160) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1SyPGz-00057u-mY (Exim 4.72) (return-path ); Mon, 06 Aug 2012 16:31:14 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1SyPGu-0004MV-VC from ; Mon, 06 Aug 2012 17:31:09 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SyPGu-0004hL-5g; Mon, 06 Aug 2012 17:31:08 +0200 Date: Mon, 6 Aug 2012 17:31:08 +0200 From: PALI Gabor Janos To: "Robert N. M. Watson" Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120806153108.GA4460@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 15:31:14 -0000 X-Keywords: X-UID: 445 Status: O Content-Length: 781 Lines: 17 On Sat, Aug 04, 2012 at 07:56:00PM -0400, Robert N. M. Watson wrote: > I'm not sure what tolerance OCaml has for allocation failures at lower > levels, but you might consider using M_NOWAIT [..] returning a failure > when memory is exhausted. This seems to be a good solution as the OCaml run-time system answers such situations with an Out_of_Memory exception which is now mapped as a panic() in the kernel. By the way, I have found the cause why the garbage collector was not running: it uses floating-point-based calculations to learn whether garbage collection is needed. I just translated the formulas to integers, but there are cases when information is simply lost due to the missing fractional bits. Now it is fixed (hopefully), and the allocation test now passes :-) From anil@recoil.org Mon Aug 06 16:37:54 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPNS-0004PH-RI (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:37:54 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:20521 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1SyPNS-0003iV-q6 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:37:54 +0100 Received: (qmail 9156 invoked by uid 634); 6 Aug 2012 15:37:54 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from dhcp-128-232-129-223.eduroam.csx.cam.ac.uk (HELO dhcp-128-232-129-223.eduroam.csx.cam.ac.uk) (128.232.129.223) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 06 Aug 2012 16:37:53 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Mirage/kFreeBSD on GitHub From: Anil Madhavapeddy In-Reply-To: <20120804233058.GA18542@caesar.elte.hu> Date: Mon, 6 Aug 2012 16:37:51 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <37D02D0E-48EA-4B7F-9E6C-C32958E9703C@recoil.org> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <20120804233058.GA18542@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 15:37:54 -0000 X-Keywords: X-UID: 446 Status: O Content-Length: 1493 Lines: 50 On 5 Aug 2012, at 00:30, PALI Gabor Janos wrote: > On Sun, Aug 05, 2012 at 12:47:57AM +0200, PALI Gabor Janos wrote: >> Perhaps the rest of the kernel is just simply too busy with = allocating the >> requested memory? >=20 > Yeah, technically all the physical memory is consumed by the module at = the > point of hanging. >=20 > db> show malloc > Type InUse MemUse Requests > mirage 967 923430K 976 > ... >=20 > (The machine has 1 GB RAM.) >=20 > Anil: what should be the expected behavior of the garbage collector? >=20 The OCaml GC will malloc() memory to grow heap chunks. It does this in=20= configurable increments [1], specifically via the major_heap_increment value. For a stress test where the working size is small, but the garbage = collected is large (as with the looping-string-alloc test), you want to tweak = these variables such that the GC will run collections much more often (the space_overhead setting), and minimise wastage of memory. This will keep = the working set of allocated memory small, and keep the kernel module = running in a fairly steady state. If the OCaml heap *does* try to grow via a malloc and it fails, there is = no easy recovery, and an Out_of_memory exception will be raised. This is = usually fairly fatal for the process and ought to be avoided. [1] OCAMLRUNPARAM variables in = http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual024.html -a From anil@recoil.org Mon Aug 06 16:42:51 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPSF-0004WX-Cd (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:42:51 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:31463 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1SyPSE-0005TF-s5 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:42:51 +0100 Received: (qmail 2497 invoked by uid 634); 6 Aug 2012 15:42:50 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from dhcp-128-232-129-223.eduroam.csx.cam.ac.uk (HELO dhcp-128-232-129-223.eduroam.csx.cam.ac.uk) (128.232.129.223) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 06 Aug 2012 16:42:50 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Mirage/kFreeBSD on GitHub From: Anil Madhavapeddy In-Reply-To: <20120806153108.GA4460@caesar.elte.hu> Date: Mon, 6 Aug 2012 16:42:48 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> <20120806153108.GA4460@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 15:42:51 -0000 X-Keywords: X-UID: 447 Status: O Content-Length: 1131 Lines: 26 On 6 Aug 2012, at 16:31, PALI Gabor Janos wrote: > On Sat, Aug 04, 2012 at 07:56:00PM -0400, Robert N. M. Watson wrote: >> I'm not sure what tolerance OCaml has for allocation failures at lower >> levels, but you might consider using M_NOWAIT [..] returning a failure >> when memory is exhausted. > > This seems to be a good solution as the OCaml run-time system answers > such situations with an Out_of_Memory exception which is now mapped as > a panic() in the kernel. > > By the way, I have found the cause why the garbage collector was not > running: it uses floating-point-based calculations to learn whether > garbage collection is needed. I just translated the formulas to integers, > but there are cases when information is simply lost due to the missing > fractional bits. > > Now it is fixed (hopefully), and the allocation test now passes :-) Excellent! That may be a patch worth upstreaming, as there are other supported platforms with soft-floats that would benefit from not requiring FP as part of the standard library. Lets separate out all these stdlib modifications at a later stage. -anil From pgj@caesar.elte.hu Mon Aug 06 16:47:00 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPWG-0004bI-8V (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:47:00 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:58645) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1SyPWF-0001rc-oI (Exim 4.72) (return-path ); Mon, 06 Aug 2012 16:47:00 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1SyPWE-0005TR-Ud from ; Mon, 06 Aug 2012 17:46:59 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SyPWE-00056j-EY; Mon, 06 Aug 2012 17:46:58 +0200 Date: Mon, 6 Aug 2012 17:46:58 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120806154658.GA19121@caesar.elte.hu> References: <20120731133034.GB7254@caesar.elte.hu> <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <20120804233058.GA18542@caesar.elte.hu> <37D02D0E-48EA-4B7F-9E6C-C32958E9703C@recoil.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <37D02D0E-48EA-4B7F-9E6C-C32958E9703C@recoil.org> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 15:47:00 -0000 X-Keywords: X-UID: 448 Status: O Content-Length: 458 Lines: 10 On Mon, Aug 06, 2012 at 04:37:51PM +0100, Anil Madhavapeddy wrote: > The OCaml GC will malloc() memory to grow heap chunks. It does this in > configurable increments [1], specifically via the major_heap_increment > value. [..] > > [1] OCAMLRUNPARAM variables in http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual024.html Okay,thanks. Note that the run-time parameters can be now set via the kern.mirage.rtparams sysctl(3) MIB (per your suggestion). From pgj@caesar.elte.hu Mon Aug 06 16:58:21 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPhF-0004oW-BR (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 16:58:21 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:32937) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1SyPhE-0003ck-S3 (Exim 4.72) (return-path ); Mon, 06 Aug 2012 16:58:21 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1SyPhB-000676-Ur from ; Mon, 06 Aug 2012 17:58:19 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1SyPhB-0005Nx-6R; Mon, 06 Aug 2012 17:58:17 +0200 Date: Mon, 6 Aug 2012 17:58:17 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy Subject: Re: Mirage/kFreeBSD on GitHub Message-ID: <20120806155817.GA19641@caesar.elte.hu> References: <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> <20120806153108.GA4460@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=AWL, BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 AWL AWL: From: address is in the auto white-list Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 15:58:21 -0000 X-Keywords: X-UID: 449 Status: O Content-Length: 965 Lines: 17 On Mon, Aug 06, 2012 at 04:42:48PM +0100, Anil Madhavapeddy wrote: > That may be a patch worth upstreaming, as there are other > supported platforms with soft-floats that would benefit from not > requiring FP as part of the standard library. Sure, my modifications in the run-time library are already separated by #ifdef _KERNEL lines. Regarding the stdlib, it is now a standalone OCaml package, but it may be possible to merge it with the upstream sources later on. However, I think that floating-point code is so popular in OCaml libraries because such values are guaranteed to be 64-bit on most platforms, and therefore let the developers play with larger numbers. Note that we also maintain the assumption that the target platform is 64-bit, so these changes will mostly work with processors of that kind. (Like my fix for the garbage collector, where every number is shifted by one billion -- perhaps I could have done it in a more elegant way, though.) From anil@recoil.org Mon Aug 06 17:14:45 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyPx7-00058D-RN (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 17:14:45 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:40597 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1SyPx7-0000bK-qg (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 17:14:45 +0100 Received: (qmail 12219 invoked by uid 634); 6 Aug 2012 16:14:45 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from dhcp-128-232-129-223.eduroam.csx.cam.ac.uk (HELO dhcp-128-232-129-223.eduroam.csx.cam.ac.uk) (128.232.129.223) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 06 Aug 2012 17:14:44 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Mirage/kFreeBSD on GitHub From: Anil Madhavapeddy In-Reply-To: <20120806155817.GA19641@caesar.elte.hu> Date: Mon, 6 Aug 2012 17:14:43 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <1FFB6FDF-9EB8-4431-97C3-FC4FEA005EF4@recoil.org> References: <2ECD7C44-C1EB-4F7D-9B32-514724E4704E@recoil.org> <6F57967E-2BCC-4582-B385-44700B14EEED@recoil.org> <20120804110217.GB3387@caesar.elte.hu> <20120804172921.GA3137@caesar.elte.hu> <20120804224757.GA11328@caesar.elte.hu> <3F56BCDB-9D14-4107-B2D6-4B2B388D2F35@cl.cam.ac.uk> <20120806153108.GA4460@caesar.elte.hu> <20120806155817.GA19641@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 16:14:45 -0000 X-Keywords: X-UID: 450 Status: O Content-Length: 1504 Lines: 40 On 6 Aug 2012, at 16:58, PALI Gabor Janos wrote: > On Mon, Aug 06, 2012 at 04:42:48PM +0100, Anil Madhavapeddy wrote: >> That may be a patch worth upstreaming, as there are other >> supported platforms with soft-floats that would benefit from not >> requiring FP as part of the standard library. >=20 > Sure, my modifications in the run-time library are already separated = by #ifdef > _KERNEL lines. Regarding the stdlib, it is now a standalone OCaml = package, but > it may be possible to merge it with the upstream sources later on. >=20 > However, I think that floating-point code is so popular in OCaml = libraries > because such values are guaranteed to be 64-bit on most platforms, and > therefore let the developers play with larger numbers. Note that we = also > maintain the assumption that the target platform is 64-bit, so these = changes > will mostly work with processors of that kind. (Like my fix for the = garbage > collector, where every number is shifted by one billion -- perhaps I = could > have done it in a more elegant way, though.) The native int is either 31- or 64-bit, but you could always use the = Int64 module (which is a boxed, precisely 64-bit value irrespective of the = target architecture). Unfortunately, the GC module has exported a float-based = interface for over a decade, so I imagine there'll be some resistance to changing = it without a really good reason. Of course, our performance results will = be a good reason, I hope! :-) -anil= From anil@recoil.org Mon Aug 06 23:34:59 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyVt5-0001Gd-5b (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:34:59 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:19104 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1SyVt4-0000vR-RS (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:34:59 +0100 Received: (qmail 10081 invoked by uid 634); 6 Aug 2012 22:34:58 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc7-cmbg14-2-0-cust238.5-4.cable.virginmedia.com (HELO [192.168.1.48]) (86.30.244.239) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 06 Aug 2012 23:34:57 +0100 From: Anil Madhavapeddy Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: new Cohttp interface progress Message-Id: Date: Mon, 6 Aug 2012 23:34:54 +0100 To: "cl-mirage@lists.cam.ac.uk List" , Jon Ludlam Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 22:34:59 -0000 X-Keywords: X-UID: 451 Status: O Content-Length: 1685 Lines: 34 I've been going through the web stack doing releases, and have tagged Re = and Uri versions. Cohttp is a little trickier as the code was a little = messy before, so I've been rewriting it to make it significantly simpler = and easier to embed in libraries. https://github.com/avsm/ocaml-cohttp/tree/v2-interface Instead of depending entirely on Lwt, I'd like to have a version of = Cohttp that can work with alternative threading libraries like Async. So = I've abstracted out the io iteration within the protocol library as a = new IO module which can eventually hide the Lwt type: https://github.com/avsm/ocaml-cohttp/blob/v2-interface/lib/IO.ml The Cohttp.IO.M monad looks like an Lwt monad, but since the interface = is so much smaller, it should be easy to plug in an Async version too. = I've also removed all exceptions from the parsing interface and = propagate options instead. Exception handling is the main difference = between cooperative threading libraries, so putting it directly into the = control flow makes life a lot easier. HTTP Request parsing works now, but I'm still pondering how to handle = HTTP bodies with the IO.M interface. We can't expose an Lwt_stream = directly, so I'll investigate how Async does iteratee IO and make sure = IO.M is compatible.=20 Anyway, the reason this is relevant to the list is because Dave wanted = to use Cohttp, but also because the IO.M technique should be useful for = other protocol implementations that need multiple threading backends = (Xenstore and DNS perhaps). It would be good to figure this out before = we embed Lwt really deeply. I'll do some more work on this tomorrow, = but ideas welcome... -anil= From Richard.Mortier@nottingham.ac.uk Mon Aug 06 23:46:11 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyW3v-0001MH-QR (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:46:11 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:51884 helo=ixe-mta-19.emailfiltering.com) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SyW3q-00059i-XS (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:46:11 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2334443904 ;7dd45020342f5d6a; Mon, 06 Aug 2012 23:45:46 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1SyW3V-0008BS-VN for cl-mirage@lists.cam.ac.uk; Mon, 06 Aug 2012 23:45:45 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Mon, 6 Aug 2012 23:45:34 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Mon, 6 Aug 2012 23:45:34 +0100 Subject: Re: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac10JTDggzqrtGZpSl6V2q6X76qOyQ== Message-ID: References: In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 22:46:11 -0000 X-Keywords: X-UID: 452 Status: O Content-Length: 1750 Lines: 43 On 6 Aug 2012, at 23:34, Anil Madhavapeddy wrote: > ... > Anyway, the reason this is relevant to the list is because Dave wanted to= use Cohttp, but also because the IO.M technique should be useful for other= protocol implementations that need multiple threading backends (Xenstore a= nd DNS perhaps). It would be good to figure this out before we embed Lwt r= eally deeply. I'll do some more work on this tomorrow, but ideas welcome..= . without having read through everything (it's been post-holiday-email all th= e way today - the joys!)... it's generally interesting how we choose to build these interfaces but my i= nitial thought is -- why do you want to support alternative threading libra= ries? elegance, or an actual reason? ;p it does seem generally worthwhile thinking this through though -- eg., i'd = also like to know where we're up to with defining network/channel etc inter= faces of suitable generality... --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Mon Aug 06 23:53:46 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyWBG-0001Nq-3H (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:53:46 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:13253 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1SyWBF-0004EY-Rl (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 06 Aug 2012 23:53:46 +0100 Received: (qmail 25123 invoked by uid 634); 6 Aug 2012 22:53:45 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc7-cmbg14-2-0-cust238.5-4.cable.virginmedia.com (HELO [192.168.1.48]) (86.30.244.239) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 06 Aug 2012 23:53:44 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: new Cohttp interface progress From: Anil Madhavapeddy In-Reply-To: Date: Mon, 6 Aug 2012 23:53:42 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 22:53:46 -0000 X-Keywords: X-UID: 453 Status: O Content-Length: 2021 Lines: 47 On 6 Aug 2012, at 23:45, Richard Mortier = wrote: >=20 > On 6 Aug 2012, at 23:34, Anil Madhavapeddy wrote: >=20 >> ... >> Anyway, the reason this is relevant to the list is because Dave = wanted to use Cohttp, but also because the IO.M technique should be = useful for other protocol implementations that need multiple threading = backends (Xenstore and DNS perhaps). It would be good to figure this = out before we embed Lwt really deeply. I'll do some more work on this = tomorrow, but ideas welcome... >=20 > without having read through everything (it's been post-holiday-email = all the way today - the joys!)... >=20 > it's generally interesting how we choose to build these interfaces but = my initial thought is -- why do you want to support alternative = threading libraries? elegance, or an actual reason? ;p >=20 > it does seem generally worthwhile thinking this through though -- eg., = i'd also like to know where we're up to with defining network/channel = etc interfaces of suitable generality... Good question. It's because each threading library has different = tradeoffs in terms of performance. Lwt has very fairly fine-grained errors, and propagates exceptions as = values (i.e. "fail" vs "raise"). Async has a different mechanism called Monitors, where errors are caught = by a separate function that keeps an eye on a chunk of computation. It = has also deprecated streams, and has Reader/Writer pipes in preference. You could also build a version of IO.M that uses native threads (where = type t would be a dummy type and not actually hold any state). The other *big* thing we would like is that the same protocol library = works on both strings and Io_pages. It should be possible to hide away = the precise choice of what we're writing into behind IO.M. Right now, there are about 5 forks of Cohttp that all implement some of = these variants (Io_page, Net.Channel, Lwt_bytes, Lwt_unix and threads). = It would be quite wise to unify them :-) -anil= From sheets@ashimaarts.com Tue Aug 07 00:12:55 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyWTn-0001js-SA (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 00:12:55 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail.ashimagroup.net ([173.203.58.206]:40441) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1SyWTn-0003ms-mu (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 00:12:55 +0100 Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) (Authenticated sender: sheets@ashimaarts.com) by mail.ashimagroup.net (Postfix) with ESMTP id DC70AC8192 for ; Mon, 6 Aug 2012 23:12:53 +0000 (UTC) Received: by wibhm6 with SMTP id hm6so1820924wib.2 for ; Mon, 06 Aug 2012 16:12:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.180.97.33 with SMTP id dx1mr22102898wib.18.1344294772593; Mon, 06 Aug 2012 16:12:52 -0700 (PDT) Received: by 10.217.3.68 with HTTP; Mon, 6 Aug 2012 16:12:52 -0700 (PDT) In-Reply-To: References: Date: Mon, 6 Aug 2012 16:12:52 -0700 Message-ID: Subject: Re: new Cohttp interface progress From: David Sheets To: Anil Madhavapeddy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Richard Mortier , Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2012 23:12:56 -0000 X-Keywords: X-UID: 454 Status: O Content-Length: 2378 Lines: 57 On Mon, Aug 6, 2012 at 3:53 PM, Anil Madhavapeddy wrote: > On 6 Aug 2012, at 23:45, Richard Mortier wrote: >> >> On 6 Aug 2012, at 23:34, Anil Madhavapeddy wrote: >> >>> ... >>> Anyway, the reason this is relevant to the list is because Dave wanted = to use Cohttp, but also because the IO.M technique should be useful for oth= er protocol implementations that need multiple threading backends (Xenstore= and DNS perhaps). It would be good to figure this out before we embed Lwt= really deeply. I'll do some more work on this tomorrow, but ideas welcome= ... >> >> without having read through everything (it's been post-holiday-email all= the way today - the joys!)... >> >> it's generally interesting how we choose to build these interfaces but m= y initial thought is -- why do you want to support alternative threading li= braries? elegance, or an actual reason? ;p >> >> it does seem generally worthwhile thinking this through though -- eg., i= 'd also like to know where we're up to with defining network/channel etc in= terfaces of suitable generality... > > Good question. It's because each threading library has different tradeoff= s in terms of performance. > > Lwt has very fairly fine-grained errors, and propagates exceptions as val= ues (i.e. "fail" vs "raise"). > > Async has a different mechanism called Monitors, where errors are caught = by a separate function that keeps an eye on a chunk of computation. It has = also deprecated streams, and has Reader/Writer pipes in preference. > > You could also build a version of IO.M that uses native threads (where ty= pe t would be a dummy type and not actually hold any state). > > The other *big* thing we would like is that the same protocol library wor= ks on both strings and Io_pages. It should be possible to hide away the pre= cise choice of what we're writing into behind IO.M. > > Right now, there are about 5 forks of Cohttp that all implement some of t= hese variants (Io_page, Net.Channel, Lwt_bytes, Lwt_unix and threads). It = would be quite wise to unify them :-) A monadic syntax akin to pa_lwt would be appreciated. I am developing an application of cohttp and I would find it quite annoying to have to switch to explicit binding. Is Async's interface fundamentally faster than Lwt's? In their present implementation? David > -anil From anil@recoil.org Tue Aug 07 09:47:06 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyfRS-0001K0-EF (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 09:47:06 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:27067 helo=dark.recoil.org) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with smtp id 1SyfRS-0006sk-mW (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 09:47:06 +0100 Received: (qmail 13449 invoked by uid 634); 7 Aug 2012 08:47:05 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from volstagg-0.srg.cl.cam.ac.uk (HELO [10.0.0.81]) (128.232.32.232) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 07 Aug 2012 09:47:05 +0100 Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: new Cohttp interface progress From: Anil Madhavapeddy In-Reply-To: Date: Tue, 7 Aug 2012 09:47:04 +0100 Content-Transfer-Encoding: 7bit Message-Id: <5D83B21F-71C2-4180-8086-B402BD27E240@recoil.org> References: To: David Sheets X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Richard Mortier , Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 08:47:06 -0000 X-Keywords: X-UID: 455 Status: O Content-Length: 1383 Lines: 30 On 7 Aug 2012, at 00:12, David Sheets wrote: > > A monadic syntax akin to pa_lwt would be appreciated. I am developing > an application of cohttp and I would find it quite annoying to have to > switch to explicit binding. Yeah. There are a few general monad p4 extensions out there, so we can try one out. However, if we make IO.M a functor, then we can simply create an Lwt sub-package that instantiates a complete version of the library that is fully Lwt-based, and can use the usual syntax extension. > > Is Async's interface fundamentally faster than Lwt's? In their present > implementation? Not so much fundamentally faster as different. There is a monitor-based error propagation model, and they seem to make less use of lazy iteratee styles (i.e. Lwt_stream) and more explicit communication (which is easier to reason about in terms of space usage). They have a *really* nice Error.t that uses lazy datastructures to efficiently send error notifications around. OTOH, Lwt has more comprehensive UNIX bindings and is more portable at this stage. I noticed that you've been adding more HTTP header parsing in your fork btw. Is your application in a state where we can merge these branches soon? I need a REST module on top of this protocol library, which it looks like you want as well. Dave, do you need HTTPS for your use of Cohttp? -anil From Dave.Scott@eu.citrix.com Tue Aug 07 10:13:38 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syfr7-0002Ik-Vx (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 10:13:37 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:22527) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1Syfr5-0007KS-qe (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 10:13:37 +0100 X-IronPort-AV: E=Sophos;i="4.77,726,1336348800"; d="scan'208";a="13879573" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 07 Aug 2012 09:13:34 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX01.citrite.net ([10.30.203.162]) with mapi; Tue, 7 Aug 2012 10:13:34 +0100 From: Dave Scott To: 'Anil Madhavapeddy' , David Sheets Date: Tue, 7 Aug 2012 10:13:33 +0100 Subject: RE: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac10eVg/9uBkBS2uQEiztjtsgap0iQAAycCQ Message-ID: <81A73678E76EA642801C8F2E4823AD210107F6B3F415@LONPMAILBOX01.citrite.net> References: <5D83B21F-71C2-4180-8086-B402BD27E240@recoil.org> In-Reply-To: <5D83B21F-71C2-4180-8086-B402BD27E240@recoil.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Richard Mortier , Jonathan Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 09:13:38 -0000 X-Keywords: X-UID: 456 Status: O Content-Length: 316 Lines: 15 Hi, Anil asked: (I assume 'Dave' =3D me in this context) > Dave, do you need HTTPS for your use of Cohttp? No, I'm happy to rely on an external SSL proxy like stunnel for now. Of course if we were to magically gain TLS support natively in Mirage, then= I wouldn't be able to resist using it ;-) Cheers, Dave From Jonathan.Ludlam@eu.citrix.com Tue Aug 07 15:34:59 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syks7-0003Bs-8u (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:34:59 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:1466) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1Syks5-0007rt-qx (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:34:58 +0100 X-IronPort-AV: E=Sophos;i="4.77,727,1336348800"; d="scan'208";a="13888180" Received: from lonpmailmx02.citrite.net ([10.30.203.163]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 07 Aug 2012 14:34:56 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX02.citrite.net ([10.30.203.163]) with mapi; Tue, 7 Aug 2012 15:34:56 +0100 From: Jonathan Ludlam To: "cl-mirage@lists.cam.ac.uk" Date: Tue, 7 Aug 2012 15:34:58 +0100 Subject: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac10qdBXgf6s3D5pQZa9Zd/KgluYCQ== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 14:34:59 -0000 X-Keywords: X-UID: 457 Status: O Content-Length: 403 Lines: 15 Hi all, I've been trying to follow the instructions on the mirage wiki to install o= pam, but have had little success so far. I'm installing on a fresh 12.04 Ub= untu system, and the problem I'm currently getting is in installing re 1.0 = - it's trying to create the directory /usr/local/share/doc. Has anyone had = this problem, or some insight into how to fix it? Cheers, Jon Sent from my iPad= From Richard.Mortier@nottingham.ac.uk Tue Aug 07 15:46:12 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syl2y-0003YX-Jb (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:46:12 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:59863 helo=ixe-mta-19.emailfiltering.com) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1Syl2t-0006KE-Wq (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:46:12 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2335207840 ;c331772b0038eff4; Tue, 07 Aug 2012 15:45:47 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1Syl2Y-0006bK-Rs for cl-mirage@lists.cam.ac.uk; Tue, 07 Aug 2012 15:45:46 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Tue, 7 Aug 2012 15:45:46 +0100 From: Richard Mortier To: Jonathan Ludlam Date: Tue, 7 Aug 2012 15:45:45 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac10q1PbgX3Z/05aTumc4VuE1wPlig== Message-ID: <7D9011CB-5CBE-41DB-BDCE-1B2FC3D4C1A4@nottingham.ac.uk> References: In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 14:46:12 -0000 X-Keywords: X-UID: 458 Status: O Content-Length: 1512 Lines: 38 On 7 Aug 2012, at 15:34, Jonathan Ludlam wrote: > Hi all, >=20 > I've been trying to follow the instructions on the mirage wiki to install= opam, but have had little success so far. I'm installing on a fresh 12.04 = Ubuntu system, and the problem I'm currently getting is in installing re 1.= 0 - it's trying to create the directory /usr/local/share/doc. Has anyone ha= d this problem, or some insight into how to fix it? not had that specifically, but as is usual for an osx/homebrew user, /usr/l= ocal is writeable by me (ie., no root privs required) -- possibly this assu= mption has crept in although it doesn't usually hold on linux systems? (more a comment than any help i'm afraid :) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From thomas.gazagnaire@gmail.com Tue Aug 07 15:52:46 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syl9J-0003jJ-V2 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:52:45 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [74.125.82.179 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-we0-f179.google.com ([74.125.82.179]:41150) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1Syl9I-0005kV-rp (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 15:52:45 +0100 Received: by weyx10 with SMTP id x10so2580315wey.38 for ; Tue, 07 Aug 2012 07:52:44 -0700 (PDT) Received: by 10.216.101.72 with SMTP id a50mr7791651weg.210.1344351164353; Tue, 07 Aug 2012 07:52:44 -0700 (PDT) Received: from saorge.home (ANice-551-1-365-7.w83-197.abo.wanadoo.fr. [83.197.205.7]) by mx.google.com with ESMTPS id t7sm31964001wix.6.2012.08.07.07.52.40 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 07 Aug 2012 07:52:42 -0700 (PDT) Sender: Thomas Gazagnaire Subject: Re: Opam installation woes Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: Date: Tue, 7 Aug 2012 16:52:38 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> References: To: Jonathan Ludlam X-Mailer: Apple Mail (2.1084) Cc: "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 14:52:46 -0000 X-Keywords: X-UID: 459 Status: O Content-Length: 682 Lines: 27 I guess the re package is missing the right configure invocation. Anil, = I think you forgot to add something like "./configure" "--disable-doc" = "--prefix" "%{prefix}%"] to build. -- Thomas On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: > Hi all, >=20 > I've been trying to follow the instructions on the mirage wiki to = install opam, but have had little success so far. I'm installing on a = fresh 12.04 Ubuntu system, and the problem I'm currently getting is in = installing re 1.0 - it's trying to create the directory = /usr/local/share/doc. Has anyone had this problem, or some insight into = how to fix it? >=20 > Cheers, >=20 > Jon >=20 >=20 > Sent from my iPad From Jonathan.Ludlam@eu.citrix.com Tue Aug 07 16:04:48 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SylKy-00049P-3o (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 16:04:48 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:55852) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SylKv-0000pJ-ra (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 16:04:48 +0100 X-IronPort-AV: E=Sophos;i="4.77,727,1336348800"; d="scan'208";a="13888882" Received: from lonpmailmx02.citrite.net ([10.30.203.163]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 07 Aug 2012 15:04:44 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX02.citrite.net ([10.30.203.163]) with mapi; Tue, 7 Aug 2012 16:04:44 +0100 From: Jonathan Ludlam To: Thomas Gazagnaire Date: Tue, 7 Aug 2012 16:04:45 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac10rfncyLnJiOW+SL2mtRCDs9KJng== Message-ID: <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> In-Reply-To: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 15:04:48 -0000 X-Keywords: X-UID: 460 Status: O Content-Length: 986 Lines: 36 Chowning /usr/local/share/doc to me now gets me past that. However, looks l= ike the uri build suffers the same problem, and it's also missing an oUnit = dependency. Jon Sent from my iPad On 7 Aug 2012, at 15:52, "Thomas Gazagnaire" wrote: > I guess the re package is missing the right configure invocation. Anil, I= think you forgot to add something like "./configure" "--disable-doc" "--pr= efix" "%{prefix}%"] to build. >=20 > -- > Thomas >=20 >=20 > On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: >=20 >> Hi all, >>=20 >> I've been trying to follow the instructions on the mirage wiki to instal= l opam, but have had little success so far. I'm installing on a fresh 12.04= Ubuntu system, and the problem I'm currently getting is in installing re 1= .0 - it's trying to create the directory /usr/local/share/doc. Has anyone h= ad this problem, or some insight into how to fix it? >>=20 >> Cheers, >>=20 >> Jon >>=20 >>=20 >> Sent from my iPad >=20 From Jonathan.Ludlam@eu.citrix.com Tue Aug 07 17:01:55 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SymEF-00063x-8j (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:01:55 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:40538) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SymEC-0006AN-YN (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:01:55 +0100 X-IronPort-AV: E=Sophos;i="4.77,727,1336348800"; d="scan'208";a="13890362" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 07 Aug 2012 16:01:51 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX01.citrite.net ([10.30.203.162]) with mapi; Tue, 7 Aug 2012 17:01:52 +0100 From: Jonathan Ludlam To: Thomas Gazagnaire Date: Tue, 7 Aug 2012 17:01:53 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac10tfTNxb/Ldb6AQJSbN148e24Wtw== Message-ID: <259F05C4-B239-4B9E-B32F-1D26D1170BC9@eu.citrix.com> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> In-Reply-To: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 16:01:55 -0000 X-Keywords: X-UID: 461 Status: O Content-Length: 1140 Lines: 41 Ok, next problem. On switching to the xen compiler, I'm getting a problem b= uilding ocamlfind: It's complaining about a missing separator in Makefile.c= onfig, but it's deleting it before I get a chance to see what's actually wr= ong. Is there any way to prevent opam from cleaning up after a failed build= ? Cheers, Jon Sent from my iPad On 7 Aug 2012, at 15:52, "Thomas Gazagnaire" wrote: > I guess the re package is missing the right configure invocation. Anil, I= think you forgot to add something like "./configure" "--disable-doc" "--pr= efix" "%{prefix}%"] to build. >=20 > -- > Thomas >=20 >=20 > On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: >=20 >> Hi all, >>=20 >> I've been trying to follow the instructions on the mirage wiki to instal= l opam, but have had little success so far. I'm installing on a fresh 12.04= Ubuntu system, and the problem I'm currently getting is in installing re 1= .0 - it's trying to create the directory /usr/local/share/doc. Has anyone h= ad this problem, or some insight into how to fix it? >>=20 >> Cheers, >>=20 >> Jon >>=20 >>=20 >> Sent from my iPad >=20 From anil@recoil.org Tue Aug 07 17:04:54 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SymH8-000665-2g (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:04:54 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:18134 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1SymH6-0006ql-Yt (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:04:53 +0100 Received: (qmail 12852 invoked by uid 634); 7 Aug 2012 16:04:52 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from volstagg-0.srg.cl.cam.ac.uk (HELO [10.0.0.81]) (128.232.32.232) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 07 Aug 2012 17:04:51 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Opam installation woes From: Anil Madhavapeddy In-Reply-To: <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> Date: Tue, 7 Aug 2012 17:04:49 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> To: Jonathan Ludlam X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Thomas Gazagnaire , "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 16:04:54 -0000 X-Keywords: X-UID: 462 Status: O Content-Length: 2099 Lines: 69 Yes, I've only been using it on MacOS X so far, where the /usr/local on = Homebrew is indeed user-writeable. Jon, any chance you could get a local install of OPAM running to fix = this, as I don't have a Linux box handy until Friday. The procedure is: $ $ rm -rf ~/.opam $ cd your/opam-repo $ opam init -kind rsync default . At this point, you can make changes in opam-repo, run "opam update" and = see them reflected immediately. $ $ We really need a Jenkins setup to try all the combination of compilers = and backends, as the Xen one has no doubt had some minor bitrot in the = past few changesets. To maintain your Xen compiler, I think you pass = "opam --debug" to preserve the state. You can modify the compiler = descriptions (in opam-repo/compiler) in the same way. -anil(packing to leave) On 7 Aug 2012, at 16:04, Jonathan Ludlam = wrote: > Chowning /usr/local/share/doc to me now gets me past that. However, = looks like the uri build suffers the same problem, and it's also missing = an oUnit dependency. >=20 > Jon >=20 > Sent from my iPad >=20 > On 7 Aug 2012, at 15:52, "Thomas Gazagnaire" = wrote: >=20 >> I guess the re package is missing the right configure invocation. = Anil, I think you forgot to add something like "./configure" = "--disable-doc" "--prefix" "%{prefix}%"] to build. >>=20 >> -- >> Thomas >>=20 >>=20 >> On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: >>=20 >>> Hi all, >>>=20 >>> I've been trying to follow the instructions on the mirage wiki to = install opam, but have had little success so far. I'm installing on a = fresh 12.04 Ubuntu system, and the problem I'm currently getting is in = installing re 1.0 - it's trying to create the directory = /usr/local/share/doc. Has anyone had this problem, or some insight into = how to fix it? >>>=20 >>> Cheers, >>>=20 >>> Jon >>>=20 >>>=20 >>> Sent from my iPad >>=20 >=20 From Jonathan.Ludlam@eu.citrix.com Tue Aug 07 17:37:03 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SymmF-0006pm-KK (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:37:03 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:23500) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1SymmE-0003JL-Sc (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 17:37:03 +0100 X-IronPort-AV: E=Sophos;i="4.77,728,1336348800"; d="scan'208";a="13891025" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 07 Aug 2012 16:37:02 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX01.citrite.net ([10.30.203.162]) with mapi; Tue, 7 Aug 2012 17:37:02 +0100 From: Jonathan Ludlam To: Anil Madhavapeddy Date: Tue, 7 Aug 2012 17:37:04 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac10ut6rlNXYij6PSkmhV6gOA4qsJQ== Message-ID: References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> In-Reply-To: <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Gazagnaire , "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 16:37:03 -0000 X-Keywords: X-UID: 463 Status: O Content-Length: 2876 Lines: 86 Thanks Anil, I'll have a play. Turns out the problem with ocamlfind is that ocamlfind.1.3.3/configure is t= rying to detect the system by doing a 'ocamlc -config | grep system', expec= ting to find one line. Unfortunately the standard library is installed in ~= /.opam/system/lib, and similarly the runtime, so it ends up spitting out se= veral lines into the Makefile.config. Not quite sure of the best way to fix this. I could patch the configure fil= e, pass extra arguments to the configure invocation or change the name of t= he directory in which the ocaml compilers are found, I guess. Jon Sent from my iPad On 7 Aug 2012, at 17:04, "Anil Madhavapeddy" wrote: > Yes, I've only been using it on MacOS X so far, where the /usr/local on H= omebrew is indeed user-writeable. >=20 > Jon, any chance you could get a local install of OPAM running to fix this= , as I don't have a Linux box handy until Friday. The procedure is: >=20 > $ > $ rm -rf ~/.opam > $ cd your/opam-repo > $ opam init -kind rsync default . >=20 > At this point, you can make changes in opam-repo, run "opam update" and s= ee them reflected immediately. >=20 > $ > $ >=20 > We really need a Jenkins setup to try all the combination of compilers an= d backends, as the Xen one has no doubt had some minor bitrot in the past f= ew changesets. To maintain your Xen compiler, I think you pass "opam --deb= ug" to preserve the state. You can modify the compiler descriptions (in op= am-repo/compiler) in the same way. >=20 > -anil(packing to leave) >=20 > On 7 Aug 2012, at 16:04, Jonathan Ludlam = wrote: >=20 >> Chowning /usr/local/share/doc to me now gets me past that. However, look= s like the uri build suffers the same problem, and it's also missing an oUn= it dependency. >>=20 >> Jon >>=20 >> Sent from my iPad >>=20 >> On 7 Aug 2012, at 15:52, "Thomas Gazagnaire" wro= te: >>=20 >>> I guess the re package is missing the right configure invocation. Anil,= I think you forgot to add something like "./configure" "--disable-doc" "--= prefix" "%{prefix}%"] to build. >>>=20 >>> -- >>> Thomas >>>=20 >>>=20 >>> On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: >>>=20 >>>> Hi all, >>>>=20 >>>> I've been trying to follow the instructions on the mirage wiki to inst= all opam, but have had little success so far. I'm installing on a fresh 12.= 04 Ubuntu system, and the problem I'm currently getting is in installing re= 1.0 - it's trying to create the directory /usr/local/share/doc. Has anyone= had this problem, or some insight into how to fix it? >>>>=20 >>>> Cheers, >>>>=20 >>>> Jon >>>>=20 >>>>=20 >>>> Sent from my iPad >>>=20 >>=20 >=20 From anil@recoil.org Tue Aug 07 19:43:14 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SyokM-0000cY-O0 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 19:43:14 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:38497 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1SyokL-0004BB-XY (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 19:43:14 +0100 Received: (qmail 25202 invoked by uid 634); 7 Aug 2012 18:43:13 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc7-cmbg14-2-0-cust238.5-4.cable.virginmedia.com (HELO [192.168.1.64]) (86.30.244.239) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 07 Aug 2012 19:43:12 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Opam installation woes From: Anil Madhavapeddy In-Reply-To: Date: Tue, 7 Aug 2012 19:43:11 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <4DB36A8A-F259-4DBD-AAA8-70054586C46F@recoil.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> To: Jonathan Ludlam X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Thomas Gazagnaire , "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 18:43:14 -0000 X-Keywords: X-UID: 464 Status: O Content-Length: 3367 Lines: 102 Hah, that's a new one. I've mailed Gerd to patch the configure script = to be more specific. =20 In the meanwhile, you could pass an explicit "-system xxx" in the = ocamlfind-1.3.3.opam script to force it to a UNIX mode (according to the = configure script, any unknown system will be treated as UNIX). -anil On 7 Aug 2012, at 17:37, Jonathan Ludlam = wrote: > Thanks Anil, I'll have a play. >=20 > Turns out the problem with ocamlfind is that ocamlfind.1.3.3/configure = is trying to detect the system by doing a 'ocamlc -config | grep = system', expecting to find one line. Unfortunately the standard library = is installed in ~/.opam/system/lib, and similarly the runtime, so it = ends up spitting out several lines into the Makefile.config. >=20 > Not quite sure of the best way to fix this. I could patch the = configure file, pass extra arguments to the configure invocation or = change the name of the directory in which the ocaml compilers are found, = I guess. >=20 > Jon >=20 > Sent from my iPad >=20 > On 7 Aug 2012, at 17:04, "Anil Madhavapeddy" wrote: >=20 >> Yes, I've only been using it on MacOS X so far, where the /usr/local = on Homebrew is indeed user-writeable. >>=20 >> Jon, any chance you could get a local install of OPAM running to fix = this, as I don't have a Linux box handy until Friday. The procedure is: >>=20 >> $ >> $ rm -rf ~/.opam >> $ cd your/opam-repo >> $ opam init -kind rsync default . >>=20 >> At this point, you can make changes in opam-repo, run "opam update" = and see them reflected immediately. >>=20 >> $ >> $ >>=20 >> We really need a Jenkins setup to try all the combination of = compilers and backends, as the Xen one has no doubt had some minor = bitrot in the past few changesets. To maintain your Xen compiler, I = think you pass "opam --debug" to preserve the state. You can modify the = compiler descriptions (in opam-repo/compiler) in the same way. >>=20 >> -anil(packing to leave) >>=20 >> On 7 Aug 2012, at 16:04, Jonathan Ludlam = wrote: >>=20 >>> Chowning /usr/local/share/doc to me now gets me past that. However, = looks like the uri build suffers the same problem, and it's also missing = an oUnit dependency. >>>=20 >>> Jon >>>=20 >>> Sent from my iPad >>>=20 >>> On 7 Aug 2012, at 15:52, "Thomas Gazagnaire" = wrote: >>>=20 >>>> I guess the re package is missing the right configure invocation. = Anil, I think you forgot to add something like "./configure" = "--disable-doc" "--prefix" "%{prefix}%"] to build. >>>>=20 >>>> -- >>>> Thomas >>>>=20 >>>>=20 >>>> On Aug 7, 2012, at 4:34 PM, Jonathan Ludlam wrote: >>>>=20 >>>>> Hi all, >>>>>=20 >>>>> I've been trying to follow the instructions on the mirage wiki to = install opam, but have had little success so far. I'm installing on a = fresh 12.04 Ubuntu system, and the problem I'm currently getting is in = installing re 1.0 - it's trying to create the directory = /usr/local/share/doc. Has anyone had this problem, or some insight into = how to fix it? >>>>>=20 >>>>> Cheers, >>>>>=20 >>>>> Jon >>>>>=20 >>>>>=20 >>>>> Sent from my iPad >>>>=20 >>>=20 >>=20 >=20 From Richard.Mortier@nottingham.ac.uk Tue Aug 07 21:55:49 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syqof-0002BG-Fn (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 21:55:49 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from thb-mta-19-tx.emailfiltering.com ([194.116.199.150]:52791 helo=thb-mta-19.emailfiltering.com) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1Syqoa-0007cZ-WG (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 21:55:49 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by thb-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2521409698 ;70d61fd69cf07220; Tue, 07 Aug 2012 21:55:23 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1SyqoF-0005fA-H3 for cl-mirage@lists.cam.ac.uk; Tue, 07 Aug 2012 21:55:23 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Tue, 7 Aug 2012 21:55:23 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Tue, 7 Aug 2012 21:55:24 +0100 Subject: Re: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac103vZ+5V2n2KCjTrGVKof/M41Hwg== Message-ID: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> References: In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 20:55:49 -0000 X-Keywords: X-UID: 465 Status: O Content-Length: 3538 Lines: 83 On 6 Aug 2012, at 23:53, Anil Madhavapeddy wrote: > On 6 Aug 2012, at 23:45, Richard Mortier wrote: >>=20 >> On 6 Aug 2012, at 23:34, Anil Madhavapeddy wrote: >>=20 >>> ... >>> Anyway, the reason this is relevant to the list is because Dave wanted = to use Cohttp, but also because the IO.M technique should be useful for oth= er protocol implementations that need multiple threading backends (Xenstore= and DNS perhaps). It would be good to figure this out before we embed Lwt= really deeply. I'll do some more work on this tomorrow, but ideas welcome= ... >>=20 >> without having read through everything (it's been post-holiday-email all= the way today - the joys!)... >>=20 >> it's generally interesting how we choose to build these interfaces but m= y initial thought is -- why do you want to support alternative threading li= braries? elegance, or an actual reason? ;p >>=20 >> it does seem generally worthwhile thinking this through though -- eg., i= 'd also like to know where we're up to with defining network/channel etc in= terfaces of suitable generality... >=20 > Good question. It's because each threading library has different tradeoff= s in terms of performance. hm- do you have pointers to any examination of this, or is it just well kno= wn? > Lwt has very fairly fine-grained errors, and propagates exceptions as val= ues (i.e. "fail" vs "raise"). >=20 > Async has a different mechanism called Monitors, where errors are caught = by a separate function that keeps an eye on a chunk of computation. It has = also deprecated streams, and has Reader/Writer pipes in preference. in some ways the different ways they do things like error handling is (to m= y mind) a more interesting distinction :) (unless the perf differences are= genuinely significant and different in multiple real applications.) > You could also build a version of IO.M that uses native threads (where ty= pe t would be a dummy type and not actually hold any state). >=20 > The other *big* thing we would like is that the same protocol library wor= ks on both strings and Io_pages. It should be possible to hide away the pre= cise choice of what we're writing into behind IO.M. by strings do you mean bigarray thingies? if so then i buy that - it would= be nice to have a single encapsulated layer that meant that all the mirage= protocol libraries could take a single dependency on it and be usable from= any other ocaml program. > Right now, there are about 5 forks of Cohttp that all implement some of t= hese variants (Io_page, Net.Channel, Lwt_bytes, Lwt_unix and threads). It = would be quite wise to unify them :-) i guess the question i'm asking is - unify, or cull? :) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From Richard.Mortier@nottingham.ac.uk Tue Aug 07 23:34:52 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SysMW-0002wS-HP (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:34:52 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:48587 helo=ixe-mta-19.emailfiltering.com) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1SysMR-0004WF-QB (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:34:52 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2335523260 ;bc0a73e6fb88185c; Tue, 07 Aug 2012 23:34:27 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1SysM6-0007o5-PP for cl-mirage@lists.cam.ac.uk; Tue, 07 Aug 2012 23:34:26 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Tue, 7 Aug 2012 23:34:07 +0100 From: Richard Mortier To: Anil Madhavapeddy , Jonathan Ludlam Date: Tue, 7 Aug 2012 23:34:06 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac107MEQMGFymji5QVm/aAGyI6D2mg== Message-ID: <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> In-Reply-To: <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Gazagnaire , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 22:34:52 -0000 X-Keywords: X-UID: 466 Status: O Content-Length: 1558 Lines: 43 On 7 Aug 2012, at 17:04, Anil Madhavapeddy wrote: > ...Jon, any chance you could get a local install of OPAM running to fix t= his, as I don't have a Linux box handy until Friday. The procedure is: >=20 > $ > $ rm -rf ~/.opam > $ cd your/opam-repo > $ opam init -kind rsync default . >=20 > At this point, you can make changes in opam-repo, run "opam update" and s= ee them reflected immediately. >=20 > $ > $ (fwiw i've just done this in my fork of opam-repo-dev, and sent a pull req.= jon, by all means give this a go if you haven't fixed it yourself already= ...) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Tue Aug 07 23:46:43 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SysXz-000399-Ic (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:46:43 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:18953 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1SysXy-0006KQ-Xm (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:46:43 +0100 Received: (qmail 6280 invoked by uid 634); 7 Aug 2012 22:46:41 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc7-cmbg14-2-0-cust238.5-4.cable.virginmedia.com (HELO [192.168.1.38]) (86.30.244.239) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 07 Aug 2012 23:46:41 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Opam installation woes From: Anil Madhavapeddy In-Reply-To: <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> Date: Tue, 7 Aug 2012 23:46:40 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Thomas Gazagnaire , Jonathan Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 22:46:43 -0000 X-Keywords: X-UID: 467 Status: O Content-Length: 1355 Lines: 38 On 7 Aug 2012, at 23:34, Richard Mortier = wrote: >=20 > On 7 Aug 2012, at 17:04, Anil Madhavapeddy wrote: >=20 >> ...Jon, any chance you could get a local install of OPAM running to = fix this, as I don't have a Linux box handy until Friday. The procedure = is: >>=20 >> $ >> $ rm -rf ~/.opam >> $ cd your/opam-repo >> $ opam init -kind rsync default . >>=20 >> At this point, you can make changes in opam-repo, run "opam update" = and see them reflected immediately. >>=20 >> $ >> $ >=20 > (fwiw i've just done this in my fork of opam-repo-dev, and sent a pull = req. jon, by all means give this a go if you haven't fixed it yourself = already...) All merged to opam-repo-dev. Btw, I don't seem to get notification of = pull requests to Mirage organisation repos, so feel free to push = directly to mirage/opam-repo-dev if it fixes your problem. I think it's high time we setup a little continuous build VM. Thomas, = is there an easy way to iterate through every package and attempt a = compilation? I could do a big 'opam install `opam list`'-style command, = but something that has reasonable logging for failures would be useful. -anil From Richard.Mortier@nottingham.ac.uk Tue Aug 07 23:49:56 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sysb6-0003Ay-0H (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:49:56 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from thb-mta-19-tx.emailfiltering.com ([194.116.199.150]:41574 helo=thb-mta-19.emailfiltering.com) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1Sysb0-0006hh-Y2 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:49:55 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by thb-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2521456578 ;513674eedc070cb4; Tue, 07 Aug 2012 23:49:30 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1Sysag-000844-5d for cl-mirage@lists.cam.ac.uk; Tue, 07 Aug 2012 23:49:30 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Tue, 7 Aug 2012 23:49:30 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Tue, 7 Aug 2012 23:49:29 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac107uddUpawR9WzQKupdlL0JRG0rA== Message-ID: References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> In-Reply-To: <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Gazagnaire , Jonathan Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 22:49:56 -0000 X-Keywords: X-UID: 468 Status: O Content-Length: 1742 Lines: 47 On 7 Aug 2012, at 23:46, Anil Madhavapeddy wrote: > On 7 Aug 2012, at 23:34, Richard Mortier wrote: >=20 >> (fwiw i've just done this in my fork of opam-repo-dev, and sent a pull r= eq. jon, by all means give this a go if you haven't fixed it yourself alre= ady...) >=20 > All merged to opam-repo-dev. Btw, I don't seem to get notification of pul= l requests to Mirage organisation repos, so feel free to push directly to m= irage/opam-repo-dev if it fixes your problem. ok, will do. > I think it's high time we setup a little continuous build VM. Thomas, is= there an easy way to iterate through every package and attempt a compilati= on? I could do a big 'opam install `opam list`'-style command, but somethin= g that has reasonable logging for failures would be useful. what would you do about package interdependencies in the case of failure? (sounds like a job for tsort ;) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Tue Aug 07 23:50:43 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sysbr-0003BZ-Id (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:50:43 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:7598 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1Sysbr-0007Dg-WU (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 07 Aug 2012 23:50:43 +0100 Received: (qmail 26949 invoked by uid 634); 7 Aug 2012 22:50:42 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from cpc7-cmbg14-2-0-cust238.5-4.cable.virginmedia.com (HELO [192.168.1.38]) (86.30.244.239) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 07 Aug 2012 23:50:42 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Opam installation woes From: Anil Madhavapeddy In-Reply-To: Date: Tue, 7 Aug 2012 23:50:41 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Thomas Gazagnaire , Jonathan Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2012 22:50:43 -0000 X-Keywords: X-UID: 469 Status: O Content-Length: 1132 Lines: 37 On 7 Aug 2012, at 23:49, Richard Mortier = wrote: >=20 > On 7 Aug 2012, at 23:46, Anil Madhavapeddy wrote: >=20 >> On 7 Aug 2012, at 23:34, Richard Mortier = wrote: >>=20 >>> (fwiw i've just done this in my fork of opam-repo-dev, and sent a = pull req. jon, by all means give this a go if you haven't fixed it = yourself already...) >>=20 >> All merged to opam-repo-dev. Btw, I don't seem to get notification of = pull requests to Mirage organisation repos, so feel free to push = directly to mirage/opam-repo-dev if it fixes your problem. >=20 > ok, will do. >=20 >> I think it's high time we setup a little continuous build VM. = Thomas, is there an easy way to iterate through every package and = attempt a compilation? I could do a big 'opam install `opam list`'-style = command, but something that has reasonable logging for failures would be = useful. >=20 >=20 > what would you do about package interdependencies in the case of = failure? >=20 > (sounds like a job for tsort ;) there's a SAT solver in OPAM for dependencies. Not my problem ;-) -a From Jonathan.Ludlam@eu.citrix.com Wed Aug 08 02:02:03 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Syuex-0004M6-KD (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 02:02:03 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:38933) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1Syuew-0005Ua-ZR (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 02:02:03 +0100 X-IronPort-AV: E=Sophos;i="4.77,730,1336348800"; d="scan'208";a="13898491" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 08 Aug 2012 01:02:02 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX01.citrite.net ([10.30.203.162]) with mapi; Wed, 8 Aug 2012 02:02:02 +0100 From: Jonathan Ludlam To: Anil Madhavapeddy Date: Wed, 8 Aug 2012 02:01:58 +0100 Subject: Re: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac11AWrEqNYak9jETM20RWcZJbYXGw== Message-ID: <1A061F11-2795-47D3-A415-1545F71FDA8E@eu.citrix.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 01:02:03 -0000 X-Keywords: X-UID: 470 Status: O Content-Length: 428 Lines: 17 On 6 Aug 2012, at 23:53, Anil Madhavapeddy wrote: >=20 > You could also build a version of IO.M that uses native threads (where ty= pe t would be a dummy type and not actually hold any state). >=20 I bet this would introduce all sorts of problems because you no longer can = assume that a block of code with no IO.bind in will run atomically. I'm sur= e I've seen a few places in the code that make that assumption. Jon From Jonathan.Ludlam@eu.citrix.com Wed Aug 08 08:51:45 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz13R-0001GR-I6 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 08:51:45 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.4 from SpamAssassin-3.3.2-1358082 * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:29714) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1Sz13Q-0001jA-pC (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 08:51:45 +0100 X-IronPort-AV: E=Sophos;i="4.77,730,1336348800"; d="scan'208";a="13901981" Received: from lonpmailmx01.citrite.net ([10.30.203.162]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 08 Aug 2012 07:51:44 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX01.citrite.net ([10.30.203.162]) with mapi; Wed, 8 Aug 2012 08:51:44 +0100 From: Jonathan Ludlam To: Anil Madhavapeddy Date: Wed, 8 Aug 2012 08:51:39 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac11OqbYfhDZ5bEWSa2rVh0MrrA/1w== Message-ID: References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> In-Reply-To: <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Gazagnaire , Richard Mortier , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 07:51:45 -0000 X-Keywords: X-UID: 471 Status: O Content-Length: 1901 Lines: 54 Hmm. Going back to the opam woes, I'm once more having issues - now with th= e standard 'inconsistent assumptions' errors and the like. I suspect this i= s because things are being installed in the wrong places. The problem I had earlier was that after the switch to the xen compiler, ~/= .opam/system/... was being reported by ocamlc. Unless I'm misunderstanding= something, shouldn't the ocamlc binaries be pointing at ~/.opam/xen/... in= stead? It looks to me a lot like stuff was being installed into ~/.opam/sys= tem when it should have been going into ~/.opam/xen - specifically the ocam= lc binaries ended up in .opam/system/bin. Is that expected behaviour? Jon On 7 Aug 2012, at 23:50, Anil Madhavapeddy wrote: >=20 > On 7 Aug 2012, at 23:49, Richard Mortier wrote: >=20 >>=20 >> On 7 Aug 2012, at 23:46, Anil Madhavapeddy wrote: >>=20 >>> On 7 Aug 2012, at 23:34, Richard Mortier wrote: >>>=20 >>>> (fwiw i've just done this in my fork of opam-repo-dev, and sent a pull= req. jon, by all means give this a go if you haven't fixed it yourself al= ready...) >>>=20 >>> All merged to opam-repo-dev. Btw, I don't seem to get notification of p= ull requests to Mirage organisation repos, so feel free to push directly to= mirage/opam-repo-dev if it fixes your problem. >>=20 >> ok, will do. >>=20 >>> I think it's high time we setup a little continuous build VM. Thomas, = is there an easy way to iterate through every package and attempt a compila= tion? I could do a big 'opam install `opam list`'-style command, but someth= ing that has reasonable logging for failures would be useful. >>=20 >>=20 >> what would you do about package interdependencies in the case of failure= ? >>=20 >> (sounds like a job for tsort ;) >=20 > there's a SAT solver in OPAM for dependencies. Not my problem ;-) >=20 > -a >=20 From Richard.Mortier@nottingham.ac.uk Wed Aug 08 08:57:46 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz19G-0001MP-2N (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 08:57:46 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:37700 helo=ixe-mta-19.emailfiltering.com) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1Sz19A-0005jD-Po (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 08:57:45 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2335763076 ;bd06492a33e98186; Wed, 08 Aug 2012 08:57:19 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp4.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1Sz18p-0000QJ-L5 for cl-mirage@lists.cam.ac.uk; Wed, 08 Aug 2012 08:57:19 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Wed, 8 Aug 2012 08:57:19 +0100 From: Richard Mortier To: Jonathan Ludlam Date: Wed, 8 Aug 2012 08:57:20 +0100 Subject: Re: Opam installation woes Thread-Topic: Opam installation woes Thread-Index: Ac11O28lOy4fMveWRcqG8Bg5FNTKwg== Message-ID: <2F36B404-E5AD-48AB-A2BC-D44691DC61D1@nottingham.ac.uk> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Gazagnaire , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 07:57:46 -0000 X-Keywords: X-UID: 472 Status: O Content-Length: 1684 Lines: 39 On 8 Aug 2012, at 08:51, Jonathan Ludlam wrote: > Hmm. Going back to the opam woes, I'm once more having issues - now with = the standard 'inconsistent assumptions' errors and the like. I suspect this= is because things are being installed in the wrong places. >=20 > The problem I had earlier was that after the switch to the xen compiler, = ~/.opam/system/... was being reported by ocamlc. Unless I'm misunderstandi= ng something, shouldn't the ocamlc binaries be pointing at ~/.opam/xen/... = instead? It looks to me a lot like stuff was being installed into ~/.opam/s= ystem when it should have been going into ~/.opam/xen - specifically the oc= amlc binaries ended up in .opam/system/bin. Is that expected behaviour? i think when i had that problem, i had missed an "eval $(opam config -env)"= at some point... (but can't suggest anything else i'm afraid) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From thomas.gazagnaire@gmail.com Wed Aug 08 09:24:40 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz1ZI-00023K-ER (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 09:24:40 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [74.125.82.51 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-wg0-f51.google.com ([74.125.82.51]:48267) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1Sz1ZH-0002En-oH (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 09:24:40 +0100 Received: by wgbed3 with SMTP id ed3so335395wgb.20 for ; Wed, 08 Aug 2012 01:24:39 -0700 (PDT) Received: by 10.180.86.226 with SMTP id s2mr627513wiz.9.1344414279412; Wed, 08 Aug 2012 01:24:39 -0700 (PDT) Received: from [192.168.0.14] (gou06-3-88-170-165-56.fbx.proxad.net. [88.170.165.56]) by mx.google.com with ESMTPS id t7sm5895728wix.6.2012.08.08.01.24.37 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 08 Aug 2012 01:24:38 -0700 (PDT) Sender: Thomas Gazagnaire Subject: Re: Opam installation woes Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: Date: Wed, 8 Aug 2012 10:20:36 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> To: Jonathan Ludlam X-Mailer: Apple Mail (2.1084) Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 08:24:40 -0000 X-Keywords: X-UID: 473 Status: O Content-Length: 1720 Lines: 44 > Hmm. Going back to the opam woes, I'm once more having issues - now = with the standard 'inconsistent assumptions' errors and the like. I = suspect this is because things are being installed in the wrong places. >=20 > The problem I had earlier was that after the switch to the xen = compiler, ~/.opam/system/... was being reported by ocamlc. Unless I'm = misunderstanding something, shouldn't the ocamlc binaries be pointing at = ~/.opam/xen/... instead? It looks to me a lot like stuff was being = installed into ~/.opam/system when it should have been going into = ~/.opam/xen - specifically the ocamlc binaries ended up in = .opam/system/bin. Is that expected behaviour? Hmm, that's very weird. I've just cloned and installed a fresh opam repo = on my ubuntu dev-box, and I've run: $ opam init mirage http://mirage.github.com/opam $ opam switch -alias xen 3.12.1+mirage-xen and everything runs fine. Could you have a look at any ocamlfind.env files you can find in = ~/.opam/ ? I'm running some more tests, but I'm a bit puzzle by your issue. Maybe = you can try to repro it with OPAMDEBUG=3D1 and send me the execution = trace ? >>>> I think it's high time we setup a little continuous build VM. = Thomas, is there an easy way to iterate through every package and = attempt a compilation? I could do a big 'opam install `opam list`'-style = command, but something that has reasonable logging for failures would be = useful. Completely agree, my tiny EC2 VM is not able to compile ocaml without = crashing, so if someone has a bigger build machine available, I would be = happy to set-up something there. you can run "opam install `opam list -short`" if you want to install all = packages. -- Thomas= From Jonathan.Ludlam@eu.citrix.com Wed Aug 08 15:00:00 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz6nn-0002fU-VR (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 14:59:59 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [66.165.176.63 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp02.citrix.com ([66.165.176.63]:56970) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1Sz6nn-0005vN-n4 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 14:59:59 +0100 X-IronPort-AV: E=Sophos;i="4.77,733,1336363200"; d="scan'208";a="204534067" Received: from ftlpmailmx01.citrite.net ([10.13.107.65]) by FTLPIPO02.CITRIX.COM with ESMTP/TLS/RC4-MD5; 08 Aug 2012 09:59:56 -0400 Received: from [10.80.118.177] (10.80.118.177) by FTLPMAILMX01.citrite.net (10.13.107.65) with Microsoft SMTP Server id 8.3.213.0; Wed, 8 Aug 2012 09:59:57 -0400 Message-ID: <502270DC.70605@eu.citrix.com> Date: Wed, 8 Aug 2012 14:59:56 +0100 From: Jon Ludlam User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: Thomas Gazagnaire Subject: Re: Opam installation woes References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> In-Reply-To: <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 14:00:00 -0000 X-Keywords: X-UID: 474 Status: O Content-Length: 2568 Lines: 56 Running the following script causes the problem. It actually fails after the opam switch because of the 'ocamlc -config | grep system' problem, but that's because the ocaml compilers have already been installed into ~/.opam/system/bin. I'm just retrying it without the 'eval `opam config -env`', but I kept it in for now as the line is in my .profile, and thus I'm certain to forget not to do it... :-) Jon #!/bin/bash set -e set -x rm -rf ~/.opam #opam init -kind rsync default /home/jon/devel/opam-repository opam init default http://mirage.github.com/opam opam remote -kind git -add dev git://github.com/mirage/opam-repo-dev opam update eval `opam config -env` opam --verbose install lwt ls ~/.opam/system/bin > before opam switch -alias xen 3.12.1+mirage-xen || true ls ~/.opam/system/bin > after On 08/08/12 09:20, Thomas Gazagnaire wrote: >> Hmm. Going back to the opam woes, I'm once more having issues - now with the standard 'inconsistent assumptions' errors and the like. I suspect this is because things are being installed in the wrong places. >> >> The problem I had earlier was that after the switch to the xen compiler, ~/.opam/system/... was being reported by ocamlc. Unless I'm misunderstanding something, shouldn't the ocamlc binaries be pointing at ~/.opam/xen/... instead? It looks to me a lot like stuff was being installed into ~/.opam/system when it should have been going into ~/.opam/xen - specifically the ocamlc binaries ended up in .opam/system/bin. Is that expected behaviour? > Hmm, that's very weird. I've just cloned and installed a fresh opam repo on my ubuntu dev-box, and I've run: > > $ opam init mirage http://mirage.github.com/opam > $ opam switch -alias xen 3.12.1+mirage-xen > > and everything runs fine. > > Could you have a look at any ocamlfind.env files you can find in ~/.opam/ ? > > I'm running some more tests, but I'm a bit puzzle by your issue. Maybe you can try to repro it with OPAMDEBUG=1 and send me the execution trace ? > > >>>>> I think it's high time we setup a little continuous build VM. Thomas, is there an easy way to iterate through every package and attempt a compilation? I could do a big 'opam install `opam list`'-style command, but something that has reasonable logging for failures would be useful. > Completely agree, my tiny EC2 VM is not able to compile ocaml without crashing, so if someone has a bigger build machine available, I would be happy to set-up something there. > > you can run "opam install `opam list -short`" if you want to install all packages. > > -- > Thomas From Jonathan.Ludlam@eu.citrix.com Wed Aug 08 15:15:19 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz72d-00032d-TV (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:15:19 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -5.1 from SpamAssassin-3.3.2-1358082 * -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high * trust * [66.165.176.89 listed in list.dnswl.dnsbl.ja.net] * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.citrix.com ([66.165.176.89]:24949) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1Sz72b-0000sG-Sn (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:15:19 +0100 X-IronPort-AV: E=Sophos;i="4.77,733,1336363200"; d="scan'208";a="33971642" Received: from ftlpmailmx02.citrite.net ([10.13.107.66]) by FTLPIPO01.CITRIX.COM with ESMTP/TLS/RC4-MD5; 08 Aug 2012 10:15:12 -0400 Received: from [10.80.118.177] (10.80.118.177) by FTLPMAILMX02.citrite.net (10.13.107.66) with Microsoft SMTP Server id 8.3.213.0; Wed, 8 Aug 2012 10:14:56 -0400 Message-ID: <5022745E.5040807@eu.citrix.com> Date: Wed, 8 Aug 2012 15:14:54 +0100 From: Jon Ludlam User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: Thomas Gazagnaire Subject: Re: Opam installation woes References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> In-Reply-To: <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 14:15:20 -0000 X-Keywords: X-UID: 475 Status: O Content-Length: 1913 Lines: 33 Removing that 'eval `opam config -env`' made the compilation succeed, but switching still causes a load of things to get installed in .opam/system/bin. Jon On 08/08/12 09:20, Thomas Gazagnaire wrote: >> Hmm. Going back to the opam woes, I'm once more having issues - now with the standard 'inconsistent assumptions' errors and the like. I suspect this is because things are being installed in the wrong places. >> >> The problem I had earlier was that after the switch to the xen compiler, ~/.opam/system/... was being reported by ocamlc. Unless I'm misunderstanding something, shouldn't the ocamlc binaries be pointing at ~/.opam/xen/... instead? It looks to me a lot like stuff was being installed into ~/.opam/system when it should have been going into ~/.opam/xen - specifically the ocamlc binaries ended up in .opam/system/bin. Is that expected behaviour? > Hmm, that's very weird. I've just cloned and installed a fresh opam repo on my ubuntu dev-box, and I've run: > > $ opam init mirage http://mirage.github.com/opam > $ opam switch -alias xen 3.12.1+mirage-xen > > and everything runs fine. > > Could you have a look at any ocamlfind.env files you can find in ~/.opam/ ? > > I'm running some more tests, but I'm a bit puzzle by your issue. Maybe you can try to repro it with OPAMDEBUG=1 and send me the execution trace ? > > >>>>> I think it's high time we setup a little continuous build VM. Thomas, is there an easy way to iterate through every package and attempt a compilation? I could do a big 'opam install `opam list`'-style command, but something that has reasonable logging for failures would be useful. > Completely agree, my tiny EC2 VM is not able to compile ocaml without crashing, so if someone has a bigger build machine available, I would be happy to set-up something there. > > you can run "opam install `opam list -short`" if you want to install all packages. > > -- > Thomas From txl@Cs.Nott.AC.UK Wed Aug 08 15:25:48 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz7Cm-0003Jn-Br (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:25:48 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score 0.9 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 HTML_MESSAGE BODY: HTML included in message * 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay * lines * 1.0 MANY_SPAN_IN_TEXT Many tags embedded within text X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:47108 helo=ixe-mta-19.emailfiltering.com) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1Sz7Cg-0006rf-mn (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:25:48 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2336208844 ;2b88f3b96149ebb6; Wed, 08 Aug 2012 15:25:22 +0100 Received: from marian.cs.nott.ac.uk ([128.243.21.16] helo=Cs.Nott.AC.UK) by smtp4.nottingham.ac.uk with smtp (Exim 4.77) (envelope-from ) id 1Sz7CL-0000B3-MX for cl-mirage@lists.cam.ac.uk; Wed, 08 Aug 2012 15:25:21 +0100 Received: from dhcp-22-238.mrl.nottingham.ac.uk by marian.Cs.Nott.AC.UK id aa14246; 8 Aug 2012 15:21 BST Message-ID: <502275CA.4070604@cs.nott.ac.uk> Date: Wed, 08 Aug 2012 15:20:58 +0100 From: work User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: cl-mirage@lists.cam.ac.uk Subject: opam/mirage and the dreamplug Content-Type: multipart/alternative; boundary="------------080900090902020908090905" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 14:25:48 -0000 X-Keywords: X-UID: 476 Status: O Content-Length: 9224 Lines: 187 This is a multi-part message in MIME format. --------------080900090902020908090905 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sorry if this turns out to be something obvious...I've just tried to install mirage on a dreamplug using opam using the instructions at openmirage.org/wiki/install. When I run: opam --verbose install cow mirage-fs Seems that git-archive is failing for some reason: 4.00.0 Install: cstruct.0.3dev Install: mirage-fs.0.3dev Install: mirage.0.3dev Install: mirage-net.0.3dev Install: cohttp.0.3dev Install: cow.0.3dev [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see /root/.opam/log/command57.{info,err,out}) [13182] ERROR: Cannot run git-archive in /root/.opam/repo/dev/git/cstruct.0.3dev Fatal error: exception Globals.Exit(66) gzip: cstruct.0.3dev.tar: No such file or directory [13179] WARNING: Command "opam-git-download git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see /root/.opam/log/command799.{info,err,out}) gzip: cstruct.0.3dev.tar: No such file or directory [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see /root/.opam/log/command57.{info,err,out}) [13182] ERROR: Cannot run git-archive in /root/.opam/repo/dev/git/cstruct.0.3dev Fatal error: exception Globals.Exit(66) [13179] ERROR: opam-git-download failed [13179] ERROR: Globals.Exit(66) [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see /root/.opam/log/command1386.{info,err,out}) [13189] ERROR: Cannot run git-archive in /root/.opam/repo/dev/git/mirage-fs.0.3dev Fatal error: exception Globals.Exit(66) gzip: mirage-fs.0.3dev.tar: No such file or directory [13186] WARNING: Command "opam-git-download git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see /root/.opam/log/command433.{info,err,out}) gzip: mirage-fs.0.3dev.tar: No such file or directory [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see /root/.opam/log/command1386.{info,err,out}) [13189] ERROR: Cannot run git-archive in /root/.opam/repo/dev/git/mirage-fs.0.3dev Fatal error: exception Globals.Exit(66) [13186] ERROR: opam-git-download failed [13186] ERROR: Globals.Exit(66) [13174] ERROR: Command failed while installing cstruct.0.3dev 'opam --verbose install cow mirage-fs' failed The directory/root/.opam/repo/dev/git/cstruct.0.3dev does exist: -rwxr-xr-x 1 root root 2915 2012-08-08 11:39 cmd -rw-r--r-- 1 root root 242 2012-08-08 11:39 Makefile -rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml -rw-r--r-- 1 root root 220 2012-08-08 11:39 README.md drwxr-xr-x 5 root root 4096 2012-08-08 11:39 unix drwxr-xr-x 4 root root 4096 2012-08-08 11:39 xen Have the same issue when I attempt an install of cstruct on its own (opam install cstruct) I get: Install: cstruct.0.3dev [13205] WARNING: Command "opam-git-download git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see /root/.opam/log/command2033.{info,err,out}) gzip: cstruct.0.3dev.tar: No such file or directory [13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see /root/.opam/log/command1090.{info,err,out}) [13206] ERROR: Cannot run git-archive in /root/.opam/repo/dev/git/cstruct.0.3dev Fatal error: exception Globals.Exit(66) [13205] ERROR: opam-git-download failed [13205] ERROR: Globals.Exit(66) [13202] ERROR: Command failed while installing cstruct.0.3dev 'opam install cstruct' failed The logs don't give much more info. Guess it's something to do with my environment but not too sure what it might be (git-archive is there...). Alternatively I've heard that mirage has been successfully installed on a dreamplug - if there are any pointers anyone can give me that'd be great. Thanks, Tom --------------080900090902020908090905 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit 
Sorry if this turns out to be something obvious...I've just tried to install mirage on a dreamplug using opam using the instructions at openmirage.org/wiki/install.  When I run:

opam --verbose install cow mirage-fs

Seems that  git-archive is failing for some reason:

4.00.0
Install: cstruct.0.3dev
Install: mirage-fs.0.3dev
Install: mirage.0.3dev
Install: mirage-net.0.3dev
Install: cohttp.0.3dev
Install: cow.0.3dev
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: cstruct.0.3dev.tar: No such file or directory
[13179] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command799.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13179] ERROR: opam-git-download failed
[13179] ERROR: Globals.Exit(66)
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13186] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see 
/root/.opam/log/command433.{info,err,out})
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
[13186] ERROR: opam-git-download failed
[13186] ERROR: Globals.Exit(66)
[13174] ERROR: Command failed while installing cstruct.0.3dev
  'opam --verbose install cow mirage-fs' failed


The directory /root/.opam/repo/dev/git/cstruct.0.3dev does exist:

-rwxr-xr-x 1 root root  2915 2012-08-08 11:39 cmd
-rw-r--r-- 1 root root   242 2012-08-08 11:39 Makefile
-rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml
-rw-r--r-- 1 root root   220 2012-08-08 11:39 README.md
drwxr-xr-x 5 root root  4096 2012-08-08 11:39 unix
drwxr-xr-x 4 root root  4096 2012-08-08 11:39 xen

Have the same issue when I attempt an install of cstruct on its own  (opam install cstruct) I get:

Install: cstruct.0.3dev
[13205] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command2033.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command1090.{info,err,out})
[13206] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13205] ERROR: opam-git-download failed
[13205] ERROR: Globals.Exit(66)
[13202] ERROR: Command failed while installing cstruct.0.3dev
  'opam install cstruct' failed

The logs don't give much more info. Guess it's something to do with my environment but not too sure what it might be (git-archive is there...).  Alternatively I've heard that mirage has been successfully installed on a dreamplug - if there are any pointers anyone can give me that'd be great.

Thanks,

Tom
--------------080900090902020908090905-- From thomas.gazagnaire@gmail.com Wed Aug 08 15:26:48 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1Sz7Dk-0003LE-R1 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:26:48 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.212.169 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-wi0-f169.google.com ([209.85.212.169]:38337) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1Sz7Dj-00050O-RX (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 15:26:48 +0100 Received: by wibhm2 with SMTP id hm2so3474321wib.2 for ; Wed, 08 Aug 2012 07:26:47 -0700 (PDT) Received: by 10.180.84.104 with SMTP id x8mr3127208wiy.20.1344436007379; Wed, 08 Aug 2012 07:26:47 -0700 (PDT) Received: from [192.168.0.14] (gou06-3-88-170-165-56.fbx.proxad.net. [88.170.165.56]) by mx.google.com with ESMTPS id ck9sm8001312wib.2.2012.08.08.07.26.45 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 08 Aug 2012 07:26:46 -0700 (PDT) Sender: Thomas Gazagnaire Subject: Re: Opam installation woes Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: <5022745E.5040807@eu.citrix.com> Date: Wed, 8 Aug 2012 16:26:43 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> <5022745E.5040807@eu.citrix.com> To: Jon Ludlam X-Mailer: Apple Mail (2.1084) Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 14:26:48 -0000 X-Keywords: X-UID: 477 Status: O Content-Length: 2336 Lines: 58 > Removing that 'eval `opam config -env`' made the compilation succeed, = but switching still causes a load of things to get installed in = .opam/system/bin. what the ouptut of "opam config -env" ? This is quite weird, as this = should not impact the way opam works. This might be related to = https://github.com/OCamlPro/opam/issues/71 which is a regression = introduced not very lately ... working on that right now with Frederic = to fix the issue, should be fixed soon. Thomas >=20 > On 08/08/12 09:20, Thomas Gazagnaire wrote: >>> Hmm. Going back to the opam woes, I'm once more having issues - now = with the standard 'inconsistent assumptions' errors and the like. I = suspect this is because things are being installed in the wrong places. >>>=20 >>> The problem I had earlier was that after the switch to the xen = compiler, ~/.opam/system/... was being reported by ocamlc. Unless I'm = misunderstanding something, shouldn't the ocamlc binaries be pointing at = ~/.opam/xen/... instead? It looks to me a lot like stuff was being = installed into ~/.opam/system when it should have been going into = ~/.opam/xen - specifically the ocamlc binaries ended up in = .opam/system/bin. Is that expected behaviour? >> Hmm, that's very weird. I've just cloned and installed a fresh opam = repo on my ubuntu dev-box, and I've run: >>=20 >> $ opam init mirage http://mirage.github.com/opam >> $ opam switch -alias xen 3.12.1+mirage-xen >>=20 >> and everything runs fine. >>=20 >> Could you have a look at any ocamlfind.env files you can find in = ~/.opam/ ? >>=20 >> I'm running some more tests, but I'm a bit puzzle by your issue. = Maybe you can try to repro it with OPAMDEBUG=3D1 and send me the = execution trace ? >>=20 >>=20 >>>>>> I think it's high time we setup a little continuous build VM. = Thomas, is there an easy way to iterate through every package and = attempt a compilation? I could do a big 'opam install `opam list`'-style = command, but something that has reasonable logging for failures would be = useful. >> Completely agree, my tiny EC2 VM is not able to compile ocaml without = crashing, so if someone has a bigger build machine available, I would be = happy to set-up something there. >>=20 >> you can run "opam install `opam list -short`" if you want to install = all packages. >>=20 >> -- >> Thomas >=20 From thomas.gazagnaire@gmail.com Wed Aug 08 19:34:21 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzB5J-0000Wo-6H (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 19:34:21 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.7 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [74.125.82.179 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-we0-f179.google.com ([74.125.82.179]:59456) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1SzB5I-0007ef-WJ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 08 Aug 2012 19:34:21 +0100 Received: by weyx10 with SMTP id x10so635522wey.38 for ; Wed, 08 Aug 2012 11:34:19 -0700 (PDT) Received: by 10.216.255.146 with SMTP id j18mr3249078wes.163.1344450859825; Wed, 08 Aug 2012 11:34:19 -0700 (PDT) Received: from [192.168.0.14] (gou06-3-88-170-165-56.fbx.proxad.net. [88.170.165.56]) by mx.google.com with ESMTPS id ep14sm10227282wid.0.2012.08.08.11.34.17 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 08 Aug 2012 11:34:18 -0700 (PDT) Sender: Thomas Gazagnaire Subject: Re: Opam installation woes Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: <5022745E.5040807@eu.citrix.com> Date: Wed, 8 Aug 2012 20:34:15 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5B6D100F-E8EE-4E03-8AFC-B7F62DDDA62B@gazagnaire.org> References: <5CDE3872-D6EF-4984-BDF7-5203BD867EA6@gazagnaire.org> <6E3DC745-9628-4925-9E5E-B2C75066F05E@eu.citrix.com> <2E2CEEEE-C473-4466-AE00-748E574E5C3D@recoil.org> <57221559-6987-4A84-A64A-B1ED450A40A5@nottingham.ac.uk> <3261198B-68B5-4D5D-9566-EA4ADE9BB87D@recoil.org> <054A8952-B589-4F63-A2D2-B063D1316037@recoil.org> <88887F19-CE98-43E6-A870-FDEA8102F33B@gazagnaire.org> <5022745E.5040807@eu.citrix.com> To: Jon Ludlam X-Mailer: Apple Mail (2.1084) Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 18:34:21 -0000 X-Keywords: X-UID: 478 Status: O Content-Length: 2437 Lines: 65 Frederic just pushed something which seems to fix the regression, so = I've made a new release of opam (0.3.2). Sorry for the mess, my plans = for the next days is to make a 0.4 release (which will improve support = for dev repositories) and set-up some automatic regression tests to = avoid this to happen again. -- Thomas I don't have anything to test=20 On Aug 8, 2012, at 4:14 PM, Jon Ludlam wrote: > Removing that 'eval `opam config -env`' made the compilation succeed, = but switching still causes a load of things to get installed in = .opam/system/bin. >=20 > Jon >=20 >=20 >=20 > On 08/08/12 09:20, Thomas Gazagnaire wrote: >>> Hmm. Going back to the opam woes, I'm once more having issues - now = with the standard 'inconsistent assumptions' errors and the like. I = suspect this is because things are being installed in the wrong places. >>>=20 >>> The problem I had earlier was that after the switch to the xen = compiler, ~/.opam/system/... was being reported by ocamlc. Unless I'm = misunderstanding something, shouldn't the ocamlc binaries be pointing at = ~/.opam/xen/... instead? It looks to me a lot like stuff was being = installed into ~/.opam/system when it should have been going into = ~/.opam/xen - specifically the ocamlc binaries ended up in = .opam/system/bin. Is that expected behaviour? >> Hmm, that's very weird. I've just cloned and installed a fresh opam = repo on my ubuntu dev-box, and I've run: >>=20 >> $ opam init mirage http://mirage.github.com/opam >> $ opam switch -alias xen 3.12.1+mirage-xen >>=20 >> and everything runs fine. >>=20 >> Could you have a look at any ocamlfind.env files you can find in = ~/.opam/ ? >>=20 >> I'm running some more tests, but I'm a bit puzzle by your issue. = Maybe you can try to repro it with OPAMDEBUG=3D1 and send me the = execution trace ? >>=20 >>=20 >>>>>> I think it's high time we setup a little continuous build VM. = Thomas, is there an easy way to iterate through every package and = attempt a compilation? I could do a big 'opam install `opam list`'-style = command, but something that has reasonable logging for failures would be = useful. >> Completely agree, my tiny EC2 VM is not able to compile ocaml without = crashing, so if someone has a bigger build machine available, I would be = happy to set-up something there. >>=20 >> you can run "opam install `opam list -short`" if you want to install = all packages. >>=20 >> -- >> Thomas >=20 From anil@recoil.org Thu Aug 09 00:10:58 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzFP0-0003KU-Ei (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 00:10:58 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score 0.9 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 HTML_MESSAGE BODY: HTML included in message * 1.0 MANY_SPAN_IN_TEXT Many tags embedded within text X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:37764 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1SzFOz-0007Eo-Xj (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 00:10:58 +0100 Received: (qmail 5445 invoked by uid 634); 8 Aug 2012 23:10:57 -0000 X-Spam-Level: * X-Spam-Status: No, hits=1.5 required=5.0 tests=ALL_TRUSTED,HTML_MESSAGE,MANY_SPAN_IN_TEXT X-Spam-Check-By: dark.recoil.org Received: from Unknown (HELO [192.168.65.255]) (207.239.114.206) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 09 Aug 2012 00:10:56 +0100 Content-Type: multipart/alternative; boundary="Apple-Mail=_5FEF81F7-D258-490E-AD20-106B4EDF05AB" Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: opam/mirage and the dreamplug From: Anil Madhavapeddy In-Reply-To: <502275CA.4070604@cs.nott.ac.uk> Date: Wed, 8 Aug 2012 15:55:13 -0700 Message-Id: <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> References: <502275CA.4070604@cs.nott.ac.uk> To: work X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2012 23:10:58 -0000 X-Keywords: X-UID: 479 Status: O Content-Length: 10505 Lines: 204 --Apple-Mail=_5FEF81F7-D258-490E-AD20-106B4EDF05AB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 which version of OPAM? This isn't a dreamplug-specific failure, so it = might be to do with the fixes in 0.3.2. I need to buy an ARM box over = here as I forgot to bring my Dreamplug along for the summer ride... -anil On 8 Aug 2012, at 07:20, work wrote: > Sorry if this turns out to be something obvious...I've just tried to = install mirage on a dreamplug using opam using the instructions at = openmirage.org/wiki/install. When I run: >=20 > opam --verbose install cow mirage-fs >=20 > Seems that git-archive is failing for some reason: >=20 > 4.00.0 > Install: cstruct.0.3dev > Install: mirage-fs.0.3dev > Install: mirage.0.3dev > Install: mirage-net.0.3dev > Install: cohttp.0.3dev > Install: cow.0.3dev > [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see=20 > /root/.opam/log/command57.{info,err,out}) > [13182] ERROR: Cannot run git-archive in=20 > /root/.opam/repo/dev/git/cstruct.0.3dev > Fatal error: exception Globals.Exit(66) > gzip: cstruct.0.3dev.tar: No such file or directory > [13179] WARNING: Command "opam-git-download=20 > git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see=20 > /root/.opam/log/command799.{info,err,out}) > gzip: cstruct.0.3dev.tar: No such file or directory > [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see=20 > /root/.opam/log/command57.{info,err,out}) > [13182] ERROR: Cannot run git-archive in=20 > /root/.opam/repo/dev/git/cstruct.0.3dev > Fatal error: exception Globals.Exit(66) > [13179] ERROR: opam-git-download failed > [13179] ERROR: Globals.Exit(66) > [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see=20 > /root/.opam/log/command1386.{info,err,out}) > [13189] ERROR: Cannot run git-archive in=20 > /root/.opam/repo/dev/git/mirage-fs.0.3dev > Fatal error: exception Globals.Exit(66) > gzip: mirage-fs.0.3dev.tar: No such file or directory > [13186] WARNING: Command "opam-git-download=20 > git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see=20 > /root/.opam/log/command433.{info,err,out}) > gzip: mirage-fs.0.3dev.tar: No such file or directory > [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see=20 > /root/.opam/log/command1386.{info,err,out}) > [13189] ERROR: Cannot run git-archive in=20 > /root/.opam/repo/dev/git/mirage-fs.0.3dev > Fatal error: exception Globals.Exit(66) > [13186] ERROR: opam-git-download failed > [13186] ERROR: Globals.Exit(66) > [13174] ERROR: Command failed while installing cstruct.0.3dev > 'opam --verbose install cow mirage-fs' failed >=20 >=20 > The directory /root/.opam/repo/dev/git/cstruct.0.3dev does exist: >=20 > -rwxr-xr-x 1 root root 2915 2012-08-08 11:39 cmd > -rw-r--r-- 1 root root 242 2012-08-08 11:39 Makefile > -rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml > -rw-r--r-- 1 root root 220 2012-08-08 11:39 README.md > drwxr-xr-x 5 root root 4096 2012-08-08 11:39 unix > drwxr-xr-x 4 root root 4096 2012-08-08 11:39 xen >=20 > Have the same issue when I attempt an install of cstruct on its own = (opam install cstruct) I get: >=20 > Install: cstruct.0.3dev > [13205] WARNING: Command "opam-git-download=20 > git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see=20 > /root/.opam/log/command2033.{info,err,out}) > gzip: cstruct.0.3dev.tar: No such file or directory > [13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see=20 > /root/.opam/log/command1090.{info,err,out}) > [13206] ERROR: Cannot run git-archive in=20 > /root/.opam/repo/dev/git/cstruct.0.3dev > Fatal error: exception Globals.Exit(66) > [13205] ERROR: opam-git-download failed > [13205] ERROR: Globals.Exit(66) > [13202] ERROR: Command failed while installing cstruct.0.3dev > 'opam install cstruct' failed >=20 > The logs don't give much more info. Guess it's something to do with my = environment but not too sure what it might be (git-archive is there...). = Alternatively I've heard that mirage has been successfully installed on = a dreamplug - if there are any pointers anyone can give me that'd be = great. >=20 > Thanks, >=20 > Tom --Apple-Mail=_5FEF81F7-D258-490E-AD20-106B4EDF05AB Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=iso-8859-1 which version of OPAM?  This isn't a dreamplug-specific failure, so it might be to do with the fixes in 0.3.2.  I need to buy an ARM box over here as I forgot to bring my Dreamplug along for the summer ride...

-anil

On 8 Aug 2012, at 07:20, work <txl@Cs.Nott.AC.UK> wrote:

Sorry if this turns out to be something obvious...I've just tried to install mirage on a dreamplug using opam using the instructions at openmirage.org/wiki/install.  When I run:

opam --verbose install cow mirage-fs

Seems that  git-archive is failing for some reason:

4.00.0
Install: cstruct.0.3dev
Install: mirage-fs.0.3dev
Install: mirage.0.3dev
Install: mirage-net.0.3dev
Install: cohttp.0.3dev
Install: cow.0.3dev
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: cstruct.0.3dev.tar: No such file or directory
[13179] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command799.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13179] ERROR: opam-git-download failed
[13179] ERROR: Globals.Exit(66)
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13186] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see 
/root/.opam/log/command433.{info,err,out})
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
[13186] ERROR: opam-git-download failed
[13186] ERROR: Globals.Exit(66)
[13174] ERROR: Command failed while installing cstruct.0.3dev
  'opam --verbose install cow mirage-fs' failed


The directory /root/.opam/repo/dev/git/cstruct.0.3dev does exist:

-rwxr-xr-x 1 root root  2915 2012-08-08 11:39 cmd
-rw-r--r-- 1 root root   242 2012-08-08 11:39 Makefile
-rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml
-rw-r--r-- 1 root root   220 2012-08-08 11:39 README.md
drwxr-xr-x 5 root root  4096 2012-08-08 11:39 unix
drwxr-xr-x 4 root root  4096 2012-08-08 11:39 xen

Have the same issue when I attempt an install of cstruct on its own  (opam install cstruct) I get:

Install: cstruct.0.3dev
[13205] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command2033.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command1090.{info,err,out})
[13206] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13205] ERROR: opam-git-download failed
[13205] ERROR: Globals.Exit(66)
[13202] ERROR: Command failed while installing cstruct.0.3dev
  'opam install cstruct' failed

The logs don't give much more info. Guess it's something to do with my environment but not too sure what it might be (git-archive is there...).  Alternatively I've heard that mirage has been successfully installed on a dreamplug - if there are any pointers anyone can give me that'd be great.

Thanks,

Tom

--Apple-Mail=_5FEF81F7-D258-490E-AD20-106B4EDF05AB-- From txl@Cs.Nott.AC.UK Thu Aug 09 09:45:50 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzONK-0003TY-2K (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 09:45:50 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamScore: sssss X-Cam-SpamDetails: score 5.5 from SpamAssassin-3.3.2-1358082 * 1.0 MISSING_HEADERS Missing To: header * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 HTML_MESSAGE BODY: HTML included in message * 3.6 NAME_EMAIL_DIFF Sender NAME is an unrelated email address * 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay * lines * 1.0 MANY_SPAN_IN_TEXT Many tags embedded within text X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:39383 helo=ixe-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SzONE-0002hz-qB (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 09:45:50 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2336833004 ;8a71249ab5483b18; Thu, 09 Aug 2012 09:45:23 +0100 Received: from marian.cs.nott.ac.uk ([128.243.21.16] helo=Cs.Nott.AC.UK) by smtp4.nottingham.ac.uk with smtp (Exim 4.77) (envelope-from ) id 1SzOMt-0001Fr-LN for cl-mirage@lists.cam.ac.uk; Thu, 09 Aug 2012 09:45:23 +0100 Received: from blunt.mrl.nottingham.ac.uk by marian.Cs.Nott.AC.UK id aa23455; 9 Aug 2012 9:44 BST Message-ID: <5023786D.90609@cs.nott.ac.uk> Date: Thu, 09 Aug 2012 09:44:29 +0100 From: "txl@cs.nott.ac.uk" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 CC: cl-mirage@lists.cam.ac.uk Subject: Re: opam/mirage and the dreamplug References: <502275CA.4070604@cs.nott.ac.uk> <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> In-Reply-To: <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> Content-Type: multipart/alternative; boundary="------------070707030709060100050302" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 08:45:50 -0000 X-Keywords: X-UID: 480 Status: O Content-Length: 11476 Lines: 246 This is a multi-part message in MIME format. --------------070707030709060100050302 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 08/08/2012 23:55, Anil Madhavapeddy wrote: > which version of OPAM? This isn't a dreamplug-specific failure, so it > might be to do with the fixes in 0.3.2. I need to buy an ARM box over > here as I forgot to bring my Dreamplug along for the summer ride... > is 0.3.1 cheers, Tom > -anil > > On 8 Aug 2012, at 07:20, work > wrote: > >> Sorry if this turns out to be something obvious...I've just tried to install mirage on a dreamplug using opam using the instructions atopenmirage.org/wiki/install . When I run: >> >> opam --verbose install cow mirage-fs >> >> Seems that git-archive is failing for some reason: >> >> 4.00.0 >> Install: cstruct.0.3dev >> Install: mirage-fs.0.3dev >> Install: mirage.0.3dev >> Install: mirage-net.0.3dev >> Install: cohttp.0.3dev >> Install: cow.0.3dev >> [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see >> /root/.opam/log/command57.{info,err,out}) >> [13182] ERROR: Cannot run git-archive in >> /root/.opam/repo/dev/git/cstruct.0.3dev >> Fatal error: exception Globals.Exit(66) >> gzip: cstruct.0.3dev.tar: No such file or directory >> [13179] WARNING: Command "opam-git-download >> git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see >> /root/.opam/log/command799.{info,err,out}) >> gzip: cstruct.0.3dev.tar: No such file or directory >> [13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see >> /root/.opam/log/command57.{info,err,out}) >> [13182] ERROR: Cannot run git-archive in >> /root/.opam/repo/dev/git/cstruct.0.3dev >> Fatal error: exception Globals.Exit(66) >> [13179] ERROR: opam-git-download failed >> [13179] ERROR: Globals.Exit(66) >> [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see >> /root/.opam/log/command1386.{info,err,out}) >> [13189] ERROR: Cannot run git-archive in >> /root/.opam/repo/dev/git/mirage-fs.0.3dev >> Fatal error: exception Globals.Exit(66) >> gzip: mirage-fs.0.3dev.tar: No such file or directory >> [13186] WARNING: Command "opam-git-download >> git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see >> /root/.opam/log/command433.{info,err,out}) >> gzip: mirage-fs.0.3dev.tar: No such file or directory >> [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see >> /root/.opam/log/command1386.{info,err,out}) >> [13189] ERROR: Cannot run git-archive in >> /root/.opam/repo/dev/git/mirage-fs.0.3dev >> Fatal error: exception Globals.Exit(66) >> [13186] ERROR: opam-git-download failed >> [13186] ERROR: Globals.Exit(66) >> [13174] ERROR: Command failed while installing cstruct.0.3dev >> 'opam --verbose install cow mirage-fs' failed >> >> >> The directory/root/.opam/repo/dev/git/cstruct.0.3dev does exist: >> >> -rwxr-xr-x 1 root root 2915 2012-08-08 11:39 cmd >> -rw-r--r-- 1 root root 242 2012-08-08 11:39 Makefile >> -rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml >> -rw-r--r-- 1 root root 220 2012-08-08 11:39 README.md >> drwxr-xr-x 5 root root 4096 2012-08-08 11:39 unix >> drwxr-xr-x 4 root root 4096 2012-08-08 11:39 xen >> >> Have the same issue when I attempt an install of cstruct on its own (opam install cstruct) I get: >> >> Install: cstruct.0.3dev >> [13205] WARNING: Command "opam-git-download >> git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see >> /root/.opam/log/command2033.{info,err,out}) >> gzip: cstruct.0.3dev.tar: No such file or directory >> [13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see >> /root/.opam/log/command1090.{info,err,out}) >> [13206] ERROR: Cannot run git-archive in >> /root/.opam/repo/dev/git/cstruct.0.3dev >> Fatal error: exception Globals.Exit(66) >> [13205] ERROR: opam-git-download failed >> [13205] ERROR: Globals.Exit(66) >> [13202] ERROR: Command failed while installing cstruct.0.3dev >> 'opam install cstruct' failed >> >> The logs don't give much more info. Guess it's something to do with my environment but not too sure what it might be (git-archive is there...). Alternatively I've heard that mirage has been successfully installed on a dreamplug - if there are any pointers anyone can give me that'd be great. >> >> Thanks, >> >> Tom > --------------070707030709060100050302 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 08/08/2012 23:55, Anil Madhavapeddy wrote:
which version of OPAM?  This isn't a dreamplug-specific failure, so it might be to do with the fixes in 0.3.2.  I need to buy an ARM box over here as I forgot to bring my Dreamplug along for the summer ride...

is 0.3.1

cheers,

Tom
-anil

On 8 Aug 2012, at 07:20, work <txl@Cs.Nott.AC.UK> wrote:

Sorry if this turns out to be something obvious...I've just tried to install mirage on a dreamplug using opam using the instructions at openmirage.org/wiki/install.  When I run:

opam --verbose install cow mirage-fs

Seems that  git-archive is failing for some reason:

4.00.0
Install: cstruct.0.3dev
Install: mirage-fs.0.3dev
Install: mirage.0.3dev
Install: mirage-net.0.3dev
Install: cohttp.0.3dev
Install: cow.0.3dev
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: cstruct.0.3dev.tar: No such file or directory
[13179] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command799.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13182] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command57.{info,err,out})
[13182] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13179] ERROR: opam-git-download failed
[13179] ERROR: Globals.Exit(66)
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13186] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev mirage-fs.0.3dev" failed (see 
/root/.opam/log/command433.{info,err,out})
gzip: mirage-fs.0.3dev.tar: No such file or directory
[13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see 
/root/.opam/log/command1386.{info,err,out})
[13189] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/mirage-fs.0.3dev
Fatal error: exception Globals.Exit(66)
[13186] ERROR: opam-git-download failed
[13186] ERROR: Globals.Exit(66)
[13174] ERROR: Command failed while installing cstruct.0.3dev
  'opam --verbose install cow mirage-fs' failed


The directory /root/.opam/repo/dev/git/cstruct.0.3dev does exist:

-rwxr-xr-x 1 root root  2915 2012-08-08 11:39 cmd
-rw-r--r-- 1 root root   242 2012-08-08 11:39 Makefile
-rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml
-rw-r--r-- 1 root root   220 2012-08-08 11:39 README.md
drwxr-xr-x 5 root root  4096 2012-08-08 11:39 unix
drwxr-xr-x 4 root root  4096 2012-08-08 11:39 xen

Have the same issue when I attempt an install of cstruct on its own  (opam install cstruct) I get:

Install: cstruct.0.3dev
[13205] WARNING: Command "opam-git-download 
git://github.com/mirage/opam-repo-dev cstruct.0.3dev" failed (see 
/root/.opam/log/command2033.{info,err,out})
gzip: cstruct.0.3dev.tar: No such file or directory
[13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see 
/root/.opam/log/command1090.{info,err,out})
[13206] ERROR: Cannot run git-archive in 
/root/.opam/repo/dev/git/cstruct.0.3dev
Fatal error: exception Globals.Exit(66)
[13205] ERROR: opam-git-download failed
[13205] ERROR: Globals.Exit(66)
[13202] ERROR: Command failed while installing cstruct.0.3dev
  'opam install cstruct' failed

The logs don't give much more info. Guess it's something to do with my environment but not too sure what it might be (git-archive is there...).  Alternatively I've heard that mirage has been successfully installed on a dreamplug - if there are any pointers anyone can give me that'd be great.

Thanks,

Tom


--------------070707030709060100050302-- From thomas.gazagnaire@gmail.com Thu Aug 09 09:57:37 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzOYj-0003gv-PE (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 09:57:37 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1358082 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.212.169 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-wi0-f169.google.com ([209.85.212.169]:39956) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1SzOYi-000338-Q6 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 09:57:37 +0100 Received: by wibhm2 with SMTP id hm2so144215wib.2 for ; Thu, 09 Aug 2012 01:57:35 -0700 (PDT) Received: by 10.216.214.208 with SMTP id c58mr1478869wep.148.1344502655824; Thu, 09 Aug 2012 01:57:35 -0700 (PDT) Received: from [192.168.0.14] (gou06-3-88-170-165-56.fbx.proxad.net. [88.170.165.56]) by mx.google.com with ESMTPS id bc2sm1059036wib.0.2012.08.09.01.57.34 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 09 Aug 2012 01:57:35 -0700 (PDT) Sender: Thomas Gazagnaire Subject: Re: opam/mirage and the dreamplug Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Gazagnaire In-Reply-To: <5023786D.90609@cs.nott.ac.uk> Date: Thu, 9 Aug 2012 10:57:31 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <4F133FC3-6281-4B80-A41C-EA1C075578C7@gazagnaire.org> References: <502275CA.4070604@cs.nott.ac.uk> <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> <5023786D.90609@cs.nott.ac.uk> To: "txl@cs.nott.ac.uk" X-Mailer: Apple Mail (2.1084) Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 08:57:37 -0000 X-Keywords: X-UID: 481 Status: O Content-Length: 3303 Lines: 104 Hi, opam runs the following command in = /root/.opam/repo/dev/git/cstruct.0.3dev: $ git archive --format=3Dtar --prefix=3Dcstruct.0.3.dev/ HEAD -o = cstruct.0.3.dev.tar maybe git is not in the right path for OPAM ? = /root/.opam/log/commandXXX.info stores the environment in which the = command run (obviously, need to replace the XXX by the right number). -- Thomas >>> /root/.opam/repo/dev/git/cstruct.0.3dev >>> Fatal error: exception Globals.Exit(66) >>> [13179] ERROR: opam-git-download failed >>> [13179] ERROR: Globals.Exit(66) >>> [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see=20= >>>=20 >>> /root/ >>> .opam/log/command1386.{info,err,out}) >>> [13189] ERROR: Cannot run git-archive in=20 >>>=20 >>> /root/ >>> .opam/repo/dev/git/mirage-fs.0.3dev >>> Fatal error: exception Globals.Exit(66) >>> gzip: mirage-fs.0.3dev.tar: No such file or directory >>> [13186] WARNING: Command "opam-git-download=20 >>>=20 >>> git://github.com/mirage/opam-repo-dev >>> mirage-fs.0.3dev" failed (see=20 >>>=20 >>> /root/ >>> .opam/log/command433.{info,err,out}) >>> gzip: mirage-fs.0.3dev.tar: No such file or directory >>> [13189] WARNING: Command "gzip -f mirage-fs.0.3dev.tar" failed (see=20= >>>=20 >>> /root/ >>> .opam/log/command1386.{info,err,out}) >>> [13189] ERROR: Cannot run git-archive in=20 >>>=20 >>> /root/ >>> .opam/repo/dev/git/mirage-fs.0.3dev >>> Fatal error: exception Globals.Exit(66) >>> [13186] ERROR: opam-git-download failed >>> [13186] ERROR: Globals.Exit(66) >>> [13174] ERROR: Command failed while installing cstruct.0.3dev >>> 'opam --verbose install cow mirage-fs' failed >>>=20 >>>=20 >>> The directory=20 >>> /root/ >>> .opam/repo/dev/git/cstruct.0.3dev does exist: >>>=20 >>> -rwxr-xr-x 1 root root 2915 2012-08-08 11:39 cmd >>> -rw-r--r-- 1 root root 242 2012-08-08 11:39 Makefile >>> -rw-r--r-- 1 root root 11420 2012-08-08 11:39 myocamlbuild.ml >>> -rw-r--r-- 1 root root 220 2012-08-08 11:39 README.md >>> drwxr-xr-x 5 root root 4096 2012-08-08 11:39 unix >>> drwxr-xr-x 4 root root 4096 2012-08-08 11:39 xen >>>=20 >>> Have the same issue when I attempt an install of cstruct on its own = (opam install cstruct) I get: >>>=20 >>> Install: cstruct.0.3dev >>> [13205] WARNING: Command "opam-git-download=20 >>>=20 >>> git://github.com/mirage/opam-repo-dev >>> cstruct.0.3dev" failed (see=20 >>>=20 >>> /root/ >>> .opam/log/command2033.{info,err,out}) >>> gzip: cstruct.0.3dev.tar: No such file or directory >>> [13206] WARNING: Command "gzip -f cstruct.0.3dev.tar" failed (see=20 >>>=20 >>> /root/ >>> .opam/log/command1090.{info,err,out}) >>> [13206] ERROR: Cannot run git-archive in=20 >>>=20 >>> /root/ >>> .opam/repo/dev/git/cstruct.0.3dev >>> Fatal error: exception Globals.Exit(66) >>> [13205] ERROR: opam-git-download failed >>> [13205] ERROR: Globals.Exit(66) >>> [13202] ERROR: Command failed while installing cstruct.0.3dev >>> 'opam install cstruct' failed >>>=20 >>> The logs don't give much more info. Guess it's something to do with = my environment but not too sure what it might be (git-archive is = there...). Alternatively I've heard that mirage has been successfully = installed on a dreamplug - if there are any pointers anyone can give me = that'd be great. >>>=20 >>> Thanks, >>>=20 >>> Tom >>>=20 >>=20 >=20 From txl@Cs.Nott.AC.UK Thu Aug 09 10:45:50 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzPJO-00055h-3K (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 10:45:50 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay * lines X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:59913 helo=ixe-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SzPJI-0006KI-r3 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 10:45:49 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2336908576 ;6f063a42c0ea255c; Thu, 09 Aug 2012 10:45:24 +0100 Received: from marian.cs.nott.ac.uk ([128.243.21.16] helo=Cs.Nott.AC.UK) by smtp3.nottingham.ac.uk with smtp (Exim 4.77) (envelope-from ) id 1SzPIx-0005QX-Og for cl-mirage@lists.cam.ac.uk; Thu, 09 Aug 2012 10:45:23 +0100 Received: from dhcp-22-227.mrl.nottingham.ac.uk by marian.Cs.Nott.AC.UK id aa02939; 9 Aug 2012 10:39 BST Message-ID: <5023855A.2070707@cs.nott.ac.uk> Date: Thu, 09 Aug 2012 10:39:38 +0100 From: work User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: cl-mirage@lists.cam.ac.uk Subject: Re: opam/mirage and the dreamplug References: <502275CA.4070604@cs.nott.ac.uk> <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> <5023786D.90609@cs.nott.ac.uk> <4F133FC3-6281-4B80-A41C-EA1C075578C7@gazagnaire.org> In-Reply-To: <4F133FC3-6281-4B80-A41C-EA1C075578C7@gazagnaire.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 09:45:50 -0000 X-Keywords: X-UID: 482 Status: O Content-Length: 701 Lines: 20 On 09/08/2012 09:57, Thomas Gazagnaire wrote: > git archive --format=tar --prefix=cstruct.0.3.dev/ HEAD -o cstruct.0.3.dev.tar Thanks... When I run this directily in /root/.opam/repo/dev/git/cstruct.0.3dev the following is spit out (and no tar is created): pax_global_header00006660000000000000000000000064120046302300014500gustar00rootroot0000000000000052 comment=94b03dcec6a2a8cabe3069d5ecd26aa349e4ea55 cstruct.0.3.dev/000077500000000000000000000000001200463023000136075ustar00rootroot00000000000000 Seems like it doesn't like the -o option as the following seems to behave: git archive --format=tar --prefix=cstruct.0.3.dev/ HEAD> cstruct.0.3.dev.tar Think I need to upgrade git... Tom From txl@Cs.Nott.AC.UK Thu Aug 09 11:25:49 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzPw5-0006JD-U2 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 11:25:49 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay * lines X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:57305 helo=ixe-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1SzPw0-0003EM-rQ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 11:25:49 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2336964224 ;9b9e83795a5f15d2; Thu, 09 Aug 2012 11:25:24 +0100 Received: from marian.cs.nott.ac.uk ([128.243.21.16] helo=Cs.Nott.AC.UK) by smtp4.nottingham.ac.uk with smtp (Exim 4.77) (envelope-from ) id 1SzPvg-000818-2c for cl-mirage@lists.cam.ac.uk; Thu, 09 Aug 2012 11:25:24 +0100 Received: from dhcp-22-227.mrl.nottingham.ac.uk by marian.Cs.Nott.AC.UK id aa06670; 9 Aug 2012 11:20 BST Message-ID: <50238ED5.5000409@cs.nott.ac.uk> Date: Thu, 09 Aug 2012 11:20:05 +0100 From: work User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: cl-mirage@lists.cam.ac.uk Subject: Re: opam/mirage and the dreamplug References: <502275CA.4070604@cs.nott.ac.uk> <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> <5023786D.90609@cs.nott.ac.uk> <4F133FC3-6281-4B80-A41C-EA1C075578C7@gazagnaire.org> <5023855A.2070707@cs.nott.ac.uk> In-Reply-To: <5023855A.2070707@cs.nott.ac.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 10:25:50 -0000 X-Keywords: X-UID: 483 Status: O Content-Length: 1583 Lines: 59 Updating git fixed this - thanks. But now am getting unsupported arch for 'armv5tel' when mirage is compiled: ++ NAME=mirage ++ VERSION=0.3dev ++ DEPS='cstruct lwt lwt.unix' ++ SYNTAX_DEPS='cstruct.syntax lwt.syntax' ++ RUNTIME=unixrun ++ LIB=oS ++ TESTS=tap_test + '[' -x configure.os ']' + . ./configure.os +++ uname -s ++ OS=Linux ++ CFLAGS='-Wall -O3' ++ case `uname -m` in +++ uname -m ++ case "$OS" in ++ ln -nsf tap_stubs_linux.c lib/tap_stubs_os.c ++ which ocamlbuild + OCAMLBUILD=/usr/local/bin/ocamlbuild ++ which ocamlfind + OCAMLFIND=/root/.opam/system/bin/ocamlfind + OCAMLBUILD_FLAGS='-classic-display -j 8' + case "$cmd" in + compile + /usr/local/bin/ocamlbuild -classic-display -j 8 mirage.all `armv5tel` is not a supported arch make[1]: *** [all] Error 255 make: *** [all] Error 2 On 09/08/2012 10:39, work wrote: > On 09/08/2012 09:57, Thomas Gazagnaire wrote: >> git archive --format=tar --prefix=cstruct.0.3.dev/ HEAD -o >> cstruct.0.3.dev.tar > Thanks... > > When I run this directily in /root/.opam/repo/dev/git/cstruct.0.3dev > the following is spit out (and no tar is created): > > pax_global_header00006660000000000000000000000064120046302300014500gustar00rootroot0000000000000052 > comment=94b03dcec6a2a8cabe3069d5ecd26aa349e4ea55 > cstruct.0.3.dev/000077500000000000000000000000001200463023000136075ustar00rootroot00000000000000 > > > Seems like it doesn't like the -o option as the following seems to > behave: > > git archive --format=tar --prefix=cstruct.0.3.dev/ HEAD> > cstruct.0.3.dev.tar > > > Think I need to upgrade git... > > Tom > From anil@recoil.org Thu Aug 09 14:49:38 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1SzT7K-0003TK-Ex (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 14:49:38 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1358082 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:45508 helo=dark.recoil.org) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with smtp id 1SzT7J-0005cE-no (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 09 Aug 2012 14:49:38 +0100 Received: (qmail 30542 invoked by uid 634); 9 Aug 2012 13:49:37 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from c-67-180-208-178.hsd1.ca.comcast.net (HELO [10.0.1.5]) (67.180.208.178) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 09 Aug 2012 14:49:36 +0100 Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: opam/mirage and the dreamplug From: Anil Madhavapeddy In-Reply-To: <50238ED5.5000409@cs.nott.ac.uk> Date: Thu, 9 Aug 2012 06:49:37 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <65BEB680-3646-4517-8771-320C1789143B@recoil.org> References: <502275CA.4070604@cs.nott.ac.uk> <0639A852-AC06-4643-BBF4-3EA04885883B@recoil.org> <5023786D.90609@cs.nott.ac.uk> <4F133FC3-6281-4B80-A41C-EA1C075578C7@gazagnaire.org> <5023855A.2070707@cs.nott.ac.uk> <50238ED5.5000409@cs.nott.ac.uk> To: work X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 13:49:38 -0000 X-Keywords: X-UID: 484 Status: O Content-Length: 1950 Lines: 71 I've pushed fixes for the configure script issues to mirage-platform and = mirage-net. Could you do an 'opam update' and see how it goes now? -anil On 9 Aug 2012, at 03:20, work wrote: > Updating git fixed this - thanks. But now am getting unsupported arch = for 'armv5tel' when mirage is compiled: >=20 > ++ NAME=3Dmirage > ++ VERSION=3D0.3dev > ++ DEPS=3D'cstruct lwt lwt.unix' > ++ SYNTAX_DEPS=3D'cstruct.syntax lwt.syntax' > ++ RUNTIME=3Dunixrun > ++ LIB=3DoS > ++ TESTS=3Dtap_test > + '[' -x configure.os ']' > + . ./configure.os > +++ uname -s > ++ OS=3DLinux > ++ CFLAGS=3D'-Wall -O3' > ++ case `uname -m` in > +++ uname -m > ++ case "$OS" in > ++ ln -nsf tap_stubs_linux.c lib/tap_stubs_os.c > ++ which ocamlbuild > + OCAMLBUILD=3D/usr/local/bin/ocamlbuild > ++ which ocamlfind > + OCAMLFIND=3D/root/.opam/system/bin/ocamlfind > + OCAMLBUILD_FLAGS=3D'-classic-display -j 8' > + case "$cmd" in > + compile > + /usr/local/bin/ocamlbuild -classic-display -j 8 mirage.all > `armv5tel` is not a supported arch > make[1]: *** [all] Error 255 > make: *** [all] Error 2 >=20 >=20 > On 09/08/2012 10:39, work wrote: >> On 09/08/2012 09:57, Thomas Gazagnaire wrote: >>> git archive --format=3Dtar --prefix=3Dcstruct.0.3.dev/ HEAD -o = cstruct.0.3.dev.tar >> Thanks... >>=20 >> When I run this directily in /root/.opam/repo/dev/git/cstruct.0.3dev = the following is spit out (and no tar is created): >>=20 >> = pax_global_header00006660000000000000000000000064120046302300014500gustar0= 0rootroot0000000000000052 = comment=3D94b03dcec6a2a8cabe3069d5ecd26aa349e4ea55 >> = cstruct.0.3.dev/000077500000000000000000000000001200463023000136075ustar00= rootroot00000000000000=20 >>=20 >> Seems like it doesn't like the -o option as the following seems to = behave: >>=20 >> git archive --format=3Dtar --prefix=3Dcstruct.0.3.dev/ HEAD> = cstruct.0.3.dev.tar >>=20 >>=20 >> Think I need to upgrade git... >>=20 >> Tom >>=20 >=20 >=20 From pgj@caesar.elte.hu Thu Aug 16 14:13:36 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T1ztI-0001zS-1N (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 16 Aug 2012 14:13:36 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.8 from SpamAssassin-3.3.2-1373273 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * 0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:54143) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1T1ztG-0002OV-ZL (Exim 4.72) (return-path ); Thu, 16 Aug 2012 14:13:35 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1T1zt9-0005Jl-Ts from ; Thu, 16 Aug 2012 15:13:31 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T1zt9-0006KT-5O; Thu, 16 Aug 2012 15:13:27 +0200 Date: Thu, 16 Aug 2012 15:13:27 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy , "Robert N. M. Watson" Subject: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120816131326.GA15479@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=AWL, BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 AWL AWL: From: address is in the auto white-list Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2012 13:13:36 -0000 X-Keywords: X-UID: 485 Status: O Content-Length: 2578 Lines: 56 Hello there, For your information, I have just added support for receiving network packets to the kFreeBSD backend of mirage-platform: https://github.com/pgj/mirage-kfreebsd/commit/c10b740218fd0af10e7b8e424d4cdde191edb251 Note that there is also a basic/netif test added to mirage-test/regress to try this out and see it in action. I would like to call for your reviews and comments. To narrate on the design choices I made, here is a brief summary on how the current implementation looks like at the moment. - After loading the built kernel module and starting up the evaluation of the Netif.create function, Mirage asks for a list of available Ethernet interfaces from the FreeBSD kernel and then assigns an lwt thread to each of them by "plugging". - Plugged interfaces are stored in a linked list and have their ng_ether(4) hook (called from ether_input()) activated and pointed to netif_ether_input(). At the same time, there is a shared ring buffer created for each of them in Mirage then passed to the C function responsible for administering the list of plugged interfaces, caml_plug_vif(). - Shared ring buffers are created as Io_pages by allocating page-aligned, contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). These are directly accessible in Mirage as character arrays. - Each shared ring buffer is currently of size 33 pages, and operates with 2048-byte slots. The buffers start with a header that maintains all the required meta information, like next position, available items, size of stored items. - Each packet arriving on any of the plugged interfaces is placed to the next available slot of the corresponding shared ring buffer with m_copydata(). - In parallel with this in Mirage, the rx_poll function is run in loop that polls for available packets in the shared ring buffer. - When rx_poll finds unprocessed packets then it runs the user-specified function on them, e.g. print the size of the packet in basic/netif. It is implemented by passing a view on the Io_page, i.e. without copying. After the user function has finished, the packet is removed from the ring. - When no packets are available on the polled interface, rx_poll sleeps for a millisecond. - Mirage is stopped when the kernel module is unloaded, which also involves unplugging the interfaces. Let me add that shared rings do not do locking at the moment, but if you are happy with the design I could proceed with implementing it. (However, I am not completely sure how to share locks between OCaml and C.) From avsm@dark.recoil.org Thu Aug 16 18:27:12 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T23qi-0000Eh-DN (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 16 Aug 2012 18:27:12 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1373273 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:7171 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T23qh-00063I-ph (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 16 Aug 2012 18:27:12 +0100 Received: (qmail 21584 invoked by uid 10000); 16 Aug 2012 17:27:10 -0000 Date: Thu, 16 Aug 2012 18:27:10 +0100 From: Anil Madhavapeddy To: PALI Gabor Janos Subject: Re: ELF Relocations and OCaml Code Message-ID: <20120816172710.GA1066@dark.recoil.org> References: <20120725101908.GA16872@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120725101908.GA16872@caesar.elte.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2012 17:27:12 -0000 X-Keywords: X-UID: 486 Status: O Content-Length: 2344 Lines: 61 On Wed, Jul 25, 2012 at 12:19:08PM +0200, PALI Gabor Janos wrote: > Hi, > > Yesterday I had a strange issue with unsupported ELF relocations while > building a kernel module out of OCaml programs for FreeBSD/amd64. I have > finally resolved the problem, but Anil suggested me to send in here for the > record. > > I am working on the FreeBSD port of the Mirage OS module. Everything seemed > to compile and link fine, until I wanted to load the module using the > kldload(8) command: > > # kldload mirage.ko > > Then the console was pounded by lot of linker errors: > > kldload: unexpected relocation type 9 > ... > kldload: unexpected relocation type 4 > link_elf_obj: symbol camlPervasives__entry undefined > linker_load_file: Unsupported file type > > kldload: can't load mirage.ko: Exec format error > > By browsing the kernel sources, it turned out that the 64-bit FreeBSD kernel > does not support ELF relocation types 9 (R_X86_64_GOTPCREL), 10 (R_X86_32) and > 4 (R_X86_64_PLT32) (among many others). > > So, in order to get rid of these, I had to compile all my OCaml sources as > position-dependent code, without support for dynamic linking. This means that > ocamlopt must be called with the following flags: -fno-PIC and -nodynlink. > > In addition, a custom OCaml stdlib has to be built in the same fashion and > used when compiling the sources. (By the way, I use -output-obj to get a > large C object file with all the referenced code, so it could be linked into > the kernel module.) This can be overridden by setting the CAMLLIB > environmental variable. > > However, note that, in the OCaml run-time sources (which gets finally linked > to the generated object code) the file called amd64.S should be also modified > to remove the last pieces of PIC addressing. That is, the lines > > #define GREL(r) r@GOTPCREL > #define GCALL(r) r@PLT > > should be rewritten as > > #define GREL(r) r > #define GCALL(r) r This change in amd64.S came from: http://caml.inria.fr/mantis/view.php?id=4795 ...which added PIC support for amd64 around ocaml-3.12. It looks like the way to fix it via a new SYS_kfreeBSD #define, as the choice of runtime can't be dynamically toggled based on -fPIC at the moment. That would be a bigger patch, and probably best left to the OCaml team to decide if it's necessary or not. -anil From avsm@dark.recoil.org Thu Aug 16 18:32:28 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T23vo-0000Ks-Tb (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 16 Aug 2012 18:32:28 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1373273 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:18772 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T23vo-0007Yz-q8 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 16 Aug 2012 18:32:28 +0100 Received: (qmail 6634 invoked by uid 10000); 16 Aug 2012 17:32:28 -0000 Date: Thu, 16 Aug 2012 18:32:27 +0100 From: Anil Madhavapeddy To: PALI Gabor Janos Subject: Re: about "hello mirage world" Message-ID: <20120816173227.GB1066@dark.recoil.org> References: <201207271441008347713@gmail.com> <20120731131926.GA7254@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120731131926.GA7254@caesar.elte.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: cl-mirage , Yiming Zhang X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2012 17:32:29 -0000 X-Keywords: X-UID: 487 Status: O Content-Length: 1887 Lines: 45 On Tue, Jul 31, 2012 at 03:19:26PM +0200, PALI Gabor Janos wrote: > Hi, > > On Fri, Jul 27, 2012 at 02:41:01PM +0100, Yiming Zhang wrote: > > First, it is required to "have mir-xen in your PATH", but where mir-xen is? > > And second, the first step is to "cd mirage.git/regress", but where > > mirage.git is? > > I think it is because the documentation has not been updated. There was a > major reorganization for the Mirage sources: mirage.git has been split into > multiple smaller repositories that now you can find under mirage.github.com > (and install via using OPAM). > > The old mirage.git repository lives at Anil's github's profile, that is where > you could find the referenced files: > > http://github.com/avsm/mirage.git > > However, I do not know whether it would make any sense to use them any more. Yes indeed. We're currently breaking up Mirage into a series of smaller libraries that are easier to integrate as libraries (both for microkernel use, and for general programming under UNIX). So the answer to your question really depends on what you're trying to do with Mirage. If you want to play with microkernels and Xen and experiment, then monolithic repository that Gabor points out in: http://github.com/avsm/mirage.git ...works perfectly fine, and the documentation at: http://tutorial.openmirage.org/#1 should mostly be correct. All new development is happening in the broken up repositories, and we're aiming for a soft launch just before XenSummit in a couple of weeks. http://xen.org/xensummit/xs12na_talks/xensummit.html I'm afraid the broken-up new system will be somewhat subject to change before then. Thomas is finishing up a major rev to OPAM which will improve its stability with the 0.4 release, but there will likely be some short-term fallout as a result. -- Anil Madhavapeddy http://anil.recoil.org From sdiris@gmail.com Fri Aug 17 10:40:34 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T2J2g-0003b5-1q (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 17 Aug 2012 10:40:34 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.9 from SpamAssassin-3.3.2-1373758 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [74.125.83.51 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (sdiris[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from * author's domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-ee0-f51.google.com ([74.125.83.51]:39323) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T2J2e-0003A0-qo (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 17 Aug 2012 10:40:33 +0100 Received: by eeke50 with SMTP id e50so884250eek.38 for ; Fri, 17 Aug 2012 02:40:32 -0700 (PDT) Received: by 10.14.203.73 with SMTP id e49mr5487215eeo.27.1345196432094; Fri, 17 Aug 2012 02:40:32 -0700 (PDT) Received: from YimingZhang-PC (c149.al.cl.cam.ac.uk. [128.232.110.149]) by mx.google.com with ESMTPS id h2sm18708618eeo.3.2012.08.17.02.40.30 (version=SSLv3 cipher=OTHER); Fri, 17 Aug 2012 02:40:31 -0700 (PDT) Date: Fri, 17 Aug 2012 10:40:33 +0100 From: "Yiming Zhang" To: "cl-mirage" References: Subject: Re: Cl-mirage Digest, Vol 7, Issue 12 Message-ID: <201208171040328786436@gmail.com> X-mailer: Foxmail 6, 15, 201, 26 [cn] Mime-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list Reply-To: sdiris@gmail.com List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 09:40:34 -0000 X-Keywords: X-UID: 488 Status: O Content-Length: 12444 Lines: 164 VGhhbmtzIHRvIEdhYm9yIGFuZCBBbmlsIQ0KDQotLS0tLS0tLS0tLS0tLS0tLS0JCQkJIA0KWWlt aW5nIFpoYW5nDQoyMDEyLTA4LTE3DQoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCreivP7Iy6O6Y2wtbWlyYWdlLXJlcXVlc3QN Creiy83I1cbao7oyMDEyLTA4LTE3IDA0OjAwOjEyDQrK1bz+yMujumNsLW1pcmFnZQ0Ks63LzaO6 DQrW98zio7pDbC1taXJhZ2UgRGlnZXN0LCBWb2wgNywgSXNzdWUgMTINCg0KU2VuZCBDbC1taXJh Z2UgbWFpbGluZyBsaXN0IHN1Ym1pc3Npb25zIHRvDQoJY2wtbWlyYWdlQGxpc3RzLmNhbS5hYy51 aw0KDQpUbyBzdWJzY3JpYmUgb3IgdW5zdWJzY3JpYmUgdmlhIHRoZSBXb3JsZCBXaWRlIFdlYiwg dmlzaXQNCglodHRwczovL2xpc3RzLmNhbS5hYy51ay9tYWlsbWFuL2xpc3RpbmZvL2NsLW1pcmFn ZQ0Kb3IsIHZpYSBlbWFpbCwgc2VuZCBhIG1lc3NhZ2Ugd2l0aCBzdWJqZWN0IG9yIGJvZHkgJ2hl bHAnIHRvDQoJY2wtbWlyYWdlLXJlcXVlc3RAbGlzdHMuY2FtLmFjLnVrDQoNCllvdSBjYW4gcmVh Y2ggdGhlIHBlcnNvbiBtYW5hZ2luZyB0aGUgbGlzdCBhdA0KCWNsLW1pcmFnZS1vd25lckBsaXN0 cy5jYW0uYWMudWsNCg0KV2hlbiByZXBseWluZywgcGxlYXNlIGVkaXQgeW91ciBTdWJqZWN0IGxp bmUgc28gaXQgaXMgbW9yZSBzcGVjaWZpYw0KdGhhbiAiUmU6IENvbnRlbnRzIG9mIENsLW1pcmFn ZSBkaWdlc3QuLi4iDQoNCg0KVG9kYXkncyBUb3BpY3M6DQoNCiAgIDEuIFJlY2VpdmluZyBOZXR3 b3JrIFBhY2tldHMgd2l0aCBNaXJhZ2Uva0ZyZWVCU0QgKFBBTEkgR2Fib3IgSmFub3MpDQogICAy LiBSZTogRUxGIFJlbG9jYXRpb25zIGFuZCBPQ2FtbCBDb2RlIChBbmlsIE1hZGhhdmFwZWRkeSkN CiAgIDMuIFJlOiBhYm91dCAiaGVsbG8gbWlyYWdlIHdvcmxkIiAoQW5pbCBNYWRoYXZhcGVkZHkp DQoNCg0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQpNZXNzYWdlOiAxDQpEYXRlOiBUaHUsIDE2IEF1ZyAyMDEy IDE1OjEzOjI3ICswMjAwDQpGcm9tOiBQQUxJIEdhYm9yIEphbm9zIDxwZ2pAZWx0ZS5odT4NClN1 YmplY3Q6IFJlY2VpdmluZyBOZXR3b3JrIFBhY2tldHMgd2l0aCBNaXJhZ2Uva0ZyZWVCU0QNClRv OiBBbmlsIE1hZGhhdmFwZWRkeSA8YW5pbEByZWNvaWwub3JnPiwJIlJvYmVydCBOLiBNLiBXYXRz b24iDQoJPHJvYmVydC53YXRzb25AY2wuY2FtLmFjLnVrPg0KQ2M6IGNsLW1pcmFnZUBsaXN0cy5j YW0uYWMudWsNCk1lc3NhZ2UtSUQ6IDwyMDEyMDgxNjEzMTMyNi5HQTE1NDc5QGNhZXNhci5lbHRl Lmh1Pg0KQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFyc2V0PXVzLWFzY2lpDQoNCkhlbGxv IHRoZXJlLA0KDQpGb3IgeW91ciBpbmZvcm1hdGlvbiwgSSBoYXZlIGp1c3QgYWRkZWQgc3VwcG9y dCBmb3IgcmVjZWl2aW5nIG5ldHdvcmsgcGFja2V0cw0KdG8gdGhlIGtGcmVlQlNEIGJhY2tlbmQg b2YgbWlyYWdlLXBsYXRmb3JtOg0KDQpodHRwczovL2dpdGh1Yi5jb20vcGdqL21pcmFnZS1rZnJl ZWJzZC9jb21taXQvYzEwYjc0MDIxOGZkMGFmMTBlN2I4ZTQyNGQ0Y2RkZTE5MWVkYjI1MQ0KDQpO b3RlIHRoYXQgdGhlcmUgaXMgYWxzbyBhIGJhc2ljL25ldGlmIHRlc3QgYWRkZWQgdG8gbWlyYWdl LXRlc3QvcmVncmVzcyB0bw0KdHJ5IHRoaXMgb3V0IGFuZCBzZWUgaXQgaW4gYWN0aW9uLg0KDQpJ IHdvdWxkIGxpa2UgdG8gY2FsbCBmb3IgeW91ciByZXZpZXdzIGFuZCBjb21tZW50cy4gIFRvIG5h cnJhdGUgb24gdGhlIGRlc2lnbg0KY2hvaWNlcyBJIG1hZGUsIGhlcmUgaXMgYSBicmllZiBzdW1t YXJ5IG9uIGhvdyB0aGUgY3VycmVudCBpbXBsZW1lbnRhdGlvbg0KbG9va3MgbGlrZSBhdCB0aGUg bW9tZW50Lg0KDQotIEFmdGVyIGxvYWRpbmcgdGhlIGJ1aWx0IGtlcm5lbCBtb2R1bGUgYW5kIHN0 YXJ0aW5nIHVwIHRoZSBldmFsdWF0aW9uIG9mDQogIHRoZSBOZXRpZi5jcmVhdGUgZnVuY3Rpb24s IE1pcmFnZSBhc2tzIGZvciBhIGxpc3Qgb2YgYXZhaWxhYmxlIEV0aGVybmV0DQogIGludGVyZmFj ZXMgZnJvbSB0aGUgRnJlZUJTRCBrZXJuZWwgYW5kIHRoZW4gYXNzaWducyBhbiBsd3QgdGhyZWFk IHRvIGVhY2ggb2YNCiAgdGhlbSBieSAicGx1Z2dpbmciLg0KDQotIFBsdWdnZWQgaW50ZXJmYWNl cyBhcmUgc3RvcmVkIGluIGEgbGlua2VkIGxpc3QgYW5kIGhhdmUgdGhlaXIgbmdfZXRoZXIoNCkN CiAgaG9vayAoY2FsbGVkIGZyb20gZXRoZXJfaW5wdXQoKSkgYWN0aXZhdGVkIGFuZCBwb2ludGVk IHRvDQogIG5ldGlmX2V0aGVyX2lucHV0KCkuICBBdCB0aGUgc2FtZSB0aW1lLCB0aGVyZSBpcyBh IHNoYXJlZCByaW5nIGJ1ZmZlcg0KICBjcmVhdGVkIGZvciBlYWNoIG9mIHRoZW0gaW4gTWlyYWdl IHRoZW4gcGFzc2VkIHRvIHRoZSBDIGZ1bmN0aW9uDQogIHJlc3BvbnNpYmxlIGZvciBhZG1pbmlz dGVyaW5nIHRoZSBsaXN0IG9mIHBsdWdnZWQgaW50ZXJmYWNlcywNCiAgY2FtbF9wbHVnX3ZpZigp Lg0KDQotIFNoYXJlZCByaW5nIGJ1ZmZlcnMgYXJlIGNyZWF0ZWQgYXMgSW9fcGFnZXMgYnkgYWxs b2NhdGluZyBwYWdlLWFsaWduZWQsDQogIGNvbnRpZ3VvdXMsIG11bHRpLXBhZ2UgbWVtb3J5IGFy ZWFzIHZpYSBGcmVlQlNEJ3MgY29udGlnbWFsbG9jKDkpLiAgVGhlc2UNCiAgYXJlIGRpcmVjdGx5 IGFjY2Vzc2libGUgaW4gTWlyYWdlIGFzIGNoYXJhY3RlciBhcnJheXMuDQoNCi0gRWFjaCBzaGFy ZWQgcmluZyBidWZmZXIgaXMgY3VycmVudGx5IG9mIHNpemUgMzMgcGFnZXMsIGFuZCBvcGVyYXRl cyB3aXRoDQogIDIwNDgtYnl0ZSBzbG90cy4gIFRoZSBidWZmZXJzIHN0YXJ0IHdpdGggYSBoZWFk ZXIgdGhhdCBtYWludGFpbnMgYWxsIHRoZQ0KICByZXF1aXJlZCBtZXRhIGluZm9ybWF0aW9uLCBs aWtlIG5leHQgcG9zaXRpb24sIGF2YWlsYWJsZSBpdGVtcywgc2l6ZSBvZg0KICBzdG9yZWQgaXRl bXMuDQoNCi0gRWFjaCBwYWNrZXQgYXJyaXZpbmcgb24gYW55IG9mIHRoZSBwbHVnZ2VkIGludGVy ZmFjZXMgaXMgcGxhY2VkIHRvIHRoZSBuZXh0DQogIGF2YWlsYWJsZSBzbG90IG9mIHRoZSBjb3Jy ZXNwb25kaW5nIHNoYXJlZCByaW5nIGJ1ZmZlciB3aXRoIG1fY29weWRhdGEoKS4NCg0KLSBJbiBw YXJhbGxlbCB3aXRoIHRoaXMgaW4gTWlyYWdlLCB0aGUgcnhfcG9sbCBmdW5jdGlvbiBpcyBydW4g aW4gbG9vcCB0aGF0DQogIHBvbGxzIGZvciBhdmFpbGFibGUgcGFja2V0cyBpbiB0aGUgc2hhcmVk IHJpbmcgYnVmZmVyLg0KICANCi0gV2hlbiByeF9wb2xsIGZpbmRzIHVucHJvY2Vzc2VkIHBhY2tl dHMgdGhlbiBpdCBydW5zIHRoZSB1c2VyLXNwZWNpZmllZA0KICBmdW5jdGlvbiBvbiB0aGVtLCBl LmcuIHByaW50IHRoZSBzaXplIG9mIHRoZSBwYWNrZXQgaW4gYmFzaWMvbmV0aWYuICBJdCBpcw0K ICBpbXBsZW1lbnRlZCBieSBwYXNzaW5nIGEgdmlldyBvbiB0aGUgSW9fcGFnZSwgaS5lLiB3aXRo b3V0IGNvcHlpbmcuICBBZnRlcg0KICB0aGUgdXNlciBmdW5jdGlvbiBoYXMgZmluaXNoZWQsIHRo ZSBwYWNrZXQgaXMgcmVtb3ZlZCBmcm9tIHRoZSByaW5nLg0KDQotIFdoZW4gbm8gcGFja2V0cyBh cmUgYXZhaWxhYmxlIG9uIHRoZSBwb2xsZWQgaW50ZXJmYWNlLCByeF9wb2xsIHNsZWVwcyBmb3Ig YQ0KICBtaWxsaXNlY29uZC4NCg0KLSBNaXJhZ2UgaXMgc3RvcHBlZCB3aGVuIHRoZSBrZXJuZWwg bW9kdWxlIGlzIHVubG9hZGVkLCB3aGljaCBhbHNvIGludm9sdmVzDQogIHVucGx1Z2dpbmcgdGhl IGludGVyZmFjZXMuDQoNCkxldCBtZSBhZGQgdGhhdCBzaGFyZWQgcmluZ3MgZG8gbm90IGRvIGxv Y2tpbmcgYXQgdGhlIG1vbWVudCwgYnV0IGlmIHlvdSBhcmUNCmhhcHB5IHdpdGggdGhlIGRlc2ln biBJIGNvdWxkIHByb2NlZWQgd2l0aCBpbXBsZW1lbnRpbmcgaXQuICAoSG93ZXZlciwgSSBhbQ0K bm90IGNvbXBsZXRlbHkgc3VyZSBob3cgdG8gc2hhcmUgbG9ja3MgYmV0d2VlbiBPQ2FtbCBhbmQg Qy4pDQoNCg0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCg0KTWVzc2FnZTogMg0K RGF0ZTogVGh1LCAxNiBBdWcgMjAxMiAxODoyNzoxMCArMDEwMA0KRnJvbTogQW5pbCBNYWRoYXZh cGVkZHkgPGFuaWxAcmVjb2lsLm9yZz4NClN1YmplY3Q6IFJlOiBFTEYgUmVsb2NhdGlvbnMgYW5k IE9DYW1sIENvZGUNClRvOiBQQUxJIEdhYm9yIEphbm9zIDxwZ2pAZWx0ZS5odT4NCkNjOiBjbC1t aXJhZ2VAbGlzdHMuY2FtLmFjLnVrDQpNZXNzYWdlLUlEOiA8MjAxMjA4MTYxNzI3MTAuR0ExMDY2 QGRhcmsucmVjb2lsLm9yZz4NCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD11cy1h c2NpaQ0KDQpPbiBXZWQsIEp1bCAyNSwgMjAxMiBhdCAxMjoxOTowOFBNICswMjAwLCBQQUxJIEdh Ym9yIEphbm9zIHdyb3RlOg0KPiBIaSwNCj4gDQo+IFllc3RlcmRheSBJIGhhZCBhIHN0cmFuZ2Ug aXNzdWUgd2l0aCB1bnN1cHBvcnRlZCBFTEYgcmVsb2NhdGlvbnMgd2hpbGUNCj4gYnVpbGRpbmcg YSBrZXJuZWwgbW9kdWxlIG91dCBvZiBPQ2FtbCBwcm9ncmFtcyBmb3IgRnJlZUJTRC9hbWQ2NC4g IEkgaGF2ZQ0KPiBmaW5hbGx5IHJlc29sdmVkIHRoZSBwcm9ibGVtLCBidXQgQW5pbCBzdWdnZXN0 ZWQgbWUgdG8gc2VuZCBpbiBoZXJlIGZvciB0aGUNCj4gcmVjb3JkLg0KPiANCj4gSSBhbSB3b3Jr aW5nIG9uIHRoZSBGcmVlQlNEIHBvcnQgb2YgdGhlIE1pcmFnZSBPUyBtb2R1bGUuICBFdmVyeXRo aW5nIHNlZW1lZA0KPiB0byBjb21waWxlIGFuZCBsaW5rIGZpbmUsIHVudGlsIEkgd2FudGVkIHRv IGxvYWQgdGhlIG1vZHVsZSB1c2luZyB0aGUNCj4ga2xkbG9hZCg4KSBjb21tYW5kOg0KPiANCj4g IyBrbGRsb2FkIG1pcmFnZS5rbw0KPiANCj4gVGhlbiB0aGUgY29uc29sZSB3YXMgcG91bmRlZCBi eSBsb3Qgb2YgbGlua2VyIGVycm9yczoNCj4gDQo+IGtsZGxvYWQ6IHVuZXhwZWN0ZWQgcmVsb2Nh dGlvbiB0eXBlIDkNCj4gLi4uDQo+IGtsZGxvYWQ6IHVuZXhwZWN0ZWQgcmVsb2NhdGlvbiB0eXBl IDQNCj4gbGlua19lbGZfb2JqOiBzeW1ib2wgY2FtbFBlcnZhc2l2ZXNfX2VudHJ5IHVuZGVmaW5l ZA0KPiBsaW5rZXJfbG9hZF9maWxlOiBVbnN1cHBvcnRlZCBmaWxlIHR5cGUNCj4gDQo+IGtsZGxv YWQ6IGNhbid0IGxvYWQgbWlyYWdlLmtvOiBFeGVjIGZvcm1hdCBlcnJvcg0KPiANCj4gQnkgYnJv d3NpbmcgdGhlIGtlcm5lbCBzb3VyY2VzLCBpdCB0dXJuZWQgb3V0IHRoYXQgdGhlIDY0LWJpdCBG cmVlQlNEIGtlcm5lbA0KPiBkb2VzIG5vdCBzdXBwb3J0IEVMRiByZWxvY2F0aW9uIHR5cGVzIDkg KFJfWDg2XzY0X0dPVFBDUkVMKSwgMTAgKFJfWDg2XzMyKSBhbmQNCj4gNCAoUl9YODZfNjRfUExU MzIpIChhbW9uZyBtYW55IG90aGVycykuDQo+IA0KPiBTbywgaW4gb3JkZXIgdG8gZ2V0IHJpZCBv ZiB0aGVzZSwgSSBoYWQgdG8gY29tcGlsZSBhbGwgbXkgT0NhbWwgc291cmNlcyBhcw0KPiBwb3Np dGlvbi1kZXBlbmRlbnQgY29kZSwgd2l0aG91dCBzdXBwb3J0IGZvciBkeW5hbWljIGxpbmtpbmcu ICBUaGlzIG1lYW5zIHRoYXQNCj4gb2NhbWxvcHQgbXVzdCBiZSBjYWxsZWQgd2l0aCB0aGUgZm9s bG93aW5nIGZsYWdzOiAtZm5vLVBJQyBhbmQgLW5vZHlubGluay4NCj4gDQo+IEluIGFkZGl0aW9u LCBhIGN1c3RvbSBPQ2FtbCBzdGRsaWIgaGFzIHRvIGJlIGJ1aWx0IGluIHRoZSBzYW1lIGZhc2hp b24gYW5kDQo+IHVzZWQgd2hlbiBjb21waWxpbmcgdGhlIHNvdXJjZXMuICAoQnkgdGhlIHdheSwg SSB1c2UgLW91dHB1dC1vYmogdG8gZ2V0IGENCj4gbGFyZ2UgQyBvYmplY3QgZmlsZSB3aXRoIGFs bCB0aGUgcmVmZXJlbmNlZCBjb2RlLCBzbyBpdCBjb3VsZCBiZSBsaW5rZWQgaW50bw0KPiB0aGUg a2VybmVsIG1vZHVsZS4pICBUaGlzIGNhbiBiZSBvdmVycmlkZGVuIGJ5IHNldHRpbmcgdGhlIENB TUxMSUINCj4gZW52aXJvbm1lbnRhbCB2YXJpYWJsZS4NCj4gDQo+IEhvd2V2ZXIsIG5vdGUgdGhh dCwgaW4gdGhlIE9DYW1sIHJ1bi10aW1lIHNvdXJjZXMgKHdoaWNoIGdldHMgZmluYWxseSBsaW5r ZWQNCj4gdG8gdGhlIGdlbmVyYXRlZCBvYmplY3QgY29kZSkgdGhlIGZpbGUgY2FsbGVkIGFtZDY0 LlMgc2hvdWxkIGJlIGFsc28gbW9kaWZpZWQNCj4gdG8gcmVtb3ZlIHRoZSBsYXN0IHBpZWNlcyBv ZiBQSUMgYWRkcmVzc2luZy4gIFRoYXQgaXMsIHRoZSBsaW5lcw0KPiANCj4gI2RlZmluZSBHUkVM KHIpIHJAR09UUENSRUwNCj4gI2RlZmluZSBHQ0FMTChyKSByQFBMVA0KPiANCj4gc2hvdWxkIGJl IHJld3JpdHRlbiBhcw0KPiANCj4gI2RlZmluZSBHUkVMKHIpIHINCj4gI2RlZmluZSBHQ0FMTChy KSByDQoNClRoaXMgY2hhbmdlIGluIGFtZDY0LlMgY2FtZSBmcm9tOg0KaHR0cDovL2NhbWwuaW5y aWEuZnIvbWFudGlzL3ZpZXcucGhwP2lkPTQ3OTUNCi4uLndoaWNoIGFkZGVkIFBJQyBzdXBwb3J0 IGZvciBhbWQ2NCBhcm91bmQgb2NhbWwtMy4xMi4NCg0KSXQgbG9va3MgbGlrZSB0aGUgd2F5IHRv IGZpeCBpdCB2aWEgYSBuZXcgU1lTX2tmcmVlQlNEICNkZWZpbmUsIGFzIHRoZQ0KY2hvaWNlIG9m IHJ1bnRpbWUgY2FuJ3QgYmUgZHluYW1pY2FsbHkgdG9nZ2xlZCBiYXNlZCBvbiAtZlBJQyBhdCB0 aGUNCm1vbWVudC4gIFRoYXQgd291bGQgYmUgYSBiaWdnZXIgcGF0Y2gsIGFuZCBwcm9iYWJseSBi ZXN0IGxlZnQgdG8gdGhlIE9DYW1sDQp0ZWFtIHRvIGRlY2lkZSBpZiBpdCdzIG5lY2Vzc2FyeSBv ciBub3QuDQoNCi1hbmlsDQoNCg0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCg0K TWVzc2FnZTogMw0KRGF0ZTogVGh1LCAxNiBBdWcgMjAxMiAxODozMjoyNyArMDEwMA0KRnJvbTog QW5pbCBNYWRoYXZhcGVkZHkgPGFuaWxAcmVjb2lsLm9yZz4NClN1YmplY3Q6IFJlOiBhYm91dCAi aGVsbG8gbWlyYWdlIHdvcmxkIg0KVG86IFBBTEkgR2Fib3IgSmFub3MgPHBnakBlbHRlLmh1Pg0K Q2M6IGNsLW1pcmFnZSA8Y2wtbWlyYWdlQGxpc3RzLmNhbS5hYy51az4sIFlpbWluZyBaaGFuZw0K CTxzZGlyaXNAZ21haWwuY29tPg0KTWVzc2FnZS1JRDogPDIwMTIwODE2MTczMjI3LkdCMTA2NkBk YXJrLnJlY29pbC5vcmc+DQpDb250ZW50LVR5cGU6IHRleHQvcGxhaW47IGNoYXJzZXQ9dXMtYXNj aWkNCg0KT24gVHVlLCBKdWwgMzEsIDIwMTIgYXQgMDM6MTk6MjZQTSArMDIwMCwgUEFMSSBHYWJv ciBKYW5vcyB3cm90ZToNCj4gSGksDQo+IA0KPiBPbiBGcmksIEp1bCAyNywgMjAxMiBhdCAwMjo0 MTowMVBNICswMTAwLCBZaW1pbmcgWmhhbmcgd3JvdGU6DQo+ID4gRmlyc3QsIGl0IGlzIHJlcXVp cmVkIHRvICJoYXZlIG1pci14ZW4gaW4geW91ciBQQVRIIiwgYnV0IHdoZXJlIG1pci14ZW4gaXM/ DQo+ID4gQW5kIHNlY29uZCwgdGhlIGZpcnN0IHN0ZXAgaXMgdG8gImNkIG1pcmFnZS5naXQvcmVn cmVzcyIsIGJ1dCB3aGVyZQ0KPiA+IG1pcmFnZS5naXQgaXM/DQo+IA0KPiBJIHRoaW5rIGl0IGlz IGJlY2F1c2UgdGhlIGRvY3VtZW50YXRpb24gaGFzIG5vdCBiZWVuIHVwZGF0ZWQuICBUaGVyZSB3 YXMgYQ0KPiBtYWpvciByZW9yZ2FuaXphdGlvbiBmb3IgdGhlIE1pcmFnZSBzb3VyY2VzOiBtaXJh Z2UuZ2l0IGhhcyBiZWVuIHNwbGl0IGludG8NCj4gbXVsdGlwbGUgc21hbGxlciByZXBvc2l0b3Jp ZXMgdGhhdCBub3cgeW91IGNhbiBmaW5kIHVuZGVyIG1pcmFnZS5naXRodWIuY29tDQo+IChhbmQg aW5zdGFsbCB2aWEgdXNpbmcgT1BBTSkuDQo+IA0KPiBUaGUgb2xkIG1pcmFnZS5naXQgcmVwb3Np dG9yeSBsaXZlcyBhdCBBbmlsJ3MgZ2l0aHViJ3MgcHJvZmlsZSwgdGhhdCBpcyB3aGVyZQ0KPiB5 b3UgY291bGQgZmluZCB0aGUgcmVmZXJlbmNlZCBmaWxlczoNCj4gDQo+IGh0dHA6Ly9naXRodWIu Y29tL2F2c20vbWlyYWdlLmdpdA0KPiANCj4gSG93ZXZlciwgSSBkbyBub3Qga25vdyB3aGV0aGVy IGl0IHdvdWxkIG1ha2UgYW55IHNlbnNlIHRvIHVzZSB0aGVtIGFueSBtb3JlLg0KDQpZZXMgaW5k ZWVkLiBXZSdyZSBjdXJyZW50bHkgYnJlYWtpbmcgdXAgTWlyYWdlIGludG8gYSBzZXJpZXMgb2Yg c21hbGxlcg0KbGlicmFyaWVzIHRoYXQgYXJlIGVhc2llciB0byBpbnRlZ3JhdGUgYXMgbGlicmFy aWVzIChib3RoIGZvciBtaWNyb2tlcm5lbA0KdXNlLCBhbmQgZm9yIGdlbmVyYWwgcHJvZ3JhbW1p bmcgdW5kZXIgVU5JWCkuICANCg0KU28gdGhlIGFuc3dlciB0byB5b3VyIHF1ZXN0aW9uIHJlYWxs eSBkZXBlbmRzIG9uIHdoYXQgeW91J3JlIHRyeWluZyB0byBkbw0Kd2l0aCBNaXJhZ2UuICBJZiB5 b3Ugd2FudCB0byBwbGF5IHdpdGggbWljcm9rZXJuZWxzIGFuZCBYZW4gYW5kDQpleHBlcmltZW50 LCB0aGVuIG1vbm9saXRoaWMgcmVwb3NpdG9yeSB0aGF0IEdhYm9yIHBvaW50cyBvdXQgaW46DQpo dHRwOi8vZ2l0aHViLmNvbS9hdnNtL21pcmFnZS5naXQNCg0KLi4ud29ya3MgcGVyZmVjdGx5IGZp bmUsIGFuZCB0aGUgZG9jdW1lbnRhdGlvbiBhdDoNCmh0dHA6Ly90dXRvcmlhbC5vcGVubWlyYWdl Lm9yZy8jMQ0Kc2hvdWxkIG1vc3RseSBiZSBjb3JyZWN0Lg0KDQpBbGwgbmV3IGRldmVsb3BtZW50 IGlzIGhhcHBlbmluZyBpbiB0aGUgYnJva2VuIHVwIHJlcG9zaXRvcmllcywgYW5kIHdlJ3JlDQph aW1pbmcgZm9yIGEgc29mdCBsYXVuY2gganVzdCBiZWZvcmUgWGVuU3VtbWl0IGluIGEgY291cGxl IG9mIHdlZWtzLg0KaHR0cDovL3hlbi5vcmcveGVuc3VtbWl0L3hzMTJuYV90YWxrcy94ZW5zdW1t aXQuaHRtbA0KDQpJJ20gYWZyYWlkIHRoZSBicm9rZW4tdXAgbmV3IHN5c3RlbSB3aWxsIGJlIHNv bWV3aGF0IHN1YmplY3QgdG8gY2hhbmdlDQpiZWZvcmUgdGhlbi4gVGhvbWFzIGlzIGZpbmlzaGlu ZyB1cCBhIG1ham9yIHJldiB0byBPUEFNIHdoaWNoIHdpbGwgaW1wcm92ZQ0KaXRzIHN0YWJpbGl0 eSB3aXRoIHRoZSAwLjQgcmVsZWFzZSwgYnV0IHRoZXJlIHdpbGwgbGlrZWx5IGJlIHNvbWUNCnNo b3J0LXRlcm0gZmFsbG91dCBhcyBhIHJlc3VsdC4NCg0KLS0gDQpBbmlsIE1hZGhhdmFwZWRkeSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGh0dHA6Ly9hbmlsLnJlY29pbC5vcmcNCg0K DQoNCkVuZCBvZiBDbC1taXJhZ2UgRGlnZXN0LCBWb2wgNywgSXNzdWUgMTINCioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCg== From Richard.Mortier@nottingham.ac.uk Sun Aug 19 19:19:56 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3A6O-0005ga-Tz (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 19 Aug 2012 19:19:56 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374176 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:60319 helo=ixe-mta-19.emailfiltering.com) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1T3A6I-00069O-SW (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 19 Aug 2012 19:19:56 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2346018776 ;de840d86be9c036a; Sun, 19 Aug 2012 19:19:30 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp4.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T3A5y-0000pS-Fk; Sun, 19 Aug 2012 19:19:30 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Sun, 19 Aug 2012 19:19:00 +0100 From: Richard Mortier To: PALI Gabor Janos Date: Sun, 19 Aug 2012 19:19:00 +0100 Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Thread-Topic: Receiving Network Packets with Mirage/kFreeBSD Thread-Index: Ac1+Nxsiff8FDc1cTS+jt34/3bMEuA== Message-ID: <87E2E550-97C1-4392-8F29-ACD1B395B434@nottingham.ac.uk> References: <20120816131326.GA15479@caesar.elte.hu> In-Reply-To: <20120816131326.GA15479@caesar.elte.hu> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "Robert N. M. Watson" , "cl-mirage@lists.cam.ac.uk" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2012 18:19:57 -0000 X-Keywords: X-UID: 489 Status: O Content-Length: 3852 Lines: 104 out of curiosity, and from a *very* naive viewpoint (so feel free to explai= n why this is a dumb question), how applicable would the same structures be= to supporting disk/other io access? (i don't have a particular use-case in mind, just asking.) On 16 Aug 2012, at 16:13, PALI Gabor Janos wrote: > Hello there, >=20 > For your information, I have just added support for receiving network pac= kets > to the kFreeBSD backend of mirage-platform: >=20 > https://github.com/pgj/mirage-kfreebsd/commit/c10b740218fd0af10e7b8e424d4= cdde191edb251 >=20 > Note that there is also a basic/netif test added to mirage-test/regress t= o > try this out and see it in action. >=20 > I would like to call for your reviews and comments. To narrate on the de= sign > choices I made, here is a brief summary on how the current implementation > looks like at the moment. >=20 > - After loading the built kernel module and starting up the evaluation of > the Netif.create function, Mirage asks for a list of available Ethernet > interfaces from the FreeBSD kernel and then assigns an lwt thread to eac= h of > them by "plugging". >=20 > - Plugged interfaces are stored in a linked list and have their ng_ether(= 4) > hook (called from ether_input()) activated and pointed to > netif_ether_input(). At the same time, there is a shared ring buffer > created for each of them in Mirage then passed to the C function > responsible for administering the list of plugged interfaces, > caml_plug_vif(). >=20 > - Shared ring buffers are created as Io_pages by allocating page-aligned, > contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). Thes= e > are directly accessible in Mirage as character arrays. >=20 > - Each shared ring buffer is currently of size 33 pages, and operates wit= h > 2048-byte slots. The buffers start with a header that maintains all the > required meta information, like next position, available items, size of > stored items. >=20 > - Each packet arriving on any of the plugged interfaces is placed to the = next > available slot of the corresponding shared ring buffer with m_copydata()= . >=20 > - In parallel with this in Mirage, the rx_poll function is run in loop th= at > polls for available packets in the shared ring buffer. >=20 > - When rx_poll finds unprocessed packets then it runs the user-specified > function on them, e.g. print the size of the packet in basic/netif. It = is > implemented by passing a view on the Io_page, i.e. without copying. Aft= er > the user function has finished, the packet is removed from the ring. >=20 > - When no packets are available on the polled interface, rx_poll sleeps f= or a > millisecond. >=20 > - Mirage is stopped when the kernel module is unloaded, which also involv= es > unplugging the interfaces. >=20 > Let me add that shared rings do not do locking at the moment, but if you = are > happy with the design I could proceed with implementing it. (However, I = am > not completely sure how to share locks between OCaml and C.) >=20 --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From pgj@caesar.elte.hu Mon Aug 20 14:46:45 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3SJZ-0006EQ-J8 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 20 Aug 2012 14:46:45 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -1.0 from SpamAssassin-3.3.2-1374711 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:59607) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1T3SJY-00045N-Rn (Exim 4.72) (return-path ); Mon, 20 Aug 2012 14:46:45 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1T3SJW-0007gM-KF from ; Mon, 20 Aug 2012 15:46:42 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T3SJV-0005Ln-9p; Mon, 20 Aug 2012 15:46:41 +0200 Date: Mon, 20 Aug 2012 15:46:41 +0200 From: PALI Gabor Janos To: Richard Mortier Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120820134641.GA18444@caesar.elte.hu> References: <20120816131326.GA15479@caesar.elte.hu> <87E2E550-97C1-4392-8F29-ACD1B395B434@nottingham.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87E2E550-97C1-4392-8F29-ACD1B395B434@nottingham.ac.uk> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: "Robert N. M. Watson" , "cl-mirage@lists.cam.ac.uk" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 13:46:45 -0000 X-Keywords: X-UID: 490 Status: O Content-Length: 826 Lines: 18 On Sun, Aug 19, 2012 at 07:19:00PM +0100, Richard Mortier wrote: > out of curiosity, and from a *very* naive viewpoint [..], how applicable > would the same structures be to supporting disk/other io access? As far as I know [1], handling I/O is implemented by two asynchronous halves in the kernel. The lower half sits on interrupt services and places the received data in (per-device) queues. In parallel with this, the upper half reads the shared queues and processes their elements. (Sending data is just the opposite.) In this particular case, the lower half is implemented in C while the upper half is in OCaml. I do not see any difficulties in adapting this to other I/O devices, but Robert or Anil may prove me wrong :-) [1] Section 6.1, Design and Implementation of the FreeBSD Operating System (1st Ed.) From pgj@caesar.elte.hu Mon Aug 20 22:59:10 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3a06-0004op-Kq (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 20 Aug 2012 22:59:10 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -1.0 from SpamAssassin-3.3.2-1374711 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:41459) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T3a04-0003Tk-sP (Exim 4.72) (return-path ); Mon, 20 Aug 2012 22:59:10 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1T3a01-0000pM-Hr from ; Mon, 20 Aug 2012 23:59:08 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T3a00-0003jb-Jz; Mon, 20 Aug 2012 23:59:04 +0200 Date: Mon, 20 Aug 2012 23:59:04 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy , "Robert N. M. Watson" Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120820215904.GA7132@caesar.elte.hu> References: <20120816131326.GA15479@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120816131326.GA15479@caesar.elte.hu> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=AWL, BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 AWL AWL: From: address is in the auto white-list Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 21:59:10 -0000 X-Keywords: X-UID: 491 Status: O Content-Length: 1019 Lines: 21 Following up to my original mail in order to gently motivate Robert and Anil to answer :-) On Thu, Aug 16, 2012 at 03:13:27PM +0200, PALI Gabor Janos wrote: > - Shared ring buffers are created as Io_pages by allocating page-aligned, > contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). These > are directly accessible in Mirage as character arrays. Io_page.get currently returns a single page with the given size (in bytes), and not an array of pages. I did not see the reason to do so, as one can always get the corresponding pages with the Io_page.to_pages function. > - Each packet arriving on any of the plugged interfaces is placed to the next > available slot of the corresponding shared ring buffer with m_copydata(). I agree that m_copydata() involves a possibly unnecessary copying. But it also contributes to flattening the packets so Mirage can work with them without too much trickery. I suppose intercepted packets should be turned into linear memory buffers at some point. From avsm@dark.recoil.org Tue Aug 21 00:03:41 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3b0X-0005Pd-RL (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 00:03:41 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374711 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:18767 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T3b0W-0003Xn-sX (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 00:03:41 +0100 Received: (qmail 320 invoked by uid 10000); 20 Aug 2012 23:03:40 -0000 Date: Tue, 21 Aug 2012 00:03:40 +0100 From: Anil Madhavapeddy To: Richard Mortier Subject: Re: new Cohttp interface progress Message-ID: <20120820230340.GA24660@dark.recoil.org> References: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 23:03:41 -0000 X-Keywords: X-UID: 492 Status: O Content-Length: 3356 Lines: 72 On Tue, Aug 07, 2012 at 09:55:24PM +0100, Richard Mortier wrote: > > Good question. It's because each threading library has different > > tradeoffs in terms of performance. > > hm- do you have pointers to any examination of this, or is it just well > known? It's simply because they have different semantics. There are some numbers in this paper from 2010 on different concurreny mechanisms in OCaml: http://hal.inria.fr/docs/00/49/32/13/PDF/lwc.pdf It doesn't cover Async as it has only be open-sourced this year, but you get the idea I hope... > > Lwt has very fairly fine-grained errors, and propagates exceptions as > > values (i.e. "fail" vs "raise"). > > > > Async has a different mechanism called Monitors, where errors are > > caught by a separate function that keeps an eye on a chunk of > > computation. It has also deprecated streams, and has Reader/Writer > > pipes in preference. > > in some ways the different ways they do things like error handling is > (to my mind) a more interesting distinction :) (unless the perf > differences are genuinely significant and different in multiple real > applications.) So I've been hacking away at this in recent days, and (thanks to help from David Sheets) have a pretty workable repo now at: http://github.com/avsm/ocaml-cohttp/tree/v2-interface The lib/ directory has the core HTTP parser and no dependency on either Lwt or Async. Instead, it defines an IO monad and all the modules are functorised against that. Then, we have lwt/ and async/ directories which implement concrete versions of the library. You can see the Lwt interface here: https://github.com/avsm/ocaml-cohttp/blob/v2-interface/lwt/cohttp_lwt.mli There is a more low-level interface available via Cohttp_lwt_raw, but most people will probably just use the Client/Server interface here. The major difference between Lwt and Async is how they handle streams. You will need the very latest Lwt (2.4.1dev) as they added an Lwt_stream.on_terminate that we use to close the socket connection once a body has been consumed by the client. I have a mirror on avsm/lwt with this required change. I'm porting my avsm/ocaml-github bindings over to use the new API to make sure any issues are shaken out, and then I will write a Mirage Net.Channel version that we can use on the website. Then, any core protocol parsing changes will be shared across all the various backends, and we have a nice reference for what Async, Lwt and Mirage-net code all looks like in one place. > > The other *big* thing we would like is that the same protocol library > > works on both strings and Io_pages. It should be possible to hide away > > the precise choice of what we're writing into behind IO.M. > > by strings do you mean bigarray thingies? if so then i buy that - it > would be nice to have a single encapsulated layer that meant that all > the mirage protocol libraries could take a single dependency on it and > be usable from any other ocaml program. Yeah. I haven't actually done this yet as I'd like to stabilise everything else. However, it shouldn't be too hard to define a sub-set of the String module and use that to directly read/write from Bigarray too. We will have to port the Re regexp library first, but that should also be very quick. -- Anil Madhavapeddy http://anil.recoil.org From avsm@dark.recoil.org Tue Aug 21 00:05:21 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3b29-0005Rl-HJ (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 00:05:21 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374711 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:44179 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T3b28-0004BK-sM (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 00:05:21 +0100 Received: (qmail 12050 invoked by uid 10000); 20 Aug 2012 23:05:20 -0000 Date: Tue, 21 Aug 2012 00:05:20 +0100 From: Anil Madhavapeddy To: Jonathan Ludlam Subject: Re: new Cohttp interface progress Message-ID: <20120820230520.GB24660@dark.recoil.org> References: <1A061F11-2795-47D3-A415-1545F71FDA8E@eu.citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1A061F11-2795-47D3-A415-1545F71FDA8E@eu.citrix.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 23:05:21 -0000 X-Keywords: X-UID: 493 Status: O Content-Length: 853 Lines: 21 On Wed, Aug 08, 2012 at 02:01:58AM +0100, Jonathan Ludlam wrote: > > On 6 Aug 2012, at 23:53, Anil Madhavapeddy wrote: > > > > > You could also build a version of IO.M that uses native threads (where > > type t would be a dummy type and not actually hold any state). > > > > I bet this would introduce all sorts of problems because you no longer > can assume that a block of code with no IO.bind in will run atomically. > I'm sure I've seen a few places in the code that make that assumption. We need to be careful, but I think it'll still work because there is exclusive ownership on the underlying channel. If every thread has a single channel allocated to it, and locks around that usage, it'll work. I'll knock one up later to make sure this is actually true :-) -- Anil Madhavapeddy http://anil.recoil.org From avsm@dark.recoil.org Tue Aug 21 03:21:21 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3e5o-0006tk-Vf (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 03:21:21 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374711 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:41796 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T3e5n-0002eC-Z5 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 03:21:20 +0100 Received: (qmail 24100 invoked by uid 10000); 21 Aug 2012 02:21:19 -0000 Date: Tue, 21 Aug 2012 03:21:19 +0100 From: Anil Madhavapeddy To: PALI Gabor Janos Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120821022119.GS24660@dark.recoil.org> References: <20120816131326.GA15479@caesar.elte.hu> <20120820215904.GA7132@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120820215904.GA7132@caesar.elte.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 02:21:21 -0000 X-Keywords: X-UID: 494 Status: O Content-Length: 2663 Lines: 66 On Mon, Aug 20, 2012 at 11:59:04PM +0200, PALI Gabor Janos wrote: > Following up to my original mail in order to gently motivate Robert and Anil > to answer :-) My (considerable) excitement about progress here has been hit by a day-long email backlog truck, so apologies for that :-) > On Thu, Aug 16, 2012 at 03:13:27PM +0200, PALI Gabor Janos wrote: > > - Shared ring buffers are created as Io_pages by allocating page-aligned, > > contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). These > > are directly accessible in Mirage as character arrays. > > Io_page.get currently returns a single page with the given size (in bytes), > and not an array of pages. I did not see the reason to do so, as one can > always get the corresponding pages with the Io_page.to_pages function. > > > > - Each packet arriving on any of the plugged interfaces is placed to the next > > available slot of the corresponding shared ring buffer with m_copydata(). > > I agree that m_copydata() involves a possibly unnecessary copying. But it also > contributes to flattening the packets so Mirage can work with them without too > much trickery. I suppose intercepted packets should be turned into linear memory > buffers at some point. This conveniently addresses both Robert and Mort's points. I think that the right thing to do is to extend the bigarray bindings to have more custom types. Right now, we have: static void caml_ba_finalize(value v) { struct caml_ba_array * b = Caml_ba_array_val(v); #ifdef _KERNEL int data_size, i; #endif switch (b->flags & CAML_BA_MANAGED_MASK) { case CAML_BA_EXTERNAL: break; case CAML_BA_MANAGED: if (b->proxy == NULL) { #ifdef _KERNEL data_size = 0; for (i = 0; i < b->num_dims; i++) data_size += b->dim[i]; contigfree(b->data, data_size, M_MIRAGE); #else free(b->data); #endif } else { How about adding new CAML_BA_MANAGED flags specifically for mbufs and disk buffers, so that we can increment the reference count and avoid copying it? Then, the bigarray pointer can look directly into the mbuf without a copy, and decrement the reference using the relevant function rather than using contigfree directly. I think it's still useful to keep the existing contigmalloc code for other future uses that need page-aligned code, but eliminating the mbuf data copy is very important for performance. It would also be good to see what the disk buffer code looks like to make sure we can also write filesystems in Mirage (which is an interest of Jon Anderson's, among others!) -- Anil Madhavapeddy http://anil.recoil.org From avsm@dark.recoil.org Tue Aug 21 03:26:37 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3eAv-0006uq-MS (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 03:26:37 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374711 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:7757 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T3eAu-0004J9-Z8 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 03:26:37 +0100 Received: (qmail 1646 invoked by uid 10000); 21 Aug 2012 02:26:36 -0000 Date: Tue, 21 Aug 2012 03:26:36 +0100 From: Anil Madhavapeddy To: PALI Gabor Janos Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120821022636.GT24660@dark.recoil.org> References: <20120816131326.GA15479@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120816131326.GA15479@caesar.elte.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 02:26:37 -0000 X-Keywords: X-UID: 495 Status: O Content-Length: 2793 Lines: 57 On Thu, Aug 16, 2012 at 03:13:27PM +0200, PALI Gabor Janos wrote: > > - After loading the built kernel module and starting up the evaluation of > the Netif.create function, Mirage asks for a list of available Ethernet > interfaces from the FreeBSD kernel and then assigns an lwt thread to each of > them by "plugging". > > - Plugged interfaces are stored in a linked list and have their ng_ether(4) > hook (called from ether_input()) activated and pointed to > netif_ether_input(). At the same time, there is a shared ring buffer > created for each of them in Mirage then passed to the C function > responsible for administering the list of plugged interfaces, > caml_plug_vif(). > > - Shared ring buffers are created as Io_pages by allocating page-aligned, > contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). These > are directly accessible in Mirage as character arrays. > > - Each shared ring buffer is currently of size 33 pages, and operates with > 2048-byte slots. The buffers start with a header that maintains all the > required meta information, like next position, available items, size of > stored items. This all sounds spot on design-wise, by the way! > - Each packet arriving on any of the plugged interfaces is placed to the next > available slot of the corresponding shared ring buffer with m_copydata(). > > - In parallel with this in Mirage, the rx_poll function is run in loop that > polls for available packets in the shared ring buffer. > > - When rx_poll finds unprocessed packets then it runs the user-specified > function on them, e.g. print the size of the packet in basic/netif. It is > implemented by passing a view on the Io_page, i.e. without copying. After > the user function has finished, the packet is removed from the ring. > > - When no packets are available on the polled interface, rx_poll sleeps for a > millisecond. What are your thoughts on a non-spinning interface here? Can each ring buffer also have an 'event channel' assigned to it, and then a single select loop can block on all the event channels? This isn't a huge deal for the initial prototype where spinning is acceptable. > Let me add that shared rings do not do locking at the moment, but if you are > happy with the design I could proceed with implementing it. (However, I am > not completely sure how to share locks between OCaml and C.) It's best to have a C interface which does an atomic test-and-set onto an OCaml array, which can then be tested from within the Lwt/kFreeBSD Activations code. The OCaml code is guaranteed to not be running when it's in a C-binding (assuming a single thread), so this should simplify things. -- Anil Madhavapeddy http://anil.recoil.org From Richard.Mortier@nottingham.ac.uk Tue Aug 21 15:01:33 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3p1R-0004at-9r (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:01:33 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374945 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:39524 helo=ixe-mta-19.emailfiltering.com) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1T3p1G-0005D4-WZ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:01:33 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2347782644 ;bacec71b2d0b5853; Tue, 21 Aug 2012 15:01:01 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp4.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T3p0v-0006HD-O3 for cl-mirage@lists.cam.ac.uk; Tue, 21 Aug 2012 15:01:01 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Tue, 21 Aug 2012 15:01:01 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Tue, 21 Aug 2012 15:01:01 +0100 Subject: Re: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac1/pWWJfz44tc61QvyhG0j70XNxYA== Message-ID: <23B43D00-28C3-4667-97C6-CC4D4AFD0CA9@nottingham.ac.uk> References: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> <20120820230340.GA24660@dark.recoil.org> In-Reply-To: <20120820230340.GA24660@dark.recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 14:01:33 -0000 X-Keywords: X-UID: 496 Status: O Content-Length: 3052 Lines: 75 On 21 Aug 2012, at 00:03, Anil Madhavapeddy wrote: > On Tue, Aug 07, 2012 at 09:55:24PM +0100, Richard Mortier wrote: >>=20 >>=20 >> hm- do you have pointers to any examination of this, or is it just well >> known? >=20 > It's simply because they have different semantics. There are some number= s > in this paper from 2010 on different concurreny mechanisms in OCaml: > http://hal.inria.fr/docs/00/49/32/13/PDF/lwc.pdf > It doesn't cover Async as it has only be open-sourced this year, but you > get the idea I hope... i get that there are a number of different ways of implementing this, and o= f designing the apis to be used; but i don't yet completely understand why = it's useful to abstract them all behind a single monad, and then allow the = programmer to choose their own implementation. and i certainly don't see t= hat programmers are likely to be in a position to make an informed choice b= ased on the performance impact of the implementation selected without going= to a *lot* of work (possibly repeatedly). though it does suggest, at least to me, that some sort of sensible automate= d performance simulation/monitoring framework would make sense, and would r= equire a wrapping of this stuff in a monad of some kind.=20 > I'm porting my avsm/ocaml-github bindings over to use the new API to make > sure any issues are shaken out, and then I will write a Mirage Net.Channe= l > version that we can use on the website. Then, any core protocol parsing > changes will be shared across all the various backends, and we have a nic= e > reference for what Async, Lwt and Mirage-net code all looks like in one > place. cool :) >> by strings do you mean bigarray thingies? if so then i buy that - it >> would be nice to have a single encapsulated layer that meant that all >> the mirage protocol libraries could take a single dependency on it and >> be usable from any other ocaml program. >=20 > Yeah. I haven't actually done this yet as I'd like to stabilise everythin= g > else. However, it shouldn't be too hard to define a sub-set of the String > module and use that to directly read/write from Bigarray too. We will > have to port the Re regexp library first, but that should also be very > quick. cool.... --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Tue Aug 21 15:18:14 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3pHa-0004xn-7a (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:18:14 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374945 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:37523 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T3pHZ-0005lM-R2 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:18:13 +0100 Received: (qmail 15805 invoked by uid 634); 21 Aug 2012 14:18:12 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from c-76-102-1-148.hsd1.ca.comcast.net (HELO [192.168.1.105]) (76.102.1.148) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 21 Aug 2012 15:18:12 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: new Cohttp interface progress From: Anil Madhavapeddy In-Reply-To: <23B43D00-28C3-4667-97C6-CC4D4AFD0CA9@nottingham.ac.uk> Date: Tue, 21 Aug 2012 07:18:10 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> <20120820230340.GA24660@dark.recoil.org> <23B43D00-28C3-4667-97C6-CC4D4AFD0CA9@nottingham.ac.uk> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 14:18:14 -0000 X-Keywords: X-UID: 497 Status: O Content-Length: 1865 Lines: 48 On 21 Aug 2012, at 07:01, Richard Mortier = wrote: > On 21 Aug 2012, at 00:03, Anil Madhavapeddy wrote: >=20 >> On Tue, Aug 07, 2012 at 09:55:24PM +0100, Richard Mortier wrote: >>>=20 >>>=20 >>> hm- do you have pointers to any examination of this, or is it just = well >>> known? >>=20 >> It's simply because they have different semantics. There are some = numbers >> in this paper from 2010 on different concurreny mechanisms in OCaml: >> http://hal.inria.fr/docs/00/49/32/13/PDF/lwc.pdf >> It doesn't cover Async as it has only be open-sourced this year, but = you >> get the idea I hope... >=20 > i get that there are a number of different ways of implementing this, = and of designing the apis to be used; but i don't yet completely = understand why it's useful to abstract them all behind a single monad, = and then allow the programmer to choose their own implementation. and i = certainly don't see that programmers are likely to be in a position to = make an informed choice based on the performance impact of the = implementation selected without going to a *lot* of work (possibly = repeatedly). >=20 > though it does suggest, at least to me, that some sort of sensible = automated performance simulation/monitoring framework would make sense, = and would require a wrapping of this stuff in a monad of some kind.=20 Purely for portability, and not tie the core parsing unnecessarily to = one of the (many) concurrency libraries out there. One of the = problems/features of Lwt is that it affects all the types so much by using it, due to = Lwt.t going everywhere that could potentially block. At this this way, it's a=20= somewhat more abstract one that can be ported to Async's Deferred.t or = be replaced with a unit when it's not needed (i.e. if someone adds deferred cothreading to OCaml directly). -anil= From Richard.Mortier@nottingham.ac.uk Tue Aug 21 15:31:23 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3pUJ-0005Ej-Kf (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:31:23 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374945 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from thb-mta-19-tx.emailfiltering.com ([194.116.199.150]:39792 helo=thb-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T3pUD-00082w-rZ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:31:23 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by thb-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2534930530 ;70a90622cfe17be8; Tue, 21 Aug 2012 15:30:57 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp4.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T3pSn-0007Rc-3d for cl-mirage@lists.cam.ac.uk; Tue, 21 Aug 2012 15:29:49 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Tue, 21 Aug 2012 15:29:38 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Tue, 21 Aug 2012 15:29:37 +0100 Subject: Re: new Cohttp interface progress Thread-Topic: new Cohttp interface progress Thread-Index: Ac1/qWSF09A6CRYiQ9uAE/CStNAS4Q== Message-ID: References: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> <20120820230340.GA24660@dark.recoil.org> <23B43D00-28C3-4667-97C6-CC4D4AFD0CA9@nottingham.ac.uk> In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 14:31:23 -0000 X-Keywords: X-UID: 498 Status: O Content-Length: 2839 Lines: 62 On 21 Aug 2012, at 15:18, Anil Madhavapeddy wrote: > On 21 Aug 2012, at 07:01, Richard Mortier wrote: >=20 >> i get that there are a number of different ways of implementing this, an= d of designing the apis to be used; but i don't yet completely understand w= hy it's useful to abstract them all behind a single monad, and then allow t= he programmer to choose their own implementation. and i certainly don't se= e that programmers are likely to be in a position to make an informed choic= e based on the performance impact of the implementation selected without go= ing to a *lot* of work (possibly repeatedly). >>=20 >> though it does suggest, at least to me, that some sort of sensible autom= ated performance simulation/monitoring framework would make sense, and woul= d require a wrapping of this stuff in a monad of some kind.=20 >=20 > Purely for portability, and not tie the core parsing unnecessarily to one > of the (many) concurrency libraries out there. One of the problems/featu= res > of Lwt is that it affects all the types so much by using it, due to Lwt.t > going everywhere that could potentially block. At this this way, it's a= =20 > somewhat more abstract one that can be ported to Async's Deferred.t or be > replaced with a unit when it's not needed (i.e. if someone adds deferred > cothreading to OCaml directly). ok, i buy the portability argument to some extent, though i thought that we= were trying to keep Lwt out of the core parsing libraries anyway, restrict= ing it to "polluting" just the server/client code itself?=20 having said all that, the discussion is rather moot if you've done it alrea= dy :) certainly, there's nothing lost by having another layer of abstracti= on so that the server and client code can abstract over the threading libra= ry. (part of the reason i'm replying this notwithstanding the point above is to= try and keep some idea of where the design is heading in my head given i h= ave no time to hack right now... :( --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Tue Aug 21 15:34:32 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3pXM-0005Ie-KQ (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:34:32 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374945 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:25782 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T3pXL-0003fJ-Sb (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 21 Aug 2012 15:34:32 +0100 Received: (qmail 20823 invoked by uid 634); 21 Aug 2012 14:34:31 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from c-76-102-1-148.hsd1.ca.comcast.net (HELO [192.168.1.105]) (76.102.1.148) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 21 Aug 2012 15:34:31 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: new Cohttp interface progress From: Anil Madhavapeddy In-Reply-To: Date: Tue, 21 Aug 2012 07:34:29 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <19363D5B-2AEE-49DD-8AB3-EC8FADDBC839@nottingham.ac.uk> <20120820230340.GA24660@dark.recoil.org> <23B43D00-28C3-4667-97C6-CC4D4AFD0CA9@nottingham.ac.uk> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Jon Ludlam , "cl-mirage@lists.cam.ac.uk List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 14:34:32 -0000 X-Keywords: X-UID: 499 Status: O Content-Length: 2519 Lines: 64 On 21 Aug 2012, at 07:29, Richard Mortier = wrote: >=20 > On 21 Aug 2012, at 15:18, Anil Madhavapeddy wrote: >=20 >> On 21 Aug 2012, at 07:01, Richard Mortier = wrote: >>=20 >>> i get that there are a number of different ways of implementing = this, and of designing the apis to be used; but i don't yet completely = understand why it's useful to abstract them all behind a single monad, = and then allow the programmer to choose their own implementation. and i = certainly don't see that programmers are likely to be in a position to = make an informed choice based on the performance impact of the = implementation selected without going to a *lot* of work (possibly = repeatedly). >>>=20 >>> though it does suggest, at least to me, that some sort of sensible = automated performance simulation/monitoring framework would make sense, = and would require a wrapping of this stuff in a monad of some kind.=20 >>=20 >> Purely for portability, and not tie the core parsing unnecessarily to = one >> of the (many) concurrency libraries out there. One of the = problems/features >> of Lwt is that it affects all the types so much by using it, due to = Lwt.t >> going everywhere that could potentially block. At this this way, = it's a=20 >> somewhat more abstract one that can be ported to Async's Deferred.t = or be >> replaced with a unit when it's not needed (i.e. if someone adds = deferred >> cothreading to OCaml directly). >=20 > ok, i buy the portability argument to some extent, though i thought = that we were trying to keep Lwt out of the core parsing libraries = anyway, restricting it to "polluting" just the server/client code = itself? Sure...and this way it's guaranteed to not pollute the core library due to the functor preventing it. I found a few places where Lwt-specific = things were used already. Also, all the cohttp forks existed for a reason: they = were all restructuring the same code, which is now hopefully more portable. > having said all that, the discussion is rather moot if you've done it = already :) certainly, there's nothing lost by having another layer of = abstraction so that the server and client code can abstract over the = threading library. Well, I'm wondering whether to apply this to the DNS code next (which is = a lot simpler, since most of it is non-Lwt parsing code). I'd like to run some more performance tests first, and prepare for the upcoming = XenSummit talk first though. -anil From pgj@caesar.elte.hu Wed Aug 22 00:28:00 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3xrc-0003ke-2D (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 00:28:00 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -1.0 from SpamAssassin-3.3.2-1374945 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:59971) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1T3xrb-0003hi-WS (Exim 4.72) (return-path ); Wed, 22 Aug 2012 00:28:00 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1T3xrY-0001bM-56 from ; Wed, 22 Aug 2012 01:27:58 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T3xrX-00077G-DL; Wed, 22 Aug 2012 01:27:55 +0200 Date: Wed, 22 Aug 2012 01:27:55 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120821232755.GA22858@caesar.elte.hu> References: <20120816131326.GA15479@caesar.elte.hu> <20120820215904.GA7132@caesar.elte.hu> <20120821022119.GS24660@dark.recoil.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120821022119.GS24660@dark.recoil.org> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 23:28:00 -0000 X-Keywords: X-UID: 500 Status: O Content-Length: 2640 Lines: 59 On Tue, Aug 21, 2012 at 03:21:19AM +0100, Anil Madhavapeddy wrote: > How about adding new CAML_BA_MANAGED flags specifically for mbufs and disk > buffers, so that we can increment the reference count and avoid copying > it? I have managed to implement this. I introduced a new management type, CAML_BA_MBUF for Bigarrays. (It is not published at GitHub yet, though.) enum caml_ba_managed { CAML_BA_EXTERNAL = 0, /* Data is not allocated by Caml */ CAML_BA_MANAGED = 0x200, /* Data is allocated by Caml */ CAML_BA_MAPPED_FILE = 0x400, /* Data is a memory mapped file */ #ifdef _KERNEL CAML_BA_MBUF = 0x800, /* Data is a FreeBSD mbuf(9) */ CAML_BA_MANAGED_MASK = 0xE00 /* Mask for "managed" bits in flags field */ #else CAML_BA_MANAGED_MASK = 0x600 /* Mask for "managed" bits in flags field */ #endif }; This is then employed when an mbuf(9) is intercepted, so I worked around m_copydata(). Yay! > I think it's still useful to keep the existing contigmalloc code for other > future uses that need page-aligned code, but eliminating the mbuf data > copy is very important for performance. All right. So, now each of the buffers are embedded into Bigarrays, that is, each page corresponds to an mbuf(9). That is, as a side effect, when packets are received in multiple fragments, they are going to be mapped to multiple Io_pages. But those fragments are part of a single object, and this relation is not expressed this way -- since the meta information stored in the mbuf(9) (the m_next pointer) does not appear on the page. (Or, it could, but I would feel that overkill.) Instead, perhaps it should be expressed by grouping the Io_pages, i.e. putting them into a common list by being part of the same packet or not. Then the lambda function in Netif.listen could receive a list of Io_pages of the same packet. However, let me add it turns out that in practice only single-mbuf(9) packets arrive from the network card, so it is not a big problem at the moment, I just wanted to mention it. Furthermore, an interesting implementation detail is how the captured buffers are passed to the Mirage side in this case. Currently I store all the native buffers coming from the card (via ether_input()) in a mutex-protected linked list in C, which is converted into an OCaml Io.page.t list via a C callback function. Note that Bigarrays are added around buffers at this point only, while the list on the C side is emptied. So packet processing hence becomes this technically: (* get_mbufs : id -> Io_page.t list *) let rx_poll ifc fn = let mbufs = get_mbufs ifc.id in Lwt_list.iter_s fn mbufs From pgj@caesar.elte.hu Wed Aug 22 00:32:07 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3xvb-0003mI-9K (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 00:32:07 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -1.0 from SpamAssassin-3.3.2-1374945 * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.1.138 listed in list.dnswl.dnsbl.ja.net] * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx3.mail.elte.hu ([157.181.1.138]:55227) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1T3xvZ-0007Vy-no (Exim 4.72) (return-path ); Wed, 22 Aug 2012 00:32:07 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx3.mail.elte.hu with esmtp (Exim) id 1T3xvY-0000HK-B5 from ; Wed, 22 Aug 2012 01:32:04 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T3xvX-0007El-NP; Wed, 22 Aug 2012 01:32:03 +0200 Date: Wed, 22 Aug 2012 01:32:03 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Message-ID: <20120821233203.GB22858@caesar.elte.hu> References: <20120816131326.GA15479@caesar.elte.hu> <20120821022636.GT24660@dark.recoil.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120821022636.GT24660@dark.recoil.org> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2012 23:32:07 -0000 X-Keywords: X-UID: 501 Status: O Content-Length: 298 Lines: 7 On Tue, Aug 21, 2012 at 03:26:36AM +0100, Anil Madhavapeddy wrote: > What are your thoughts on a non-spinning interface here? Can each ring > buffer also have an 'event channel' assigned to it, and then a single > select loop can block on all the event channels? Okay, I will see what I can do. From anil@recoil.org Wed Aug 22 01:39:51 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T3yz9-0004Ek-Om (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 01:39:51 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1374945 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:3165 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T3yz9-0002Kd-QB (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 01:39:51 +0100 Received: (qmail 2154 invoked by uid 634); 22 Aug 2012 00:39:50 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from visnet-57.csl.sri.com (HELO visnet-57.csl.sri.com) (130.107.98.57) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Wed, 22 Aug 2012 01:39:49 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD From: Anil Madhavapeddy In-Reply-To: <20120821232755.GA22858@caesar.elte.hu> Date: Tue, 21 Aug 2012 17:39:39 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <8FB9C110-4514-4F42-AD33-5EA8A4847B40@recoil.org> References: <20120816131326.GA15479@caesar.elte.hu> <20120820215904.GA7132@caesar.elte.hu> <20120821022119.GS24660@dark.recoil.org> <20120821232755.GA22858@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "Robert N. M. Watson" , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 00:39:51 -0000 X-Keywords: X-UID: 502 Status: O Content-Length: 3924 Lines: 116 On 21 Aug 2012, at 16:27, PALI Gabor Janos wrote: > On Tue, Aug 21, 2012 at 03:21:19AM +0100, Anil Madhavapeddy wrote: >> How about adding new CAML_BA_MANAGED flags specifically for mbufs and = disk >> buffers, so that we can increment the reference count and avoid = copying >> it? >=20 > I have managed to implement this. I introduced a new management type, > CAML_BA_MBUF for Bigarrays. (It is not published at GitHub yet, = though.) >=20 > enum caml_ba_managed { > CAML_BA_EXTERNAL =3D 0, /* Data is not allocated by Caml */ > CAML_BA_MANAGED =3D 0x200, /* Data is allocated by Caml */ > CAML_BA_MAPPED_FILE =3D 0x400, /* Data is a memory mapped file */ > #ifdef _KERNEL > CAML_BA_MBUF =3D 0x800, /* Data is a FreeBSD mbuf(9) */ > CAML_BA_MANAGED_MASK =3D 0xE00 /* Mask for "managed" bits in flags = field */ > #else > CAML_BA_MANAGED_MASK =3D 0x600 /* Mask for "managed" bits in flags = field */ > #endif > }; >=20 > This is then employed when an mbuf(9) is intercepted, so I worked = around > m_copydata(). Yay! Superb! Do you have any insight into what the equivalent disk API is? >> I think it's still useful to keep the existing contigmalloc code for = other >> future uses that need page-aligned code, but eliminating the mbuf = data >> copy is very important for performance. >=20 > All right. So, now each of the buffers are embedded into Bigarrays, = that is, > each page corresponds to an mbuf(9). That is, as a side effect, when = packets > are received in multiple fragments, they are going to be mapped to = multiple > Io_pages. But those fragments are part of a single object, and this = relation > is not expressed this way -- since the meta information stored in the = mbuf(9) > (the m_next pointer) does not appear on the page. (Or, it could, but = I would > feel that overkill.) >=20 > Instead, perhaps it should be expressed by grouping the Io_pages, i.e. = putting > them into a common list by being part of the same packet or not. Then = the=20 > lambda function in Netif.listen could receive a list of Io_pages of = the same > packet. However, let me add it turns out that in practice only = single-mbuf(9) > packets arrive from the network card, so it is not a big problem at = the > moment, I just wanted to mention it. Interesting problem. In Xen, all the fragments come in as completely = separate pages with no connection beyond the Netif interface signalling that they = are continued fragments from a previous entry in the buffer. How about allocating them all as separate Io_pages and incrementing the = mbuf ref count several times? Although, as you say, if our hardware doesn't generate them for the moment, it's good enough to mark a TODO and = revisit this in the future. The listen interface requires them to be coalesced at = the moment I think, as our cstruct extension assumes a contiguous buffer. We could extend cstruct to detect the size of the fragments and index = into the correct Bigarray automatically. What's the minimum fragment size in = FreeBSD these days? > Furthermore, an interesting implementation detail is how the captured = buffers > are passed to the Mirage side in this case. Currently I store all the = native > buffers coming from the card (via ether_input()) in a mutex-protected = linked > list in C, which is converted into an OCaml Io.page.t list via a C = callback > function. Note that Bigarrays are added around buffers at this point = only, > while the list on the C side is emptied. >=20 > So packet processing hence becomes this technically: >=20 > (* get_mbufs : id -> Io_page.t list *) >=20 > let rx_poll ifc fn =3D > let mbufs =3D get_mbufs ifc.id in > Lwt_list.iter_s fn > mbufs Sounds sensible! Haris here in SRI is working on a simulator backend = that will exercise the *other* end of the stack (the TCP logic in particular), so = this is coming together quite nicely. -anil From robert.watson@cl.cam.ac.uk Wed Aug 22 08:44:01 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T45bd-00014L-L8 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:44:01 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375431 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:52679) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T45bc-0004C3-rV (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:44:01 +0100 Received: from [192.168.2.118] (host86-143-223-13.range86-143.btcentralplus.com [86.143.223.13]) by cyrus.watson.org (Postfix) with ESMTPSA id 8268E46B09; Wed, 22 Aug 2012 03:43:58 -0400 (EDT) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20120816131326.GA15479@caesar.elte.hu> Date: Wed, 22 Aug 2012 08:43:56 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120816131326.GA15479@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 07:44:01 -0000 X-Keywords: X-UID: 503 Status: O Content-Length: 3964 Lines: 91 On 16 Aug 2012, at 14:13, PALI Gabor Janos wrote: > For your information, I have just added support for receiving network = packets > to the kFreeBSD backend of mirage-platform: Excellent news -- sorry for the delay in responding, things have been a = bit congested here, lately :-). > - Plugged interfaces are stored in a linked list and have their = ng_ether(4) > hook (called from ether_input()) activated and pointed to > netif_ether_input(). At the same time, there is a shared ring buffer > created for each of them in Mirage then passed to the C function > responsible for administering the list of plugged interfaces, > caml_plug_vif(). The pfil(9) KPI in FreeBSD defines 'PFIL_TYPE_IFNET ' but doesn't = implement it -- but it sounds like that is what Mirage/kFreeBSD wants. = It might be worth implementing it and seeing if ng_ether(4) can also = register using the same mechanism. > - Shared ring buffers are created as Io_pages by allocating = page-aligned, > contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). = These > are directly accessible in Mirage as character arrays. For the kernel OCaml stack, is physically contiguous memory required? = Normally the FreeBSD VM system will return virtually (and likely = physically) contiguous memory for kernel memory allocations, but = allowing it to use physically non-contiguous memory gives the VM system = flexibility. I.e., is there a reason not to just use malloc(9), which = for large allocation sizes, simply requests pages from the VM system = rather than using the slab allocator? > - Each shared ring buffer is currently of size 33 pages, and operates = with > 2048-byte slots. The buffers start with a header that maintains all = the > required meta information, like next position, available items, size = of > stored items. >=20 > - Each packet arriving on any of the plugged interfaces is placed to = the next > available slot of the corresponding shared ring buffer with = m_copydata(). As we talked about briefly a couple of days ago on IRC, it would be = great if we could avoid the mandatory data copy here. Allowing mbuf = cluster memory to transparently flow into (and out of) the OCaml = runtime, subject to the PL runtime itself, would perhaps allow the copy = to be avoided where not required by Mirage, which helps with memory = footprint, cache footprint, etc. In principle, at the point where the = mbuf is snarfed by Mirage, it should have exclusive ownership of that = meta-data, and often exclusive ownership of the memory pointed to by the = mbuf -- although if there are attempts to write, you might at that point = need to duplicate the data if it's a shared mbuf. E.g., if Mirage is = doing loopback NFS to the NFS server, and mbufs are pointing at pages in = the buffer cache, writing back to the buffer cache may be undesirable. = :-) > - In parallel with this in Mirage, the rx_poll function is run in loop = that > polls for available packets in the shared ring buffer. >=20 > - When rx_poll finds unprocessed packets then it runs the = user-specified > function on them, e.g. print the size of the packet in basic/netif. = It is > implemented by passing a view on the Io_page, i.e. without copying. = After > the user function has finished, the packet is removed from the ring. >=20 > - When no packets are available on the polled interface, rx_poll = sleeps for a > millisecond. What was the eventual conclusion on the ability to directly dispatch = Mirage instances from the low-level interrupt thread? For the default = network stack, we measured significant reductions in latency when = switching to that as the default model, as well as efficiency = improvements under load: packets are dropped before entering the NIC = descriptor ring, rather than at the asynchronous dispatch point to a = higher-level thread. Otherwise, you throw away all the cycles used to = process the packet up until that point. Robert= From robert.watson@cl.cam.ac.uk Wed Aug 22 08:48:43 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T45gB-00018K-EX (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:48:43 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375431 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:18986) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1T45gA-0005Ey-p4 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:48:43 +0100 Received: from [192.168.2.118] (host86-143-223-13.range86-143.btcentralplus.com [86.143.223.13]) by cyrus.watson.org (Postfix) with ESMTPSA id B330946B2D; Wed, 22 Aug 2012 03:48:40 -0400 (EDT) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20120820134641.GA18444@caesar.elte.hu> Date: Wed, 22 Aug 2012 08:48:39 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <1BE0213A-6E76-4501-A699-A286C4497CF6@cl.cam.ac.uk> References: <20120816131326.GA15479@caesar.elte.hu> <87E2E550-97C1-4392-8F29-ACD1B395B434@nottingham.ac.uk> <20120820134641.GA18444@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: Richard Mortier , "cl-mirage@lists.cam.ac.uk" , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 07:48:43 -0000 X-Keywords: X-UID: 504 Status: O Content-Length: 1666 Lines: 43 On 20 Aug 2012, at 14:46, PALI Gabor Janos wrote: > On Sun, Aug 19, 2012 at 07:19:00PM +0100, Richard Mortier wrote: >> out of curiosity, and from a *very* naive viewpoint [..], how = applicable >> would the same structures be to supporting disk/other io access? >=20 > As far as I know [1], handling I/O is implemented by two asynchronous = halves > in the kernel. The lower half sits on interrupt services and places = the > received data in (per-device) queues. In parallel with this, the = upper half > reads the shared queues and processes their elements. (Sending data = is just > the opposite.) >=20 > In this particular case, the lower half is implemented in C while the = upper > half is in OCaml. I do not see any difficulties in adapting this to = other > I/O devices, but Robert or Anil may prove me wrong :-) Disk I/O operations are a lot like packets, especially lately with high = TPS flash-based storage devices (vis FusionIO, etc). The KPIs are all a = bit different, but there's no reason to think that could easily be = worked through. An interesting question is where else storage-esque = requests might be sent -- the places, off the top of my head, that sound = plausible are: (a) A hypervisor running under the OS hosting Mirage/kFreeBSD -- e.g., = as though Mirage were running straight on the hypervisor. (b) OS services underlying Mirage/kFreeBSD -- e.g., the block storage = mechanism (geom), file system, etc. (c) Userspace running on top of FreeBSD -- e.g., a miraged that can = proxy requests somewhere. However, I'm ignorant of Mirage's storage-esque facilities -- ordinary = file system? Flag transactional thingy? Robert= From robert.watson@cl.cam.ac.uk Wed Aug 22 08:58:07 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T45pH-0001UD-Bs (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:58:07 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375431 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from cyrus.watson.org ([65.122.17.42]:30388) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1T45pG-0008R4-p9 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 08:58:07 +0100 Received: from [192.168.2.118] (host86-143-223-13.range86-143.btcentralplus.com [86.143.223.13]) by cyrus.watson.org (Postfix) with ESMTPSA id BFE9846B1A; Wed, 22 Aug 2012 03:58:05 -0400 (EDT) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD Mime-Version: 1.0 (Apple Message framework v1280) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: <20120820215904.GA7132@caesar.elte.hu> Date: Wed, 22 Aug 2012 08:58:04 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <566DB3AF-401B-4D6D-9BA1-8D6282DCAE74@cl.cam.ac.uk> References: <20120816131326.GA15479@caesar.elte.hu> <20120820215904.GA7132@caesar.elte.hu> To: PALI Gabor Janos X-Mailer: Apple Mail (2.1280) Cc: cl-mirage@lists.cam.ac.uk, Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 07:58:07 -0000 X-Keywords: X-UID: 505 Status: O Content-Length: 1635 Lines: 36 On 20 Aug 2012, at 22:59, PALI Gabor Janos wrote: >> - Each packet arriving on any of the plugged interfaces is placed to = the next >> available slot of the corresponding shared ring buffer with = m_copydata(). >=20 > I agree that m_copydata() involves a possibly unnecessary copying. = But it also > contributes to flattening the packets so Mirage can work with them = without too > much trickery. I suppose intercepted packets should be turned into = linear memory > buffers at some point. My intuition is that, except in unusual and special situations, packets = in the inbound path will always turn up contiguously stored in a buffer. = The only real exceptions will be when odd encapsulations are used, when = there is header-splitting, or perhaps during a transition to jumbogram = use before a receive descriptor ring has been repopulated with larger = buffer sizes. In the outbound path from the regular network stack, again = it's probably only in situations with odd encapsulations or non-bulk = sending patterns that this really comes up. The receive stream socket = buffers have an algorithm for packing small packets but chaining large = ones, and a very similar approach could be used here: if you get = non-contiguous data, you can always "compress" it into dynamically = allocated contiguous buffers -- in fact, you can ask the network stack = to do this for you, but you can optimise for the contiguous case pretty = reasonably. If you do end up seeing a lot of packets fragmented over = non-contiguous buffers, it's probably due to a device driver or network = stack bug, which should be fixed. :-) Robert= From anil@recoil.org Wed Aug 22 16:43:04 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T4D5E-0006kY-3p (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 16:43:04 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375431 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:10518 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T4D5D-00089S-YG (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 16:43:04 +0100 Received: (qmail 12461 invoked by uid 634); 22 Aug 2012 15:43:03 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from c-76-102-1-148.hsd1.ca.comcast.net (HELO [192.168.1.105]) (76.102.1.148) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Wed, 22 Aug 2012 16:43:02 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD From: Anil Madhavapeddy In-Reply-To: Date: Wed, 22 Aug 2012 08:43:00 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120816131326.GA15479@caesar.elte.hu> To: "Robert N. M. Watson" X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: PALI Gabor Janos , cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 15:43:04 -0000 X-Keywords: X-UID: 506 Status: O Content-Length: 1890 Lines: 42 On 22 Aug 2012, at 00:43, "Robert N. M. Watson" = wrote: >=20 >> - Plugged interfaces are stored in a linked list and have their = ng_ether(4) >> hook (called from ether_input()) activated and pointed to >> netif_ether_input(). At the same time, there is a shared ring buffer >> created for each of them in Mirage then passed to the C function >> responsible for administering the list of plugged interfaces, >> caml_plug_vif(). >=20 > The pfil(9) KPI in FreeBSD defines 'PFIL_TYPE_IFNET ' but doesn't = implement it -- but it sounds like that is what Mirage/kFreeBSD wants. = It might be worth implementing it and seeing if ng_ether(4) can also = register using the same mechanism. I'd completely forgotten about netgraph. ng_ether(4) and ng_socket(4) = might be very useful indeed for ongoing prototyping. >> - Shared ring buffers are created as Io_pages by allocating = page-aligned, >> contiguous, multi-page memory areas via FreeBSD's contigmalloc(9). = These >> are directly accessible in Mirage as character arrays. >=20 > For the kernel OCaml stack, is physically contiguous memory required? = Normally the FreeBSD VM system will return virtually (and likely = physically) contiguous memory for kernel memory allocations, but = allowing it to use physically non-contiguous memory gives the VM system = flexibility. I.e., is there a reason not to just use malloc(9), which = for large allocation sizes, simply requests pages from the VM system = rather than using the slab allocator? >=20 The OCaml heap memory does not need to be physically contiguous. Heap = extents are allocated in megabyte-ish chunks and tracked using a sparse = hashtable in memory. However, Io_pages are addressed from within OCaml = as a normal array, and so do need to virtually contiguous (but not = necessarily physically unless required by the DMA subsystem). -anil= From anil@recoil.org Wed Aug 22 17:04:18 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T4DPl-0007ZR-VW (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 17:04:17 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375431 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:33278 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T4DPk-0003Y2-ZN (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 22 Aug 2012 17:04:17 +0100 Received: (qmail 19103 invoked by uid 634); 22 Aug 2012 16:04:16 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from c-76-102-1-148.hsd1.ca.comcast.net (HELO [192.168.1.105]) (76.102.1.148) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Wed, 22 Aug 2012 17:04:16 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: Receiving Network Packets with Mirage/kFreeBSD From: Anil Madhavapeddy In-Reply-To: <1BE0213A-6E76-4501-A699-A286C4497CF6@cl.cam.ac.uk> Date: Wed, 22 Aug 2012 09:04:14 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120816131326.GA15479@caesar.elte.hu> <87E2E550-97C1-4392-8F29-ACD1B395B434@nottingham.ac.uk> <20120820134641.GA18444@caesar.elte.hu> <1BE0213A-6E76-4501-A699-A286C4497CF6@cl.cam.ac.uk> To: "Robert N. M. Watson" X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: PALI Gabor Janos , Richard Mortier , "cl-mirage@lists.cam.ac.uk" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 16:04:18 -0000 X-Keywords: X-UID: 507 Status: O Content-Length: 2186 Lines: 58 On 22 Aug 2012, at 00:48, "Robert N. M. Watson" = wrote: >=20 > On 20 Aug 2012, at 14:46, PALI Gabor Janos wrote: >=20 >> On Sun, Aug 19, 2012 at 07:19:00PM +0100, Richard Mortier wrote: >>> out of curiosity, and from a *very* naive viewpoint [..], how = applicable >>> would the same structures be to supporting disk/other io access? >>=20 >> As far as I know [1], handling I/O is implemented by two asynchronous = halves >> in the kernel. The lower half sits on interrupt services and places = the >> received data in (per-device) queues. In parallel with this, the = upper half >> reads the shared queues and processes their elements. (Sending data = is just >> the opposite.) >>=20 >> In this particular case, the lower half is implemented in C while the = upper >> half is in OCaml. I do not see any difficulties in adapting this to = other >> I/O devices, but Robert or Anil may prove me wrong :-) >=20 > Disk I/O operations are a lot like packets, especially lately with = high TPS flash-based storage devices (vis FusionIO, etc). The KPIs are = all a bit different, but there's no reason to think that could easily be = worked through. An interesting question is where else storage-esque = requests might be sent -- the places, off the top of my head, that sound = plausible are: >=20 > (a) A hypervisor running under the OS hosting Mirage/kFreeBSD -- e.g., = as though Mirage were running straight on the hypervisor. > (b) OS services underlying Mirage/kFreeBSD -- e.g., the block storage = mechanism (geom), file system, etc. > (c) Userspace running on top of FreeBSD -- e.g., a miraged that can = proxy requests somewhere. >=20 > However, I'm ignorant of Mirage's storage-esque facilities -- ordinary = file system? Flag transactional thingy? We have libs for: - FAT16/32/64 - Simple k/v - https://github.com/Incubaid/baardskeerder (CoW B-tree) - NFSv3 (needs porting, but fairly minor) The Block interface is a ring buffer, with the underlying protocol = shared with Net (a ring buffer with req/responses), and there is no = special filesystem layer. Applications currently just link against = their desired fs or k-v library. -anil= From Dave.Scott@eu.citrix.com Sat Aug 25 18:16:44 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5JyW-0007Rj-5Z (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 18:16:44 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -2.6 from SpamAssassin-3.3.2-1375947 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [62.200.22.115 listed in list.dnswl.dnsbl.ja.net] * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from smtp.eu.citrix.com ([62.200.22.115]:57967) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1T5JyV-0003jY-oA (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 18:16:44 +0100 X-IronPort-AV: E=Sophos;i="4.80,311,1344211200"; d="scan'208";a="14181955" Received: from lonpmailmx02.citrite.net ([10.30.203.163]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/RC4-MD5; 25 Aug 2012 17:16:42 +0000 Received: from LONPMAILBOX01.citrite.net ([10.30.224.160]) by LONPMAILMX02.citrite.net ([10.30.203.163]) with mapi; Sat, 25 Aug 2012 18:16:41 +0100 From: Dave Scott To: "cl-mirage@lists.cam.ac.uk Mailing List" Date: Sat, 25 Aug 2012 18:16:30 +0100 Subject: urls.txt-- format change? Thread-Topic: urls.txt-- format change? Thread-Index: Ac2C5WRzZWEfOiIXRKqJURndFv88fQ== Message-ID: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 17:16:44 -0000 X-Keywords: X-UID: 508 Status: O Content-Length: 415 Lines: 22 Hi, I'm hitting a problem reinstalling the mirage stuff via opam which I think = is caused by a urls.txt format change. If you compare http://opam.ocamlpro.com/urls.txt with http://mirage.github.com/opam/urls.txt it looks like a re-arrangement. The error I get with "opam init default ...= " is consistent with it mis-parsing the file. Anyone know how to regenerate the urls.txt? Thanks, --=20 Dave Scott From anil@recoil.org Sat Aug 25 18:21:26 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5K34-0007T7-9P (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 18:21:26 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:42405 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T5K33-0008DM-Wl (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 18:21:26 +0100 Received: (qmail 19998 invoked by uid 634); 25 Aug 2012 17:21:24 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from pool-71-177-225-111.lsanca.fios.verizon.net (HELO [192.168.1.148]) (71.177.225.111) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Sat, 25 Aug 2012 18:21:24 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: urls.txt-- format change? From: Anil Madhavapeddy In-Reply-To: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> Date: Sat, 25 Aug 2012 10:21:20 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> To: Dave Scott X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 17:21:26 -0000 X-Keywords: X-UID: 509 Status: O Content-Length: 1739 Lines: 55 Thomas has done an epic set of changes in 0.4 which totally re-do the = package format to be more sensible, and adds checksums for packages, and = removes the need for a separate dev repository. In the very short term, you can run 'opam-convert-repo-0.3' to the new = format on an existing repo, and here are the instructions Thomas sent me = earlier to add a new package. 1. git clone git://github.com/OCamlPro/opam-repository.git 2. git remote -add local opam-repository 3. cd opam-repository/packages/ 4. cp -r batteries.1.4.3 mlorg.xxx 5. [edit mlorg.xxx/descr with the right description] 6. [edit mlorg.xxx/url with the right archive name and checksum] 7. opam update [normally it should have picked the new mlorg package] 8. opam install mlorg 9. [in case of error, edit mlorg.xxx/opam and goto 7] In the new world order, you can simply add a 'url: "git://foo"' to the = url file in a package, rather than the separate setup we had before. I = guess we need to maintain a fork for now with our dev packages, so I am = running the conversion script right now to update mirage.github.com. This is hopefully the last of the big packaging changes (also famous = last words). -anil On 25 Aug 2012, at 10:16, Dave Scott wrote: > Hi, >=20 > I'm hitting a problem reinstalling the mirage stuff via opam which I = think is caused by a urls.txt format change. If you compare >=20 > http://opam.ocamlpro.com/urls.txt >=20 > with >=20 > http://mirage.github.com/opam/urls.txt >=20 > it looks like a re-arrangement. The error I get with "opam init = default ..." is consistent with it mis-parsing the file. >=20 > Anyone know how to regenerate the urls.txt? >=20 > Thanks, >=20 > --=20 > Dave Scott >=20 >=20 From anil@recoil.org Sat Aug 25 19:28:17 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5L5k-0007fS-W7 (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 19:28:17 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:6509 helo=dark.recoil.org) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with smtp id 1T5L5k-0003cQ-XJ (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 19:28:16 +0100 Received: (qmail 24105 invoked by uid 634); 25 Aug 2012 18:28:16 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from pool-71-177-225-111.lsanca.fios.verizon.net (HELO [192.168.1.148]) (71.177.225.111) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Sat, 25 Aug 2012 19:28:15 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: urls.txt-- format change? From: Anil Madhavapeddy In-Reply-To: Date: Sat, 25 Aug 2012 11:28:11 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <4A6DD509-303E-4D2B-9980-4313E20B19DA@recoil.org> References: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> To: Dave Scott X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 18:28:17 -0000 X-Keywords: X-UID: 510 Status: O Content-Length: 2436 Lines: 73 I've done a really quick re-jig of the repositories, so you can try: $ opam init http://mirage.github.com/opam-0.4 $ opam remote -add local http://mirage.github.com/opam-0.4-dev $ opem install lwt # due to a bug in the package files $ opam install mirage I'm still finishing up the dev conversion (a bit later today) and will = get mirage-www building with it (wont take too long). It's probably = easier to clone the repos yourself (mirage/opam-repository and = mirage/opam-repo-dev) and run "opam-mk-repo" yourself for now. -anil On 25 Aug 2012, at 10:21, Anil Madhavapeddy wrote: > Thomas has done an epic set of changes in 0.4 which totally re-do the = package format to be more sensible, and adds checksums for packages, and = removes the need for a separate dev repository. >=20 > In the very short term, you can run 'opam-convert-repo-0.3' to the new = format on an existing repo, and here are the instructions Thomas sent me = earlier to add a new package. >=20 > 1. git clone git://github.com/OCamlPro/opam-repository.git > 2. git remote -add local opam-repository > 3. cd opam-repository/packages/ > 4. cp -r batteries.1.4.3 mlorg.xxx > 5. [edit mlorg.xxx/descr with the right description] > 6. [edit mlorg.xxx/url with the right archive name and checksum] > 7. opam update > [normally it should have picked the new mlorg package] > 8. opam install mlorg > 9. [in case of error, edit mlorg.xxx/opam and goto 7] >=20 > In the new world order, you can simply add a 'url: "git://foo"' to the = url file in a package, rather than the separate setup we had before. I = guess we need to maintain a fork for now with our dev packages, so I am = running the conversion script right now to update mirage.github.com. >=20 > This is hopefully the last of the big packaging changes (also famous = last words). >=20 > -anil >=20 > On 25 Aug 2012, at 10:16, Dave Scott wrote: >=20 >> Hi, >>=20 >> I'm hitting a problem reinstalling the mirage stuff via opam which I = think is caused by a urls.txt format change. If you compare >>=20 >> http://opam.ocamlpro.com/urls.txt >>=20 >> with >>=20 >> http://mirage.github.com/opam/urls.txt >>=20 >> it looks like a re-arrangement. The error I get with "opam init = default ..." is consistent with it mis-parsing the file. >>=20 >> Anyone know how to regenerate the urls.txt? >>=20 >> Thanks, >>=20 >> --=20 >> Dave Scott >>=20 >>=20 >=20 >=20 From thomas.gazagnaire@gmail.com Sat Aug 25 20:02:35 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5Lcx-00005N-TQ (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 20:02:35 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.9 from SpamAssassin-3.3.2-1375947 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.212.179 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (thomas.gazagnaire[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from * author's domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-wi0-f179.google.com ([209.85.212.179]:51175) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T5Lcw-0004oU-pr (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sat, 25 Aug 2012 20:02:35 +0100 Received: by wibhq4 with SMTP id hq4so1571983wib.2 for ; Sat, 25 Aug 2012 12:02:34 -0700 (PDT) Received: by 10.180.105.163 with SMTP id gn3mr13912029wib.2.1345921353792; Sat, 25 Aug 2012 12:02:33 -0700 (PDT) Received: from [10.172.218.205] ([37.160.26.93]) by mx.google.com with ESMTPS id dv1sm7135162wib.10.2012.08.25.12.02.30 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 25 Aug 2012 12:02:32 -0700 (PDT) References: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> <4A6DD509-303E-4D2B-9980-4313E20B19DA@recoil.org> In-Reply-To: <4A6DD509-303E-4D2B-9980-4313E20B19DA@recoil.org> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: <92B48648-1B39-4651-BF85-F745012E5E80@gmail.com> X-Mailer: iPhone Mail (9B206) From: Thomas Gazagnaire Subject: Re: urls.txt-- format change? Date: Sat, 25 Aug 2012 21:02:23 +0200 To: Anil Madhavapeddy Cc: Dave Scott , "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 19:02:36 -0000 X-Keywords: X-UID: 511 Status: O Content-Length: 2690 Lines: 83 btw you can init & remote -add github repo addresses directly -- Thomas Le 25 ao=C3=BBt 2012 =C3=A0 20:28, Anil Madhavapeddy a =C3= =A9crit : > I've done a really quick re-jig of the repositories, so you can try: >=20 > $ opam init http://mirage.github.com/opam-0.4 > $ opam remote -add local http://mirage.github.com/opam-0.4-dev > $ opem install lwt # due to a bug in the package files > $ opam install mirage >=20 > I'm still finishing up the dev conversion (a bit later today) and will get= mirage-www building with it (wont take too long). It's probably easier to c= lone the repos yourself (mirage/opam-repository and mirage/opam-repo-dev) an= d run "opam-mk-repo" yourself for now. >=20 > -anil >=20 > On 25 Aug 2012, at 10:21, Anil Madhavapeddy wrote: >=20 >> Thomas has done an epic set of changes in 0.4 which totally re-do the pac= kage format to be more sensible, and adds checksums for packages, and remove= s the need for a separate dev repository. >>=20 >> In the very short term, you can run 'opam-convert-repo-0.3' to the new fo= rmat on an existing repo, and here are the instructions Thomas sent me earli= er to add a new package. >>=20 >> 1. git clone git://github.com/OCamlPro/opam-repository.git >> 2. git remote -add local opam-repository >> 3. cd opam-repository/packages/ >> 4. cp -r batteries.1.4.3 mlorg.xxx >> 5. [edit mlorg.xxx/descr with the right description] >> 6. [edit mlorg.xxx/url with the right archive name and checksum] >> 7. opam update >> [normally it should have picked the new mlorg package] >> 8. opam install mlorg >> 9. [in case of error, edit mlorg.xxx/opam and goto 7] >>=20 >> In the new world order, you can simply add a 'url: "git://foo"' to the ur= l file in a package, rather than the separate setup we had before. I guess w= e need to maintain a fork for now with our dev packages, so I am running the= conversion script right now to update mirage.github.com. >>=20 >> This is hopefully the last of the big packaging changes (also famous last= words). >>=20 >> -anil >>=20 >> On 25 Aug 2012, at 10:16, Dave Scott wrote: >>=20 >>> Hi, >>>=20 >>> I'm hitting a problem reinstalling the mirage stuff via opam which I thi= nk is caused by a urls.txt format change. If you compare >>>=20 >>> http://opam.ocamlpro.com/urls.txt >>>=20 >>> with >>>=20 >>> http://mirage.github.com/opam/urls.txt >>>=20 >>> it looks like a re-arrangement. The error I get with "opam init default .= .." is consistent with it mis-parsing the file. >>>=20 >>> Anyone know how to regenerate the urls.txt? >>>=20 >>> Thanks, >>>=20 >>> --=20 >>> Dave Scott >>>=20 >>>=20 >>=20 >>=20 >=20 >=20 From anil@recoil.org Sun Aug 26 00:52:26 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5Q9S-0001we-7V (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 00:52:26 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:48591 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T5Q9R-0006RR-rC (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 00:52:26 +0100 Received: (qmail 10662 invoked by uid 634); 25 Aug 2012 23:52:25 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from 198.sub-166-250-69.myvzw.com (HELO [192.168.1.2]) (166.250.69.198) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Sun, 26 Aug 2012 00:52:24 +0100 Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: urls.txt-- format change? From: Anil Madhavapeddy In-Reply-To: <92B48648-1B39-4651-BF85-F745012E5E80@gmail.com> Date: Sat, 25 Aug 2012 16:52:20 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <655297C0-CC33-4E30-B57D-150050D94133@recoil.org> References: <67FD6763-9555-4ABD-A227-9FC9E31B5322@eu.citrix.com> <4A6DD509-303E-4D2B-9980-4313E20B19DA@recoil.org> <92B48648-1B39-4651-BF85-F745012E5E80@gmail.com> To: Thomas Gazagnaire X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Dave Scott , "cl-mirage@lists.cam.ac.uk Mailing List" X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 23:52:26 -0000 X-Keywords: X-UID: 512 Status: O Content-Length: 3139 Lines: 98 Very cool! It's a tiny bit broken under Linux with git repositories https://github.com/OCamlPro/opam/issues/77 ...but I think I've fixed it with my avsm/opam fix-linux-dir-cp branch. = Am doing a release of cstruct so we have versions of whatever we have = right now too. -anil On 25 Aug 2012, at 12:02, Thomas Gazagnaire = wrote: > btw you can init & remote -add github repo addresses directly >=20 > -- > Thomas >=20 >=20 > Le 25 ao=FBt 2012 =E0 20:28, Anil Madhavapeddy a = =E9crit : >=20 >> I've done a really quick re-jig of the repositories, so you can try: >>=20 >> $ opam init http://mirage.github.com/opam-0.4 >> $ opam remote -add local http://mirage.github.com/opam-0.4-dev >> $ opem install lwt # due to a bug in the package files >> $ opam install mirage >>=20 >> I'm still finishing up the dev conversion (a bit later today) and = will get mirage-www building with it (wont take too long). It's = probably easier to clone the repos yourself (mirage/opam-repository and = mirage/opam-repo-dev) and run "opam-mk-repo" yourself for now. >>=20 >> -anil >>=20 >> On 25 Aug 2012, at 10:21, Anil Madhavapeddy wrote: >>=20 >>> Thomas has done an epic set of changes in 0.4 which totally re-do = the package format to be more sensible, and adds checksums for packages, = and removes the need for a separate dev repository. >>>=20 >>> In the very short term, you can run 'opam-convert-repo-0.3' to the = new format on an existing repo, and here are the instructions Thomas = sent me earlier to add a new package. >>>=20 >>> 1. git clone git://github.com/OCamlPro/opam-repository.git >>> 2. git remote -add local opam-repository >>> 3. cd opam-repository/packages/ >>> 4. cp -r batteries.1.4.3 mlorg.xxx >>> 5. [edit mlorg.xxx/descr with the right description] >>> 6. [edit mlorg.xxx/url with the right archive name and checksum] >>> 7. opam update >>> [normally it should have picked the new mlorg package] >>> 8. opam install mlorg >>> 9. [in case of error, edit mlorg.xxx/opam and goto 7] >>>=20 >>> In the new world order, you can simply add a 'url: "git://foo"' to = the url file in a package, rather than the separate setup we had before. = I guess we need to maintain a fork for now with our dev packages, so I = am running the conversion script right now to update mirage.github.com. >>>=20 >>> This is hopefully the last of the big packaging changes (also famous = last words). >>>=20 >>> -anil >>>=20 >>> On 25 Aug 2012, at 10:16, Dave Scott = wrote: >>>=20 >>>> Hi, >>>>=20 >>>> I'm hitting a problem reinstalling the mirage stuff via opam which = I think is caused by a urls.txt format change. If you compare >>>>=20 >>>> http://opam.ocamlpro.com/urls.txt >>>>=20 >>>> with >>>>=20 >>>> http://mirage.github.com/opam/urls.txt >>>>=20 >>>> it looks like a re-arrangement. The error I get with "opam init = default ..." is consistent with it mis-parsing the file. >>>>=20 >>>> Anyone know how to regenerate the urls.txt? >>>>=20 >>>> Thanks, >>>>=20 >>>> --=20 >>>> Dave Scott >>>>=20 >>>>=20 >>>=20 >>>=20 >>=20 >>=20 >=20 From anil@recoil.org Sun Aug 26 01:52:46 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5R5p-0002HN-VZ (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 01:52:46 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:15997 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T5R5p-0000MZ-r3 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 01:52:45 +0100 Received: (qmail 3966 invoked by uid 634); 26 Aug 2012 00:52:45 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from pool-71-177-225-111.lsanca.fios.verizon.net (HELO [192.168.1.148]) (71.177.225.111) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Sun, 26 Aug 2012 01:52:44 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: opam and myocamlbuild.ml files From: Anil Madhavapeddy In-Reply-To: <20120719083308.GD24288@dark.recoil.org> Date: Sat, 25 Aug 2012 17:52:41 -0700 Content-Transfer-Encoding: 7bit Message-Id: <4AD758B2-905C-4F95-BCFE-CA3967F0D2A7@recoil.org> References: <42145234-747E-4254-A83F-A3BA29CDCEDA@nottingham.ac.uk> <658FD234-2748-472C-800E-BB35D7E75433@recoil.org> <46C88A77-65F0-4DE3-A9C0-25270C9423B9@nottingham.ac.uk> <524F04D4-BF81-4EC3-9EDF-DD7FCF2F85E6@recoil.org> <20120719083308.GD24288@dark.recoil.org> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Mirage List X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 00:52:46 -0000 X-Keywords: X-UID: 513 Status: O Content-Length: 2180 Lines: 53 On 19 Jul 2012, at 01:33, Anil Madhavapeddy wrote: > On Tue, Jul 17, 2012 at 07:13:09PM +0100, Richard Mortier wrote: >> >> On 17 Jul 2012, at 18:27, Anil Madhavapeddy wrote: >> >>> On 17 Jul 2012, at 17:52, Richard Mortier >>> wrote: >>> >>>> >>>> On 17 Jul 2012, at 10:35, Anil Madhavapeddy wrote: >>>> >>>>> Good question. We could move to only generating the setup.ml for >>>>> stable releases, and not for the working copy. The diff spam is >>>>> really irritating when bug bisecting... >>>> >>>> agree; at risk of repeating in other mail- could we even just move to >>>> oasis being a dependency and let opam handle it? >>> >>> Could do; but that would require different OPAM files for dev and >>> releases. Just needs someone to make sure that works... >> >> hm- don't follow. why does that require different opam files? >> >> (perhaps a more general question here is- what's the difference in >> concept between dev and release versions of an opam package?) > > A release package would ideally not require a dependency on OASIS, as its > designed like autoconf: the autogenerated files only need an OCaml > installation. > > So I was thinking that the dev packages would call "oasis setup" before > invoking make (to generate the ocamlbuild plugin), and released packages > would have that done already, to reduce the source dependencies for people > not using OPAM (which, believe it or not, is most people). To follow up on this, we should *not* introduce a compile-time dependency on OASIS at this stage. The reason is that our tool-chains don't do a very good job of cross-compiles and so a 'mirage-3.12.1-xen' OPAM toolchain might not be capable of producing UNIX binaries. And of course, when you are in 'OPAM xen mode', you cannot access the OASIS binary, which is probably installed in 'system' mode (in ~/.opam/system/bin). I've reverted the DNS dependency on OASIS which fixes the problem for now. There'll be plenty of time to worry about cross-compilation when integrating Gabor's kFreeBSD backend at the same time, which is even more sensitive to such issues than the Xen one :-) -anil From Richard.Mortier@nottingham.ac.uk Sun Aug 26 19:39:12 2012 Received: from ppsw-43.csi.cam.ac.uk ([131.111.8.143]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5hjs-0002Vu-5x (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 19:39:12 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:37907 helo=ixe-mta-19.emailfiltering.com) by ppsw-43.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.149]:25) with esmtp id 1T5hjm-0001Qg-np (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 19:39:12 +0100 Received: from smtp4.nottingham.ac.uk ([128.243.220.65]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2352365200 ;dff194cab2351cec; Sun, 26 Aug 2012 19:38:46 +0100 Received: from uiwexhub02.ad.nottingham.ac.uk ([128.243.15.132]) by smtp4.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T5hjS-0002Uy-0O for cl-mirage@lists.cam.ac.uk; Sun, 26 Aug 2012 19:38:46 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB02.ad.nottingham.ac.uk ([2002:80f3:f84::80f3:f84]) with mapi; Sun, 26 Aug 2012 19:38:29 +0100 From: Richard Mortier To: Anil Madhavapeddy Date: Sun, 26 Aug 2012 19:38:31 +0100 Subject: Re: opam and myocamlbuild.ml files Thread-Topic: opam and myocamlbuild.ml files Thread-Index: Ac2DufvhO1AsJl/eT1m4B+dPFLL8mQ== Message-ID: References: <42145234-747E-4254-A83F-A3BA29CDCEDA@nottingham.ac.uk> <658FD234-2748-472C-800E-BB35D7E75433@recoil.org> <46C88A77-65F0-4DE3-A9C0-25270C9423B9@nottingham.ac.uk> <524F04D4-BF81-4EC3-9EDF-DD7FCF2F85E6@recoil.org> <20120719083308.GD24288@dark.recoil.org> <4AD758B2-905C-4F95-BCFE-CA3967F0D2A7@recoil.org> In-Reply-To: <4AD758B2-905C-4F95-BCFE-CA3967F0D2A7@recoil.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Mirage List X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:39:12 -0000 X-Keywords: X-UID: 514 Status: O Content-Length: 1866 Lines: 55 On 26 Aug 2012, at 01:52, Anil Madhavapeddy wrote: > To follow up on this, we should *not* introduce a compile-time dependency > on OASIS at this stage. >=20 > The reason is that our tool-chains don't do a very good job of cross-comp= iles > and so a 'mirage-3.12.1-xen' OPAM toolchain might not be capable of produ= cing > UNIX binaries. hm- i guess i don't use the -xen toolchain very often - could you expand a = bit on why this is an issue? (or is that tantamount to just fixing it cf t= he mirage/bsd comment below? :) > And of course, when you are in 'OPAM xen mode', you cannot access the OAS= IS > binary, which is probably installed in 'system' mode (in ~/.opam/system/b= in). > I've reverted the DNS dependency on OASIS which fixes the problem for now= . ok- what's the current "best practice" as a replacement for OASIS? > There'll be plenty of time to worry about cross-compilation when integrat= ing > Gabor's kFreeBSD backend at the same time, which is even more sensitive t= o > such issues than the Xen one :-) "oh good"? :) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From anil@recoil.org Sun Aug 26 19:58:42 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T5i2k-0002eI-3f (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 19:58:42 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:42199 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T5i2i-00040F-T1 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Sun, 26 Aug 2012 19:58:42 +0100 Received: (qmail 9028 invoked by uid 634); 26 Aug 2012 18:58:40 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from 171.sub-166-250-64.myvzw.com (HELO [192.168.1.2]) (166.250.64.171) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Sun, 26 Aug 2012 19:58:40 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: opam and myocamlbuild.ml files From: Anil Madhavapeddy In-Reply-To: Date: Sun, 26 Aug 2012 11:58:36 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <1ADDFE8A-A381-45E2-99F3-CB977435C542@recoil.org> References: <42145234-747E-4254-A83F-A3BA29CDCEDA@nottingham.ac.uk> <658FD234-2748-472C-800E-BB35D7E75433@recoil.org> <46C88A77-65F0-4DE3-A9C0-25270C9423B9@nottingham.ac.uk> <524F04D4-BF81-4EC3-9EDF-DD7FCF2F85E6@recoil.org> <20120719083308.GD24288@dark.recoil.org> <4AD758B2-905C-4F95-BCFE-CA3967F0D2A7@recoil.org> To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Mirage List X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:58:42 -0000 X-Keywords: X-UID: 515 Status: O Content-Length: 2601 Lines: 67 On 26 Aug 2012, at 11:38, Richard Mortier = wrote: >=20 > On 26 Aug 2012, at 01:52, Anil Madhavapeddy wrote: >=20 >> To follow up on this, we should *not* introduce a compile-time = dependency >> on OASIS at this stage. >>=20 >> The reason is that our tool-chains don't do a very good job of = cross-compiles >> and so a 'mirage-3.12.1-xen' OPAM toolchain might not be capable of = producing >> UNIX binaries. >=20 > hm- i guess i don't use the -xen toolchain very often - could you = expand a bit on why this is an issue? (or is that tantamount to just = fixing it cf the mirage/bsd comment below? :) In the Mirage/Xen stdlib, we overwrite various core libraries with the = Xen-specialised version (notably Bigarray). So if we use happen to use = those changed libraries in a bit of code intended for UNIX delivery, = things go wrong. The reason is a pretty standard problem in cross-compilation: we need to = separate the *host* toolchain from the *target* toolchain. =20 The host toolchain includes all camlp4 extensions (which only run during = the compilation process), and also any platform binaries such as OASIS = that are used to regenerate makefiles. The target toolchain is all the code that eventually makes it into the = binary, including the *output* of the camlp4 host extensions (which is = standard OCaml code). Solving this comprehensively will require phase 2 of the Grand Packaging = Plan, which is to extend OPAM to integrate with the build system of the = applications. Thomas/OCamlPro have already begun this via the ocp-build = system (which is used to build OPAM), but it's a little while away from = being usable within Mirage itself. For the moment however, there is no need to panic. Aside from OASIS = itself, there isn't a driving need to separate the toolchains until we = come to the FreeBSD target (which is really very sensitive to issues = such as the presence of floating point). >=20 >> And of course, when you are in 'OPAM xen mode', you cannot access the = OASIS >> binary, which is probably installed in 'system' mode (in = ~/.opam/system/bin). >> I've reverted the DNS dependency on OASIS which fixes the problem for = now. >=20 > ok- what's the current "best practice" as a replacement for OASIS? Nothing! Just use OASIS, but checkin the generated files, and all will = just work. However, if your library needs C extensions, then somewhat more delicate = surgery will be required. I'm working on that for Cryptokit at the = moment, as ocaml-dns now has a requirement on that due to the DNSSEC = code. -a= From anil@recoil.org Mon Aug 27 23:50:30 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T688c-00011V-PS (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 27 Aug 2012 23:50:30 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:36688 helo=dark.recoil.org) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with smtp id 1T688b-0004nM-qU (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Mon, 27 Aug 2012 23:50:30 +0100 Received: (qmail 14416 invoked by uid 634); 27 Aug 2012 22:50:28 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from Unknown (HELO [10.183.90.174]) (38.96.16.254) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Mon, 27 Aug 2012 23:50:28 +0100 From: Anil Madhavapeddy Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: oasis patch that adds an .nobj.o target Message-Id: Date: Mon, 27 Aug 2012 15:50:24 -0700 To: David Scott , Mirage List Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 22:50:30 -0000 X-Keywords: X-UID: 516 Status: O Content-Length: 3618 Lines: 90 This patch against darcs-oasis is sufficient to add a "native output = obj" target that outputs a standalone object file. Just one more little = rule required to do the Xen objcopy/link, but this will also be useful = for kFreeBSD too. This one is mainly for Dave as he's working on = ocaml-xenstore atm. I'll submit an upstreamable patch shortly and have a Xen plugin to OASIS = that will make all our build pains waft away into a gentle summer = breeze. -anil avsm@debian:~/src/darcs/oasis$ darcs diff diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.249486856 +0100 +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.301486856 +0100 @@ -165,6 +165,15 @@ ) t.lib_c; =20 + (* Add output_obj rules mapped to .nobj.o *) + let native_output_obj x =3D + OC.link_gen "cmx" "cmxa" !Options.ext_lib = [!Options.ext_obj; "cmi"]=20 + OC.ocamlopt_link_prog + (fun tags -> tags++"ocaml"++"link"++"byte"++"output_obj") = x + in + rule "ocaml: cmx* and o* -> .nobj.o" ~prod:"%.nobj.o" = ~deps:["%.cmx"; "%.o"] + (native_output_obj "%.cmx" "%.nobj.o"); + (* Add flags *) List.iter (fun (tags, cond_specs) -> diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.249486856 +0100 +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.301486856 +0100 @@ -86,6 +86,7 @@ =20 (* When one link an OCaml library/binary/package, one should = use -linkpkg *) flag ["ocaml"; "link"; "program"] & A"-linkpkg"; + flag ["ocaml"; "link"; "output_obj"] & A"-linkpkg"; =20 (* For each ocamlfind package one inject the -package option = when * compiling, computing dependencies, generating documentation = and diff -rN -u old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml --- old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.249486856 +0100 +++ new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.301486856 +0100 @@ -158,9 +158,9 @@ let acc =3D match bs.bs_compiled_object with | Native -> - (target ".native") :: acc + (target ".native") :: (target ".nobj.o") :: = acc | Best when bool_of_string (is_native ()) -> - (target ".native") :: acc + (target ".native") :: (target ".nobj.o") :: = acc | Byte | Best -> (target ".byte") :: acc @@ -928,11 +928,11 @@ let ext =3D match bs.bs_compiled_object with | Best -> - "{native,byte}" + "{native,byte,nobj.o}" | Byte -> "byte" | Native -> - "native" + "native,nobj.o" in prepend_bs_path bs (OASISUnixPath.replace_extension= From anil@recoil.org Tue Aug 28 23:37:18 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T6UPO-0001l1-1q (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 28 Aug 2012 23:37:18 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:22435 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T6UPN-0007LL-Qe (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Tue, 28 Aug 2012 23:37:17 +0100 Received: (qmail 10892 invoked by uid 634); 28 Aug 2012 22:37:17 -0000 X-Spam-Level: * X-Spam-Status: No, hits=-1.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: dark.recoil.org Received: from Unknown (HELO [10.183.90.174]) (38.96.16.254) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Tue, 28 Aug 2012 23:37:16 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: oasis patch that adds an .nobj.o target From: Anil Madhavapeddy In-Reply-To: Date: Tue, 28 Aug 2012 15:37:11 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: David Scott , Mirage List X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 22:37:18 -0000 X-Keywords: X-UID: 517 Status: O Content-Length: 4357 Lines: 109 I've added in the Xen rules to OASIS now too, so that you can do = 'ocamlbuild sleep.xen' in ocaml-xenstore successfully. My forked = repository is at: https://github.com/avsm/oasis/tree/add-xen Dave, this works on ocaml-xenstore now if you regenerate myocamlbuild.ml = with this OASIS (but dont forget to move the existing myocamlbuild.ml = out of the way first). I'm figuring out how to add this as a plugin, so it'll only get added = when specified as 'Plugin: xen' in OASIS. Also, we need the Executable = field in _oasis to build stuff with a .xen suffix. -anil On 27 Aug 2012, at 15:50, Anil Madhavapeddy wrote: > This patch against darcs-oasis is sufficient to add a "native output = obj" target that outputs a standalone object file. Just one more little = rule required to do the Xen objcopy/link, but this will also be useful = for kFreeBSD too. This one is mainly for Dave as he's working on = ocaml-xenstore atm. >=20 > I'll submit an upstreamable patch shortly and have a Xen plugin to = OASIS that will make all our build pains waft away into a gentle summer = breeze. >=20 > -anil >=20 > avsm@debian:~/src/darcs/oasis$ darcs diff > diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml > --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.249486856 +0100 > +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.301486856 +0100 > @@ -165,6 +165,15 @@ > ) > t.lib_c; >=20 > + (* Add output_obj rules mapped to .nobj.o *) > + let native_output_obj x =3D > + OC.link_gen "cmx" "cmxa" !Options.ext_lib = [!Options.ext_obj; "cmi"]=20 > + OC.ocamlopt_link_prog > + (fun tags -> = tags++"ocaml"++"link"++"byte"++"output_obj") x > + in > + rule "ocaml: cmx* and o* -> .nobj.o" ~prod:"%.nobj.o" = ~deps:["%.cmx"; "%.o"] > + (native_output_obj "%.cmx" "%.nobj.o"); > + > (* Add flags *) > List.iter > (fun (tags, cond_specs) -> > diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml > --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.249486856 +0100 > +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.301486856 +0100 > @@ -86,6 +86,7 @@ >=20 > (* When one link an OCaml library/binary/package, one should = use -linkpkg *) > flag ["ocaml"; "link"; "program"] & A"-linkpkg"; > + flag ["ocaml"; "link"; "output_obj"] & A"-linkpkg"; >=20 > (* For each ocamlfind package one inject the -package option = when > * compiling, computing dependencies, generating documentation = and > diff -rN -u old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml > --- old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.249486856 +0100 > +++ new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.301486856 +0100 > @@ -158,9 +158,9 @@ > let acc =3D > match bs.bs_compiled_object with > | Native -> > - (target ".native") :: acc > + (target ".native") :: (target ".nobj.o") :: = acc > | Best when bool_of_string (is_native ()) -> > - (target ".native") :: acc > + (target ".native") :: (target ".nobj.o") :: = acc > | Byte > | Best -> > (target ".byte") :: acc > @@ -928,11 +928,11 @@ > let ext =3D > match bs.bs_compiled_object with > | Best -> > - "{native,byte}" > + "{native,byte,nobj.o}" > | Byte -> > "byte" > | Native -> > - "native" > + "native,nobj.o" > in > prepend_bs_path bs > (OASISUnixPath.replace_extension From scott.dj@gmail.com Wed Aug 29 10:46:49 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T6erJ-00051v-3z (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 29 Aug 2012 10:46:49 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.9 from SpamAssassin-3.3.2-1375947 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.220.179 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (scott.dj[at]gmail.com) * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from * author's domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-vc0-f179.google.com ([209.85.220.179]:64302) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T6erH-0003PF-sU (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 29 Aug 2012 10:46:49 +0100 Received: by vcqp16 with SMTP id p16so353023vcq.38 for ; Wed, 29 Aug 2012 02:46:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.33.234 with SMTP id u10mr780451vei.49.1346233607244; Wed, 29 Aug 2012 02:46:47 -0700 (PDT) Received: by 10.58.213.168 with HTTP; Wed, 29 Aug 2012 02:46:47 -0700 (PDT) In-Reply-To: References: Date: Wed, 29 Aug 2012 10:46:47 +0100 Message-ID: Subject: Re: oasis patch that adds an .nobj.o target From: David Scott To: Anil Madhavapeddy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Mirage List X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 09:46:49 -0000 X-Keywords: X-UID: 518 Status: O Content-Length: 4829 Lines: 126 Hi, Thanks for that -- "ocamlbuild xen/sleep.xen" is working for me now (modulo bugs which are obviously mine in origin) I also think I'm beginning to get the hang of opam; 'switching' to 'system' to install stuff needed by oasis and then 'switching' back to the xen toolchain for actual builds. Cheers, Dave On Tue, Aug 28, 2012 at 11:37 PM, Anil Madhavapeddy wrote= : > I've added in the Xen rules to OASIS now too, so that you can do 'ocamlbu= ild sleep.xen' in ocaml-xenstore successfully. My forked repository is at: > > https://github.com/avsm/oasis/tree/add-xen > > Dave, this works on ocaml-xenstore now if you regenerate myocamlbuild.ml = with this OASIS (but dont forget to move the existing myocamlbuild.ml out o= f the way first). > > I'm figuring out how to add this as a plugin, so it'll only get added whe= n specified as 'Plugin: xen' in OASIS. Also, we need the Executable field= in _oasis to build stuff with a .xen suffix. > > -anil > > On 27 Aug 2012, at 15:50, Anil Madhavapeddy wrote: > >> This patch against darcs-oasis is sufficient to add a "native output obj= " target that outputs a standalone object file. Just one more little rule = required to do the Xen objcopy/link, but this will also be useful for kFree= BSD too. This one is mainly for Dave as he's working on ocaml-xenstore atm= . >> >> I'll submit an upstreamable patch shortly and have a Xen plugin to OASIS= that will make all our build pains waft away into a gentle summer breeze. >> >> -anil >> >> avsm@debian:~/src/darcs/oasis$ darcs diff >> diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml new-oas= is/src/plugins/ocamlbuild/MyOCamlbuildBase.ml >> --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml 2012-08-= 24 18:45:09.249486856 +0100 >> +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml 2012-08-= 24 18:45:09.301486856 +0100 >> @@ -165,6 +165,15 @@ >> ) >> t.lib_c; >> >> + (* Add output_obj rules mapped to .nobj.o *) >> + let native_output_obj x =3D >> + OC.link_gen "cmx" "cmxa" !Options.ext_lib [!Options.ext_obj= ; "cmi"] >> + OC.ocamlopt_link_prog >> + (fun tags -> tags++"ocaml"++"link"++"byte"++"output_obj")= x >> + in >> + rule "ocaml: cmx* and o* -> .nobj.o" ~prod:"%.nobj.o" ~deps:[= "%.cmx"; "%.o"] >> + (native_output_obj "%.cmx" "%.nobj.o"); >> + >> (* Add flags *) >> List.iter >> (fun (tags, cond_specs) -> >> diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml new-= oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml >> --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml 2012-08-= 24 18:45:09.249486856 +0100 >> +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml 2012-08-= 24 18:45:09.301486856 +0100 >> @@ -86,6 +86,7 @@ >> >> (* When one link an OCaml library/binary/package, one should use= -linkpkg *) >> flag ["ocaml"; "link"; "program"] & A"-linkpkg"; >> + flag ["ocaml"; "link"; "output_obj"] & A"-linkpkg"; >> >> (* For each ocamlfind package one inject the -package option whe= n >> * compiling, computing dependencies, generating documentation a= nd >> diff -rN -u old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml new-oas= is/src/plugins/ocamlbuild/OCamlbuildPlugin.ml >> --- old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml 2012-08-= 24 18:45:09.249486856 +0100 >> +++ new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml 2012-08-= 24 18:45:09.301486856 +0100 >> @@ -158,9 +158,9 @@ >> let acc =3D >> match bs.bs_compiled_object with >> | Native -> >> - (target ".native") :: acc >> + (target ".native") :: (target ".nobj.o") :: a= cc >> | Best when bool_of_string (is_native ()) -> >> - (target ".native") :: acc >> + (target ".native") :: (target ".nobj.o") :: ac= c >> | Byte >> | Best -> >> (target ".byte") :: acc >> @@ -928,11 +928,11 @@ >> let ext =3D >> match bs.bs_compiled_object with >> | Best -> >> - "{native,byte}" >> + "{native,byte,nobj.o}" >> | Byte -> >> "byte" >> | Native -> >> - "native" >> + "native,nobj.o" >> in >> prepend_bs_path bs >> (OASISUnixPath.replace_extension > --=20 Dave Scott From Richard.Mortier@nottingham.ac.uk Wed Aug 29 10:55:29 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T6ezh-0005IT-3i (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 29 Aug 2012 10:55:29 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:39833 helo=ixe-mta-19.emailfiltering.com) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T6eza-0006sV-sB (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Wed, 29 Aug 2012 10:55:29 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2354615628 ;85f2d6b4eeb63168; Wed, 29 Aug 2012 10:55:02 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T6ezG-0003UF-G4 for cl-mirage@lists.cam.ac.uk; Wed, 29 Aug 2012 10:55:02 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Wed, 29 Aug 2012 10:55:02 +0100 From: Richard Mortier To: David Scott Date: Wed, 29 Aug 2012 10:55:04 +0100 Subject: Re: oasis patch that adds an .nobj.o target Thread-Topic: oasis patch that adds an .nobj.o target Thread-Index: Ac2FzFucjBQVCngTQx2S0X7D6E9nog== Message-ID: References: In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Mirage List , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 09:55:29 -0000 X-Keywords: X-UID: 519 Status: O Content-Length: 1544 Lines: 44 On 29 Aug 2012, at 10:46, David Scott wrote: > Hi, >=20 > Thanks for that -- "ocamlbuild xen/sleep.xen" is working for me now > (modulo bugs which are obviously mine in origin) >=20 > I also think I'm beginning to get the hang of opam; 'switching' to > 'system' to install stuff needed by oasis and then 'switching' back to > the xen toolchain for actual builds. ... care to expand? :) (more seriously, this feels like the kind of documentation/detail that woul= d be really handy to have recorded in the mirage blog/docs/tutorial/blah...= not just the "here's how to switch" docs as exist now but "here's a specif= ic example of when to switch and why") (apologies if that's already there and i've not been keeping up.) --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From scott.dj@gmail.com Thu Aug 30 11:27:48 2012 Received: from ppsw-51.csi.cam.ac.uk ([131.111.8.151]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T71yW-0005vx-2E (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 11:27:48 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score 0.2 from SpamAssassin-3.3.2-1375947 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [209.85.212.51 listed in list.dnswl.dnsbl.ja.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (scott.dj[at]gmail.com) * 0.0 DKIM_ADSP_CUSTOM_MED No valid author signature, adsp_override is * CUSTOM_MED * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid * 0.9 NML_ADSP_CUSTOM_MED ADSP custom_med hit, and not from a mailing * list X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mail-vb0-f51.google.com ([209.85.212.51]:44801) by ppsw-51.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.148]:25) with esmtp id 1T71yV-0000O3-Y0 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 11:27:48 +0100 Received: by vbbfn1 with SMTP id fn1so1527608vbb.38 for ; Thu, 30 Aug 2012 03:27:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.175.130 with SMTP id ca2mr2234448vdc.112.1346322466663; Thu, 30 Aug 2012 03:27:46 -0700 (PDT) Received: by 10.58.213.168 with HTTP; Thu, 30 Aug 2012 03:27:46 -0700 (PDT) In-Reply-To: References: Date: Thu, 30 Aug 2012 11:27:46 +0100 Message-ID: Subject: Re: oasis patch that adds an .nobj.o target From: David Scott To: Richard Mortier Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Mirage List , Anil Madhavapeddy X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2012 10:27:48 -0000 X-Keywords: X-UID: 520 Status: O Content-Length: 2692 Lines: 78 On Wed, Aug 29, 2012 at 10:55 AM, Richard Mortier wrote: > > On 29 Aug 2012, at 10:46, David Scott wrote: >> I also think I'm beginning to get the hang of opam; 'switching' to >> 'system' to install stuff needed by oasis and then 'switching' back to >> the xen toolchain for actual builds. > > ... care to expand? > > :) > > (more seriously, this feels like the kind of documentation/detail that wo= uld be really handy to have recorded in the mirage blog/docs/tutorial/blah.= .. not just the "here's how to switch" docs as exist now but "here's a spec= ific example of when to switch and why") > > (apologies if that's already there and i've not been keeping up.) I agree we'll need some decent tutorials to explain how to use this stuff. I think this is what I did: (it was more of a random walk in reality but here goes) I've got a debian install with access to a lot of fairly stable ocaml packages. When I wanted to use Anil's branch of oasis I switched back to 'system' so that I could 'apt-get install' all the dependencies quickly (assuming that opam hasn't got all the packages yet). I then discovered that I needed one or two package upgrades but I just quickly downloaded and manually built these, installing them in the .opam directory. When I had Anil's oasis I used it to regenerate the build scripts for the mirage xenstore code I'm working on. I then switched back to '3.12...xen' to build and link the mirage xen kernel -- this requires a slightly modified base system. So for the website we could have a set of these: As a busy ... insert description of what I actually do ... I like the way opam can keep separate stable code which I need for everyday tasks, from the less stable and/or cutdown stuff that I need for cutting-edge mirage xen development (ahem). :-) Cheers, Dave > > -- > Cheers, > > R. > > > > > This message and any attachment are intended solely for the addressee and= may contain confidential information. If you have received this message in= error, please send it back to me, and immediately delete it. Please do n= ot use, copy or disclose the information contained in this message or in an= y attachment. Any views or opinions expressed by the author of this email = do not necessarily reflect the views of the University of Nottingham. > > This message has been checked for viruses but the contents of an attachme= nt > may still contain software viruses which could damage your computer syste= m: > you are advised to perform your own checks. Email communications with the > University of Nottingham may be monitored as permitted by UK legislation. --=20 Dave Scott From anil@recoil.org Thu Aug 30 19:44:14 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T79iw-0001uI-GR (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 19:44:14 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:34286 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T79iv-0000EY-Qv (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 19:44:14 +0100 Received: (qmail 23111 invoked by uid 634); 30 Aug 2012 18:44:13 -0000 Received: from 183.sub-166-250-37.myvzw.com (HELO [192.168.1.2]) (166.250.37.183) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 30 Aug 2012 19:44:13 +0100 Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: oasis patch that adds an .nobj.o target From: Anil Madhavapeddy In-Reply-To: Date: Thu, 30 Aug 2012 11:44:08 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <965D626F-F757-4E15-AC60-BA872AE3B4B3@recoil.org> References: To: David Scott X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Mirage List X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2012 18:44:14 -0000 X-Keywords: X-UID: 521 Status: O Content-Length: 6049 Lines: 158 I've put in a further changeset into OASIS to add a new "Target" field to the Executable section. Just set this to "xen" and it will build = that target instead. You still need CompiledObject:native as that will put the correct _tags entries in for the .nobj.o target (which is the final link phase). See: = https://github.com/avsm/ocaml-xenstore/commit/0c1b80ddba612487b7f97ff4c244= 4a10e451196d ...for what I mean in ocaml-xenstore. So given that we should be able to compile the libraries with *either* --enable-unix or --enable-xen, and the appropriate binaries will be = built with the current toolchain and OASIS. Good enough for a first release! The OPAM packaging can signal the current environment that is set via environment variables --- currently MIRAGE_OS and MIRAGE_XEN. I guess the final step is a ./configure script that detects these and translates them into the correct 'ocaml setup.ml --enable-foo' flags. Will do that when I'm off this train and in the office... -anil On 29 Aug 2012, at 02:46, David Scott wrote: > Hi, >=20 > Thanks for that -- "ocamlbuild xen/sleep.xen" is working for me now > (modulo bugs which are obviously mine in origin) >=20 > I also think I'm beginning to get the hang of opam; 'switching' to > 'system' to install stuff needed by oasis and then 'switching' back to > the xen toolchain for actual builds. >=20 > Cheers, > Dave >=20 > On Tue, Aug 28, 2012 at 11:37 PM, Anil Madhavapeddy = wrote: >> I've added in the Xen rules to OASIS now too, so that you can do = 'ocamlbuild sleep.xen' in ocaml-xenstore successfully. My forked = repository is at: >>=20 >> https://github.com/avsm/oasis/tree/add-xen >>=20 >> Dave, this works on ocaml-xenstore now if you regenerate = myocamlbuild.ml with this OASIS (but dont forget to move the existing = myocamlbuild.ml out of the way first). >>=20 >> I'm figuring out how to add this as a plugin, so it'll only get added = when specified as 'Plugin: xen' in OASIS. Also, we need the Executable = field in _oasis to build stuff with a .xen suffix. >>=20 >> -anil >>=20 >> On 27 Aug 2012, at 15:50, Anil Madhavapeddy wrote: >>=20 >>> This patch against darcs-oasis is sufficient to add a "native output = obj" target that outputs a standalone object file. Just one more little = rule required to do the Xen objcopy/link, but this will also be useful = for kFreeBSD too. This one is mainly for Dave as he's working on = ocaml-xenstore atm. >>>=20 >>> I'll submit an upstreamable patch shortly and have a Xen plugin to = OASIS that will make all our build pains waft away into a gentle summer = breeze. >>>=20 >>> -anil >>>=20 >>> avsm@debian:~/src/darcs/oasis$ darcs diff >>> diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml >>> --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.249486856 +0100 >>> +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildBase.ml = 2012-08-24 18:45:09.301486856 +0100 >>> @@ -165,6 +165,15 @@ >>> ) >>> t.lib_c; >>>=20 >>> + (* Add output_obj rules mapped to .nobj.o *) >>> + let native_output_obj x =3D >>> + OC.link_gen "cmx" "cmxa" !Options.ext_lib = [!Options.ext_obj; "cmi"] >>> + OC.ocamlopt_link_prog >>> + (fun tags -> = tags++"ocaml"++"link"++"byte"++"output_obj") x >>> + in >>> + rule "ocaml: cmx* and o* -> .nobj.o" ~prod:"%.nobj.o" = ~deps:["%.cmx"; "%.o"] >>> + (native_output_obj "%.cmx" "%.nobj.o"); >>> + >>> (* Add flags *) >>> List.iter >>> (fun (tags, cond_specs) -> >>> diff -rN -u old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml >>> --- old-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.249486856 +0100 >>> +++ new-oasis/src/plugins/ocamlbuild/MyOCamlbuildFindlib.ml = 2012-08-24 18:45:09.301486856 +0100 >>> @@ -86,6 +86,7 @@ >>>=20 >>> (* When one link an OCaml library/binary/package, one should = use -linkpkg *) >>> flag ["ocaml"; "link"; "program"] & A"-linkpkg"; >>> + flag ["ocaml"; "link"; "output_obj"] & A"-linkpkg"; >>>=20 >>> (* For each ocamlfind package one inject the -package option = when >>> * compiling, computing dependencies, generating = documentation and >>> diff -rN -u old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml >>> --- old-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.249486856 +0100 >>> +++ new-oasis/src/plugins/ocamlbuild/OCamlbuildPlugin.ml = 2012-08-24 18:45:09.301486856 +0100 >>> @@ -158,9 +158,9 @@ >>> let acc =3D >>> match bs.bs_compiled_object with >>> | Native -> >>> - (target ".native") :: acc >>> + (target ".native") :: (target ".nobj.o") = :: acc >>> | Best when bool_of_string (is_native ()) -> >>> - (target ".native") :: acc >>> + (target ".native") :: (target ".nobj.o") = :: acc >>> | Byte >>> | Best -> >>> (target ".byte") :: acc >>> @@ -928,11 +928,11 @@ >>> let ext =3D >>> match bs.bs_compiled_object with >>> | Best -> >>> - "{native,byte}" >>> + "{native,byte,nobj.o}" >>> | Byte -> >>> "byte" >>> | Native -> >>> - "native" >>> + "native,nobj.o" >>> in >>> prepend_bs_path bs >>> (OASISUnixPath.replace_extension >>=20 >=20 >=20 >=20 > --=20 > Dave Scott >=20 From anil@recoil.org Thu Aug 30 19:46:23 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T79l1-0001v2-5r (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 19:46:23 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from recoil.dh.bytemark.co.uk ([89.16.177.154]:28844 helo=dark.recoil.org) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with smtp id 1T79l0-0000t5-Po (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Thu, 30 Aug 2012 19:46:23 +0100 Received: (qmail 19747 invoked by uid 634); 30 Aug 2012 18:46:21 -0000 Received: from 183.sub-166-250-37.myvzw.com (HELO [192.168.1.2]) (166.250.37.183) (smtp-auth username remote@recoil.org, mechanism cram-md5) by dark.recoil.org (qpsmtpd/0.84) with ESMTPA; Thu, 30 Aug 2012 19:46:21 +0100 Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\)) Subject: Re: oasis patch that adds an .nobj.o target From: Anil Madhavapeddy In-Reply-To: Date: Thu, 30 Aug 2012 11:46:17 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <2751919B-08F8-4A97-B89A-A67E66ED7C80@recoil.org> References: To: Richard Mortier X-Mailer: Apple Mail (2.1485) X-Virus-Checked: Checked by ClamAV on dark.recoil.org Cc: Mirage List , David Scott X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2012 18:46:23 -0000 X-Keywords: X-UID: 522 Status: O Content-Length: 1325 Lines: 38 On 29 Aug 2012, at 02:55, Richard Mortier = wrote: >=20 > On 29 Aug 2012, at 10:46, David Scott wrote: >=20 >> Hi, >>=20 >> Thanks for that -- "ocamlbuild xen/sleep.xen" is working for me now >> (modulo bugs which are obviously mine in origin) >>=20 >> I also think I'm beginning to get the hang of opam; 'switching' to >> 'system' to install stuff needed by oasis and then 'switching' back = to >> the xen toolchain for actual builds. >=20 > ... care to expand? >=20 > :) >=20 > (more seriously, this feels like the kind of documentation/detail that = would be really handy to have recorded in the mirage = blog/docs/tutorial/blah... not just the "here's how to switch" docs as = exist now but "here's a specific example of when to switch and why") >=20 > (apologies if that's already there and i've not been keeping up.) In a nutshell: you switch when you need a different toolchain, and you = need a different toolchain for different targets (unix, xen, kfreebsd, = ns3, javascript). Each toolchain will have platform-specific hacks and = restrictions that may render it unable to target other binaries. Now that I've got OASIS working, I'm going to ripple up the changes = through to mirage-www, and then we can get the blog updatable again. The = joys of self-hosting ;-) -anil= From Richard.Mortier@nottingham.ac.uk Fri Aug 31 11:53:08 2012 Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T7OqZ-0006hU-Vl (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 31 Aug 2012 11:53:07 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -0.1 from SpamAssassin-3.3.2-1375947 * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from ixe-mta-19-tx.emailfiltering.com ([194.116.198.150]:40413 helo=ixe-mta-19.emailfiltering.com) by ppsw-41.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.146]:25) with esmtp id 1T7OqQ-0004wB-Q7 (Exim 4.72) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 31 Aug 2012 11:53:07 +0100 Received: from smtp3.nottingham.ac.uk ([128.243.44.55]) by ixe-mta-19.emailfiltering.com with emfmta (version 4.8.5.104) by TLS id 2356770688 ;85d67dfd6e93c0e6; Fri, 31 Aug 2012 11:52:37 +0100 Received: from uiwexhub01.ad.nottingham.ac.uk ([128.243.15.133]) by smtp3.nottingham.ac.uk with esmtps (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T7Oq5-0007cI-Iu for cl-mirage@lists.cam.ac.uk; Fri, 31 Aug 2012 11:52:37 +0100 Received: from EXCHANGE1.ad.nottingham.ac.uk ([fe80::60c2:2879:5a5a:29a6]) by UIWEXHUB01.ad.nottingham.ac.uk ([2002:80f3:f85::80f3:f85]) with mapi; Fri, 31 Aug 2012 11:52:37 +0100 From: Richard Mortier To: Mirage List Date: Fri, 31 Aug 2012 11:52:37 +0100 Subject: experiencing problems... Thread-Topic: experiencing problems... Thread-Index: Ac2HZrwJ+RA3zMD6SLKTuX6cacrUoQ== Message-ID: <3D2C25BE-204C-415A-908C-C4054885ECD8@nottingham.ac.uk> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 10:53:08 -0000 X-Keywords: X-UID: 523 Status: O Content-Length: 2098 Lines: 49 trying (finally!) to get setup to do the performance test of dns-cstruct vs= dns-bitstring. seem to be having some problems which are probably all solv= ed somewhere. (and i also seem to be having problems getting my irc tunnel= s setup to connect from the office else i'd ask this on irc...) * http://tutorial.openmirage.org/ currently returns a connection refused * opam init default ... per the website fails complaining that a number of = urls can't be found, eg., http://mirage.github.com/opam/0o0755. dug out ani= l's last mail pointing to the opam-0.4 repo and that seems to work though. * minor niggle- just did a base install, and then opam switch-ed to xen and= then unix-direct versions to get them installed. seemed to use wget to dow= nload urls.txt a number of times, without overwriting, so i now have urls.t= xt.[1-9] in addition to urls.txt in /home/rmm/.opam/repo/default/ -- is tha= t intentional? * (opam install dns) dependency on oasis, re, uri not captured * (unix) opam install re fails as docs want to be built into /usr/local/sha= re/doc/ lost track of opam development slightly - for the last two items, should i = just clone the repo at (or ?) and fix things? --=20 Cheers, R. = This message and any attachment are intended solely for the addressee a= nd may contain confidential information. If you have received this mess= age in error, please send it back to me, and immediately delete it. P= lease do not use, copy or disclose the information contained in this me= ssage or in any attachment. Any views or opinions expressed by the aut= hor of this email do not necessarily reflect the views of the Universit= y of Nottingham.=0D=0A=0D=0AThis message has been checked for viruses b= ut the contents of an attachment=0D=0Amay still contain software viruse= s which could damage your computer system:=0D=0Ayou are advised to perf= orm your own checks. Email communications with the=0D=0AUniversity of N= ottingham may be monitored as permitted by UK legislation.= From pgj@caesar.elte.hu Fri Aug 31 18:48:38 2012 Received: from ppsw-50.csi.cam.ac.uk ([131.111.8.150]) by lists-2.csi.cam.ac.uk (lists.cam.ac.uk [131.111.8.15]:25) with esmtp id 1T7VKg-0000ZO-KK (Exim 4.70) for cl-mirage@lists.cam.ac.uk (return-path ); Fri, 31 Aug 2012 18:48:38 +0100 X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: score -1.0 from SpamAssassin-3.3.2-1375947 * -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low * trust * [157.181.151.9 listed in list.dnswl.dnsbl.ja.net] * -0.2 RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * -0.1 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Cam-ScannerInfo: http://www.cam.ac.uk/cs/email/scanner/ Received: from mx2.mail.elte.hu ([157.181.151.9]:49407) by ppsw-50.csi.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) with esmtp id 1T7VKe-0005yC-qT (Exim 4.72) (return-path ); Fri, 31 Aug 2012 18:48:38 +0100 Received: from login01.caesar.elte.hu ([157.181.151.130]) by mx2.mail.elte.hu with esmtp (Exim) id 1T7VKY-0000xI-11 from ; Fri, 31 Aug 2012 19:48:32 +0200 Received: (Authenticated sender: pgj) by login01.caesar.elte.hu with local (Exim 4.72) (envelope-from ) id 1T7VKX-00047e-H3; Fri, 31 Aug 2012 19:48:29 +0200 Date: Fri, 31 Aug 2012 19:48:29 +0200 From: PALI Gabor Janos To: Anil Madhavapeddy , "Robert N. M. Watson" Subject: Sending Network Packets with Mirage/kFreeBSD Message-ID: <20120831174828.GA12697@caesar.elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Pali Gabor Janos X-ELTE-SpamScore: -2.0 X-ELTE-SpamLevel: X-ELTE-SpamCheck: no X-ELTE-SpamVersion: ELTE 2.0 X-ELTE-SpamCheck-Details: score=-2.0 required=5.9 tests=BAYES_00 autolearn=no SpamAssassin version=3.3.1 -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: cl-mirage@lists.cam.ac.uk X-BeenThere: cl-mirage@lists.cam.ac.uk X-Mailman-Version: 2.1.8 Precedence: list List-Id: MirageOS development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 17:48:38 -0000 X-Keywords: X-UID: 524 Status: O Content-Length: 1327 Lines: 28 Hello there, For your information, I have just updated the Mirage/kFreeBSD repository at GitHub that now implements support for sending packets: https://github.com/pgj/mirage-kfreebsd/commit/a51477bb5e1c60fd34efbf211d153a46093ec4ba The current binding technically maps Mirage IO pages or list of IO pages to FreeBSD mbuf(9) buffers while it avoids copying their contents. As a side-product, it introduces reference counting on the mbuf(9) values, so they are shared between the NIC driver sending the packets and the OCaml run-time system. The trick is to create EXTREF type mbuf(9)s so I can refer to external memory locations via m_data and help the reference counting via m_ext.ext_free. I also added two simple tests, like an ARP probe and sending ICMP echo requests to check the functionality. The code for these come from mirage-net package that I also added in this commit (with some minor changes, but it builds): https://github.com/pgj/mirage-kfreebsd/commit/0ff5b2cd7ab0975e3f2ee1bd89f8e5dbf028b102 They seem to be working, however I am not quite happy the current memory management, as each time I unload the modules, I see many leaked allocations. I checked the code and the IO pages are released by the GC, but something else is still stuck in the memory. Do you have any ideas how to track down these?