Re: [MirageOS-devel] Building mirage-www fails

[Nik Sultana <ns441@xxxxxxxxx>]

Thanks Anil, I could build the Ubuntu 14.04 (Trusty) container following 
your instructions, with one modification: referencing "avsm/docker-opam" 
instead of "avsm/docker".
I haven't tested the other containers since I don't have ready access to 
a CentOS box. This patch will work if the other distributions use a 
version of sh that's compatible to that used in Trusty.
I tested the problem with the semantics of && under -e by replacing the 
line
  make configure && make depend && make build
with
  false && make configure && make depend && make build
and verified that I still got "MIRAGE WORKS" printed at the end after 
running
  ./build.sh ubuntu-trusty-4.01.0 mirage-skeleton

Then I unfolded
  false && make configure && make depend && make build
into
  false
  make configure
  make depend
  make build
and reran, and verified that I did NOT get "MIRAGE WORKS" printed at the 
end.
Finally I removed "false", and reran.
I did get "MIRAGE WORKS" printed at the end.

At the risk of asking an over-asked question: what's the procedure for 
the PR, do I need to quote an issue number etc, or can I just go ahead 
and create a PR from a fork?
Nik

On 2014-10-22 14:03, Anil Madhavapeddy wrote:
> CCing the devel list so everyone knows how this works.
>
> If you have a Linux box with Docker installed, you can clone the
> precise environment by:
>
> $ sudo docker pull avsm/docker
>
> This gives you access to a number of 'tags' (which are predefined
> builds of OPAM/OCaml on various filesystems). Full list of tags are:
>
> https://registry.hub.docker.com/u/avsm/docker-opam/tags/manage/
>
> You can use a tag directly via:
>
> $ sudo docker build -t avsm/docker:ubuntu-trusty-4.01.0 <cmd>
>
> The scripts referenced below already do this, and are all in:
> https://github.com/mirage/is-mirage-broken
>
> The cron.sh is the one that's run regularly and sync with the
> Docker registry and builds every combination.
>
> To recreate one of them, just run build.sh directly; e.g.
>
> ./build.sh ubuntu-trusty-4.01.0 mirage-skeleton
>
> If you already have a suitable environment with OPAM and OCaml
> installed system-wide, you can just run the shell scripts in
> is-mirage-broken/scripts/ directly to fix them.
>
> -anil
>
> On 22 Oct 2014, at 13:55, Nik Sultana <ns441@xxxxxxxxx> wrote:
>
> > Hi Anil, is this something I can try locally to see if the modified 
> > script works, or do I need to run the script on a build server 
> > somewhere? Feel free to point me to any "BUILD" notes i could use for 
> > this, in case I can test it and make a PR, if it's still an open 
> > issue.
> > Best,
> > Nik
> >
> > On 2014-10-21 16:18, Richard Mortier wrote:
> > > if you read the docs i quoted, that's exactly as documented:
> > > > > > > > 	â The -e setting shall be ignored when executing the compound 
> > > > > > > > list
> > > > > > > > following the while, until, if, or elif reserved word, a 
> > > > > > > > pipeline
> > > > > > > > beginning with the ! reserved word, or any command of an AND-OR 
> > > > > > > > list
> > > > > > > > other than the last.
> > > it might be rather counter-intuitive and of dubious merit, but it's
> > > the documented behaviour...
> > > On 21 Oct 2014, at 16:01, Nik Sultana <ns441@xxxxxxxxx> wrote:
> > > > I think Richard might be right. The semantics of && seems to be a 
> > > > bit
> > > > counterintuitive in this case. Consider this script:
> > > > #!/bin/sh
> > > > set -e #line1
> > > > false && echo hello #line2
> > > > echo hello2
> > > > false #line4
> > > > echo world
> > > > It gives the output:
> > > > hello2
> > > > which is odd, because we expect the script to be terminated at line2 
> > > > --
> > > > the line "echo hello2" should never be reached. Instead, the script
> > > > continues on and is only terminated at line4.
> > > > Just to test the effect of "set -e", commenting line1 and running 
> > > > the
> > > > script gives the output:
> > > > hello2
> > > > world
> > > > On 2014-10-21 15:49, Anil Madhavapeddy wrote:
> > > > > The && should catch that failure.  Consider:
> > > > > #!/bin/sh -ex
> > > > > exit 1 && echo hello
> > > > > echo world
> > > > > Executing that:
> > > > > $ ./test.sh
> > > > > + exit 1
> > > > > If the make depend fails, I'd expect the && to fail, and the 
> > > > > resulting
> > > > > non-zero exit code to cause the whole shell to exit.
> > > > > -anil
> > > > > On 21 Oct 2014, at 15:37, Richard Mortier
> > > > > <Richard.Mortier@xxxxxxxxxxxxxxxx> wrote:
> > > > > > but isn't it the "make depend" step that's failing as far as the
> > > > > > is-mirage-broken/scripts/mirage-www script is concerned -- hence 
> > > > > > it
> > > > > > won't exit according to point 2. below (-e is ignored if it's any
> > > > > > command of an AND-OR list other than the last).
> > > > > > with -e on, seems expected practice is to issue the commands
> > > > > > individually i.e.
> > > > > > make depend
> > > > > > make configure
> > > > > > make build
> > > > > > ...and let the -e fail them if required. (assuming you don't want 
> > > > > > to
> > > > > > accumulate info about failures of depend/configure on other 
> > > > > > platforms
> > > > > > before bailing out.)
> > > > > > On 21 Oct 2014, at 15:18, Anil Madhavapeddy <anil@xxxxxxxxxx> 
> > > > > > wrote:
> > > > > > > It's here:
> > > > > > > https://github.com/mirage/is-mirage-broken/blob/master/scripts/mirage-www
> > > > > > > And the `make` invocations do seem to exit with a non-zero exit 
> > > > > > > code.
> > > > > > > -ani
> > > > > > > On 21 Oct 2014, at 15:02, Richard Mortier
> > > > > > > <Richard.Mortier@xxxxxxxxxxxxxxxx> wrote:
> > > > > > > > not sure precisely where the script being executed actually is 
> > > > > > > > but
> > > > > > > > could it be because it's a command in a multi-command pipeline?
> > > > > > > > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
> > > > > > > > -e
> > > > > > > > When this option is on, when any command fails (for any of the
> > > > > > > > reasons listed in Consequences of Shell Errors or by returning 
> > > > > > > > an
> > > > > > > > exit status greater than zero), the shell immediately shall exit
> > > > > > > > with the following exceptions:
> > > > > > > > 	â The failure of any individual command in a multi-command 
> > > > > > > > pipeline
> > > > > > > > shall not cause the shell to exit. Only the failure of the 
> > > > > > > > pipeline
> > > > > > > > itself shall be considered.
> > > > > > > > 	â The -e setting shall be ignored when executing the compound 
> > > > > > > > list
> > > > > > > > following the while, until, if, or elif reserved word, a 
> > > > > > > > pipeline
> > > > > > > > beginning with the ! reserved word, or any command of an AND-OR 
> > > > > > > > list
> > > > > > > > other than the last.
> > > > > > > > 	â If the exit status of a compound command other than a 
> > > > > > > > subshell
> > > > > > > > command was the result of a failure while -e was being ignored, 
> > > > > > > > then
> > > > > > > > -e shall not apply to this command.
> > > > > > > > This requirement applies to the shell environment and each 
> > > > > > > > subshell
> > > > > > > > environment separately. For example, in:
> > > > > > > > set -e; (false; echo one) | cat; echo two
> > > > > > > > the false command causes the subshell to exit without executing 
> > > > > > > > echo
> > > > > > > > one; however, echo two is executed because the exit status of 
> > > > > > > > the
> > > > > > > > pipeline (false; echo one) | cat is zero.
> > > > > > > > On 21 Oct 2014, at 14:31, Anil Madhavapeddy <anil@xxxxxxxxxx> 
> > > > > > > > wrote:
> > > > > > > > > Great!  For the rest of the list, it looks like mirage-dev is
> > > > > > > > > working with the external OPAM solver (aspcud, on 
> > > > > > > > > Ubuntu-trusty),
> > > > > > > > > but not with the internal one (CentOS builds):
> > > > > > > > > https://github.com/mirage/is-mirage-broken/tree/master/logs
> > > > > > > > > This is probably due to our excessive use of depopts; I will
> > > > > > > > > replace those with explicit virtual packages (e.g. dns-mirage) 
> > > > > > > > > that
> > > > > > > > > encode all the dependencies they need.
> > > > > > > > > More mysteriously, the `set -e` in the shell script isn't 
> > > > > > > > > causing
> > > > > > > > > it to terminate early, so the `MIRAGE WORKS` string is always
> > > > > > > > > printed.  We need a is-mirage-working-working script to fix our
> > > > > > > > > is-mirage-working script :-)
> > > > > > > > > -anil
> > > > > > > > > On 20 Oct 2014, at 15:09, Luke Dunstan 
> > > > > > > > > <lukedunstan81@xxxxxxxxx>
> > > > > > > > > wrote:
> > > > > > > > > > Thanks, I was able to compile and run mirage-www on a Docker 
> > > > > > > > > > image
> > > > > > > > > > with 4.01.
> > > > > > > > > > Luke
> > > > > > > > > > On 20 October 2014 09:28, Anil Madhavapeddy <anil@xxxxxxxxxx>
> > > > > > > > > > wrote:
> > > > > > > > > > On 19 Oct 2014, at 03:17, Luke Dunstan 
> > > > > > > > > > <lukedunstan81@xxxxxxxxx>
> > > > > > > > > > wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > I tried to build mirage-www as per
> > > > > > > > > > > http://openmirage.org/wiki/mirage-www
> > > > > > > > > > > $ env NET=socket FS=crunch mirage configure --unix
> > > > > > > > > > > (success)
> > > > > > > > > > > $ make depend
> > > > > > > > > > > (success*)
> > > > > > > > > > > $ make
> > > > > > > > > > > camlbuild -classic-display -use-ocamlfind -pkgs
> > > > > > > > > > > lwt.syntax,conduit.mirage,cow.syntax,cowabloga,cstruct,io-page,io-page.unix,lwt,mirage-console.unix,mirage-fs-unix,mirage-http,mirage-types,mirage-types.lwt,tcpip.stack-socket
> > > > > > > > > > > -tags 
> > > > > > > > > > > "syntax(camlp4o),annot,bin_annot,strict_sequence,principal"
> > > > > > > > > > > -cflag -g -lflags -g,-linkpkg main.native
> > > > > > > > > > > ocamlfind ocamlc -c -g -annot -bin-annot -principal
> > > > > > > > > > > -strict-sequence -package tcpip.stack-socket -package
> > > > > > > > > > > mirage-types.lwt -package mirage-types -package mirage-http
> > > > > > > > > > > -package mirage-fs-unix -package mirage-console.unix -package 
> > > > > > > > > > > lwt
> > > > > > > > > > > -package io-page.unix -package io-page -package cstruct 
> > > > > > > > > > > -package
> > > > > > > > > > > cowabloga -package cow.syntax -package conduit.mirage 
> > > > > > > > > > > -package
> > > > > > > > > > > lwt.syntax -syntax camlp4o -o main.cmo main.ml
> > > > > > > > > > > + ocamlfind ocamlc -c -g -annot -bin-annot -principal
> > > > > > > > > > > -strict-sequence -package tcpip.stack-socket -package
> > > > > > > > > > > mirage-types.lwt -package mirage-types -package mirage-http
> > > > > > > > > > > -package mirage-fs-unix -package mirage-console.unix -package 
> > > > > > > > > > > lwt
> > > > > > > > > > > -package io-page.unix -package io-page -package cstruct 
> > > > > > > > > > > -package
> > > > > > > > > > > cowabloga -package cow.syntax -package conduit.mirage 
> > > > > > > > > > > -package
> > > > > > > > > > > lwt.syntax -syntax camlp4o -o main.cmo main.ml
> > > > > > > > > > > File "main.ml", line 98, characters 2-13:
> > > > > > > > > > > Error: Unbound module OS
> > > > > > > > > > > Command exited with code 2.
> > > > > > > > > > > make: *** [main.native] Error 10
> > > > > > > > > > > To try to ensure that this was reproducible I also tried 
> > > > > > > > > > > starting
> > > > > > > > > > > from the "avsm/docker-opam:ubuntu-trusty-4.01.0" Docker image 
> > > > > > > > > > > and
> > > > > > > > > > > got the same error. However, I had to change the ounit 
> > > > > > > > > > > package
> > > > > > > > > > > URL because forge.ocamlcore.org is apparently down.
> > > > > > > > > > > Also, I had to "apt-get install libssl-dev" to get past the
> > > > > > > > > > > "configure" / "make depend" step because it was apparently
> > > > > > > > > > > required to compile ssl-0.4.7, so maybe that is related to 
> > > > > > > > > > > the
> > > > > > > > > > > problem?
> > > > > > > > > > Hi Luke,
> > > > > > > > > > Thanks for the bug report, and bonus points for trying the 
> > > > > > > > > > Docker
> > > > > > > > > > repository out.  I've fixed the underlying issue that was 
> > > > > > > > > > causing
> > > > > > > > > > the unbound OS module to show up -- it's because of some 
> > > > > > > > > > recent
> > > > > > > > > > shuffling of dependencies in console handling, and this 
> > > > > > > > > > resulted
> > > > > > > > > > in mirage-unix no longer being implicitly depended on.  I've
> > > > > > > > > > modified the mirage tool to explicitly add the dependency in.
> > > > > > > > > > In order to help keep things working across the increasing 
> > > > > > > > > > number
> > > > > > > > > > of distributions, I've also created a couple of cron jobs that
> > > > > > > > > > automatically build things and publish the logs
> > > > > > > > > > - The Docker registry now has OPAM tags for Ubuntu/CentOS on 
> > > > > > > > > > OCaml
> > > > > > > > > > 4.01 and 4.02:
> > > > > > > > > > https://registry.hub.docker.com/u/avsm/docker-opam/builds_history/15669/
> > > > > > > > > > https://github.com/avsm/docker-opam has the sources
> > > > > > > > > > - https://github.com/mirage/is-mirage-broken runs as a cronjob
> > > > > > > > > > (cron.sh) on blobs.openmirage.org daily and pushes the logs to 
> > > > > > > > > > the
> > > > > > > > > > logs/ directory in this repo.  When that's completed running 
> > > > > > > > > > (and
> > > > > > > > > > we've fixed the build breakages!), it should be easy to get it 
> > > > > > > > > > on
> > > > > > > > > > the website as a badge so we can see the status of repos at a
> > > > > > > > > > glance.
> > > > > > > > > > About to hop on a flight now, so I've left the cron job 
> > > > > > > > > > running
> > > > > > > > > > and will check it out tomorrow.  If anyone else can test out 
> > > > > > > > > > the
> > > > > > > > > > Docker images in the meanwhile, feel free...
> > > > > > > > > > -anil
> > > > > > > > > > _______________________________________________
> > > > > > > > > > MirageOS-devel mailing list
> > > > > > > > > > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> > > > > > > > > > http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> > > > > > > > > _______________________________________________
> > > > > > > > > MirageOS-devel mailing list
> > > > > > > > > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> > > > > > > > > http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> > > > > > > > --
> > > > > > > > Cheers,
> > > > > > > > R.
> > > > > > > > _______________________________________________
> > > > > > > > MirageOS-devel mailing list
> > > > > > > > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> > > > > > > > http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> > > > > > --
> > > > > > Cheers,
> > > > > > R.
> > > > > > _______________________________________________
> > > > > > MirageOS-devel mailing list
> > > > > > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> > > > > > http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
> > > > > _______________________________________________
> > > > > MirageOS-devel mailing list
> > > > > MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> > > > > http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel