Re: [MirageOS-devel] Patches for building mirage libraries with `opam2nix`

[Tim Cuthbertson <tim@xxxxxxxxxxx>]

On Tue, Sep 1, 2015 at 6:47 PM, Anil Madhavapeddy <anil@xxxxxxxxxx> wrote:
> On 1 Sep 2015, at 08:27, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>>
>> Hi all,
>>
>> A little while ago I managed to get mirage's xen libraries and
>> dependencies building in nix [0]. This worked, but I didn't fancy
>> keeping those packages up to date manually. So I also started working
>> on `opam2nix`[1], a tool which generates nix expressions from an opam
>> repository. It's still work in progress, but has managed to build most
>> things I've thrown at it so far.
>>
>> The main problem is when build scripts assume that they'll be
>> installed into `opam config var prefix`/lib/<library>, or that
>> pkgconfig specs will be found in `opam config var
>> prefix`/lib/pkgconfig. In the case of `opam2nix`, dependencies can be
>> found via `ocamlfind query`, while the destination will be a
>> completely different path which is populated in $PREFIX,
>> $OCAMLFIND_DESTDIR and opam variables (like %{lib}% etc.)
>>
>> I've made a number of changes to make these build scripts more lenient
>> - e.g. rather than always overriding $PKG_CONFIG_PATH, they'll only do
>> that if all needed dependencies aren't already present. And also
>> respecting $PREFIX if it's set (but falling back to `opam config var
>> prefix` if not):
>>
>> --------
>>
>> gmp-xen:
>> https://github.com/gfxmonk/opam-repository/commits/mirage-fixes
>>
>> io-page:
>> https://github.com/gfxmonk/io-page/commits/master
>>
>> mirage-platform:
>> https://github.com/gfxmonk/mirage-platform/commits/master
>> (note: the second of those two commits
>> [6121921ef6e666f021a61f3570840108927f90d8] removes something that I
>> _think_ was unnecessary, but it's worth verifying that with someone
>> who knows more about how the libraries depend on each other)
>>
>> tcpip:
>> https://github.com/gfxmonk/mirage-tcpip/commits/master
>>
>> zarith-xen:
>> I had to rewrite the zarith-xen build script, because I couldn't come
>> up with a simple enough change which would support both the `opam` and
>> `opam2nix` cases. So this is less of a patch than a note that it
>> doesn't currently work, and maybe someone will have better ideas about
>> how to make it work:
>> https://github.com/gfxmonk/opam2nix/blob/71887935dc30a744aa26f1f5af75678355d2d74e/src/nix/overrides/zarith-xen/install.sh
>>
>> --------
>>
>> It'd be great if the maintainers of these libraries are happy to merge
>> my changes, or otherwise discuss other ways of making these libraries
>> build in an I-can't-believe-it's-not-opam environment ;). I thought
>> I'd discuss it here rather than creating individual pull requests,
>> because this should provide a lot more context.
>
> We should absolutely support a non-OPAM build environment.  That's
> purely there because of "build laziness", but will also be a problem
> when it comes to doing upstream packaging into Debian/*BSD/etc.
>
> I haven't reviewed your changes above in detail as I'm about to fly
> to Vancouver to ICFP, but please do send pull requests and we can
> discuss them on the relevant GitHub repos.

Thanks, done :)

> Do you think it's worth putting a Nix repository build for Mirage
> into the cron-based builds so that we can spot when it breaks against
> trunk?

I don't know. It could be a lot of wasted work since (I think) it
should only really be breakable by buildscript changes, which I assume
are pretty rare compared to code changes.

Right now opam2nix is just something I'm working on, but I'm hoping to
roll it into nix proper one day - at which point I believe
http://hydra.nixos.org/ may take care of detecting broken releases.
That's probably good enough for a while, but certainly if mirage devs
(who themselves might prefer working on trunk) want to make sure the
nix expressions are always valid, we could look at setting up some
trunk integration.

>> The main downside is that these changes make debugging a little
>> trickier, since the build scripts are slightly more complex. Also, it
>> might mean that a dependency could be picked up from a system package
>> rather than from `opam`.
>>
>> It may also be a better idea in general to just feed more information
>> to the scripts directly from the `opam` file (via %{lib}% and friends)
>> rather than have them assume too much about how they'll be installed.
>> I didn't want to go down this route because it makes for more drastic
>> changes, but I'm happy to try it if you're game.
>
> There is an `opam-query` plugin which makes it easier to invoke from
> shell scripts or Makefiles, so that it can be used as a default option
> if an override isn't provided.  That might help...

Possibly, although the main knowledge required is "where is <x>
dependency installed", and "where should _I_ install myself". I don't
think `opam-query` will be able to resolve those kinds of questions,
and it's probably cleaner to inject this information into command line
arguments anyway, rather than the build script asking for it. opam2nix
can act just like opam already for most interpolated variables
(%{lib}% and friends), thanks to opam-lib - all that's needed is for
the build script to accept this information on the command line, or in
an env var.

Cheers,
 - Tim.

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel