[MirageOS-devel] Cohttp, and the SSL saga update

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

I've been merging patches from Clark Gaebel to merge SSL support into the 
Cohttp backend, and it uncovered a bunch of latent (not Clark's fault!) issues 
that led to cascading commits from me, so I thought I'd send a wider mail about 
it.  In Cohttp.0.10.0, Async is an optional dependency, with Cohttp_async 
installed if it's present.  We therefore tell RWO readers to:

$ opam install async cohttp

Now, with the SSL patches in Cohttp-dev, we now need:

$ opam install async async_ssl cohttp

(or more simply, just "async_ssl cohttp" due to the implicit dependency).

The twist is for users upgrading OPAM who won't have (the new) async_ssl 
installed.  For them, the optional dependency on (async_ssl & async) will no 
longer be satisfied, and the entire Cohttp_async will magically vanish after an 
upgrade.

This is pretty non-intuitive behaviour and will result in a lot of confusion, 
so I've broken up Cohttp to separate the connection handling for Async and Lwt 
into a separate Conduit library (https://github.com/mirage/ocaml-conduit).  The 
Conduit library uses build-time macros to detect if Async-SSL and Lwt-SSL are 
available, and slots in dynamic failures if they aren't present.  It can also 
be used by other network libraries to deal with SSL and non-SSL connection 
setup more easily.

A few bugs uncovered while doing this:

- Async SSL uses Ctypes via the libffi mode.  This uncovers a latent bug in 
ocamlfind which calculates link order in the wrong direction, and so linking 
fails if Async_ssl is used as a library.  We need to fix this in ocamlfind, but 
we can move forward by releasing the Ctypes C stub generation and using that in 
Async SSL instead.  Jeremy Yallop has done so in his fork of Async_ssl 
(https://github.com/yallop/async_ssl in the stub-generation branch) and this 
works great.

- Conduit/Lwt/SSL uncovers a bug in the C bindings, which I backported here and 
waiting an ack: https://github.com/ocaml/opam-repository/pull/1896
It's been a couple of years since the last OCaml-SSL C bindings release, so 
shifting towards a Ctypes binding here would be strongly preferable (David 
Sheets is working on these).

- I want the Conduit library to hide away the details of which SSL library 
we're using, and permit (for example) the OCaml TLS library that Hannes and 
David are working on to be used as an alternative. 

- The SSL interface currently exposed by both Async-SSL and Lwt-SSL seem to be 
on the surface only, and don't give easy access to certificate validation 
functions.  This is a pretty serious omission for Internet-facing use, so 
expect some library changes in Conduit to expose these in a more complete way.

- I couldn't figure out how to use pa_optcomp with Oasis successfully.  Every 
option seemed to break some other build aspect, and I got a bit sick of it and 
wrote a shell script for Conduit builds instead.  I'll probably migrate it to 
Daniel Buenzli's scripts after the library settles down.

The other new thing in Cohttp that made this release take longer is that Rudi 
Grinberg also fixed the dependency on Re_str, which it turns out is not POSIX 
thread safe, and so breaks XAPI (but not Mirage, since we don't use preemptive 
threads).  This caused some upstream packages to fail to build since they 
forgot to explicitly depend on Re_str, and so can't find it once the Cohttp 
Re_str dependency broke.  All this is fixed in a series of OPAM pulls at:

https://github.com/ocaml/opam-repository/pull/1896

I'm going to do a few things to try to prevent this sort of tangle in the 
future and ensure Cohttp stays working in the trunk:

- Move it from avsm/ocaml-cohttp to mirage/ocaml-cohttp.  This was blocked on 
syncing the Releases metadata, and I added two new executables to ocaml-github 
to do just this in v0.8.3.  `git-sync-github-releases` and `git-upload-release` 
provide the facility to copy Releases metadata between repos conveniently via 
the command line.  I will repo transfer avsm/ocaml-cohttp to 
mirage/ocaml-cohttp tomorrow.

- I'll give Rudi Grinberg merge access to the new master.  He and David Sheets 
(who already has merge access) have been providing higher quality patches than 
me for a while :-)

- Remove the autobuild files from the repository and use `git-upload-release` 
to only include them for release tarballs.  This will help stop multiple pulls 
from conflicting all the time.  I'm also investigating fast build solutions, as 
this is starting to be a limiting factor.

I'll merge the OPAM packages tomorrow unless someone finds anything critical!

cheers,
Anil
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel