Re: [MirageOS-devel] Compiling C components for mirage-xen

[Tim Cuthbertson <tim@xxxxxxxxxxx>]

On Mon, Jan 11, 2016 at 9:48 PM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
> On 11 January 2016 at 09:06, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>> On Mon, Jan 11, 2016 at 4:52 AM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
>>> On 10 January 2016 at 10:15, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>>>> On Fri, Jan 8, 2016 at 11:14 PM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
>>>>> On 8 January 2016 at 11:19, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>>>>>> I'd like to use `Bcrypt` from the `safepass` opam module in passe[0].
>>>>>> There is sadly no `safepass-xen` package, and trying to just use
>>>>>> `safepass` results in link errors:
>>>>>>
>>>>>>
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>>> `BF_crypt':
>>>>>> (.text+0x1b97): undefined reference to `__errno_location'
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>>> `BF_crypt':
>>>>>> (.text+0x2112): undefined reference to `__errno_location'
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>>> `_crypt_blowfish_rn':
>>>>>> (.text+0x21ea): undefined reference to `__errno_location'
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>>> `_crypt_gensalt_blowfish_rn':
>>>>>> (.text+0x2575): undefined reference to `__errno_location'
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>>> `_crypt_gensalt_blowfish_rn':
>>>>>> (.text+0x25a6): undefined reference to `__errno_location'
>>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o):(.text+0x25b7):
>>>>>> more undefined references to `__errno_location' follow
>>>>>>
>>>>>>
>>>>>> I'm assuming I'd need to customise the build process tp produce an
>>>>>> appropriate C library as with `gmp-xen`, `zarith-xen` etc. The trouble
>>>>>> is twofold:
>>>>>>
>>>>>>  - I don't really know what the thing I'm doing is. Is mirage-xen a
>>>>>> different architecture? A different ABI? Do C libraries need to be
>>>>>> cross-compiled, or just compiled with particular flags so that they'll
>>>>>> work on xen-minios instead of gnu/linux?
>>>>>
>>>>> C libraries need to be compiled with different headers and flags (in
>>>>> particular, -mno-red-zone). Using libraries compiled for Linux will
>>>>> often appear to work, but may fail when interrupts occur, so the build
>>>>> makes it hard to use them by accident.
>>>>>
>>>>> However, OCaml binaries can be shared (ocamlopt doesn't use a red zone
>>>>> and the OCaml ABI is the same).
>>>>>
>>>>>>  - The existing scripts seem to be pretty ad-hoc - zarith gets away
>>>>>> mainly with particular CFLAGS
>>>>>> (https://github.com/ocaml/opam-repository/blob/master/packages/zarith-xen/zarith-xen.1.4/files/mirage-install.sh),
>>>>>
>>>>> It's using pkg-config to get the important flags. The others are
>>>>> optimisations (except -fno-builtin, which is either not needed or
>>>>> should be supplied by Mini-OS).
>>>>>
>>>>>> while gmp looks like a full cross-compilation using `--target` (and
>>>>>> then does some other sneaky things).
>>>>>>
>>>>>> safepass uses OASIS for its build system, which doesn't seem to have
>>>>>> any cross-compilation functionality built in as far as I could tell.
>>>>>> Can anyone give me pointers on what I'd need to do to safepass' build
>>>>>> scripts to get this working in mirage-xen, or at the least some
>>>>>> clarification on why it doesn't work as-is and how cross compilation
>>>>>> works for C libraries used in mirage-xen?
>>>>>>
>>>>>> [0]: https://github.com/timbertson/passe
>>>>>
>>>>> The io-page library provides a fairly simple template:
>>>>>
>>>>> https://github.com/mirage/io-page/blob/master/_oasis
>>>>>
>>>>> It builds io_page_unix and io_page_xen libraries. The C flags are
>>>>> added to the main META file with:
>>>>>
>>>>> XMETAExtraLines: xen_linkopts = "-lio_page_xen_stubs"
>>>>>
>>>>> There's a command to get the Xen flags:
>>>>>
>>>>> PostConfCommand: ocaml postconf.ml
>>>>>
>>>>> The Xen source file (stub_alloc_pages_xen.c) is just a symlink to the
>>>>> Unix one (you need a separate file because OASIS can't compile the
>>>>> same source path with different flags).
>>>>
>>>> Thanks, Thomas. - that helps quite a lot
>>>>
>>>> I think I've followed the same pattern now, but still haven't been
>>>> able to get safepass building.
>>>>
>>>> I'm using the following for CCOpt:
>>>>
>>>> `pkg-config --cflags --static mirage-xen-ocaml`
>>>>
>>>> Now the linking phase fails with:
>>>>
>>>> + ocamlfind ocamlmklib -o src/safepass_stubs src/crypt_blowfish.o
>>>> src/bcrypt_stub.o
>>>> ld: errno: TLS definition in /nix/store/<...>-glibc-2.21/lib/libc.so.6
>>>> section .tbss mismatches non-TLS reference in src/crypt_blowfish.o
>>>> /nix/store/<...>-glibc-2.21/lib/libc.so.6: could not read symbols: Bad 
>>>> value
>>>> collect2: error: ld returned 1 exit status
>>>> Command exited with code 2.
>>>> E: Failure("Command ''ocamlbuild' src/libsafepass_stubs.a
>>>> src/dllsafepass_stubs.so src/safepass.cma src/safepass.cmxa
>>>> src/safepass.a src/safepass.cmxs -tag debug' terminated with error
>>>> code 10")
>>>
>>> Try running it as "ocamlfind ocamlmklib -verbose" to see what commands
>>> it's actually using. For io-page, I get:
>>>
>>> $ ocamlfind ocamlmklib -verbose -o lib/io_page_xen_stubs
>>> lib/stub_alloc_pages_xen.o
>>> Effective set of compiler predicates: autolink,byte,native
>>> + ocamlmklib -ocamlc ocamlc.opt -ocamlopt ocamlopt.opt -verbose -o
>>> lib/io_page_xen_stubs lib/stub_alloc_pages_xen.o
>>> + gcc -shared  -o lib/dllio_page_xen_stubs.so lib/stub_alloc_pages_xen.o
>>> + ar rc lib/libio_page_xen_stubs.a  lib/stub_alloc_pages_xen.o; ranlib
>>> lib/libio_page_xen_stubs.a
>>>
>>> So it doesn't seem to be using ld at all, just ar.
>>
>> Tracing it with strace, that `gcc -shared` process ends up invoking
>> `ld` internally.
>>
>> I tried running ocamlmklib with the `-failsafe` option, and it does
>> continue past this and produce the static .a library using `ar`. But
>> if I try and put that flag into the larger build it doesn't get much
>> further, because it hits the same issue when creating
>> src/safepass.cmxs:
>>
>> + ocamlfind ocamlopt -shared -I src src/safepass.cmxa src/bcrypt.cmx
>> -o src/safepass.cmxs
>> ld: errno: TLS definition in libc.so.6 section .tbss mismatches
>> non-TLS reference in src/libsafepass_stubs.a(crypt_blowfish.o)
>> libc.so.6: could not read symbols: Bad value
>> collect2: error: ld returned 1 exit status
>>
>>>> This seems to be something special about errno being thread-local in
>>>> glibc. Googling turns up a lot of advice on fixing this problem, but
>>>> all of it assumes you're just building on linux for linux.
>>>> bcrypt_stubs.c does <errno.h>, which I believe is being provided by
>>>> minios-xen-0.8/include/minios-xen/errno.h.
>>>>
>>>> However, at link time I don't know where errno is supposed to come
>>>> from. It seems to be coming from glibc, but shouldn't it be coming
>>>> from minios instead given that's where the errno header comes from? I
>>>> poked around with `nm` in minios' built files but couldn't find
>>>> anything that actually defined errno.
>>>
>>> It's from mirage-xen:
>>>
>>> https://github.com/mirage/mirage-platform/blob/c035439de703edbfdd096044afdbab85f4cbf61f/bindings/main.c#L30
>>
>> Which compiled libraries does that end up in though?
>>
>> I tried `nm`-ing all the built files from mirage-{xen,xen-posix,xen-ocaml} 
>> with:
>>
>> $ find /nix/store/inxv9vdvnz8cjs6236fmalj0mnaq3sgx-mirage-xen-ocaml-2.3.4
>> /nix/store/vbz4pivjk196dw5h8rmg68qxh5m44cn9-mirage-xen-posix-2.3.4
>> /nix/store/24lm5990pxzp71nsgnffcmb2i7n41fck-mirage-xen-2.3.3 -type f
>> -exec bash -c 'if nm {} 2>/dev/null | grep -q errno; then echo {}; nm
>> {} | grep errno; fi' \;
>> /nix/store/inxv9vdvnz8cjs6236fmalj0mnaq3sgx-mirage-xen-ocaml-2.3.4/lib/mirage-xen-ocaml/libxenasmrun.a
>>                  U errno
>>                  U errno
>>                  U errno
>>                  U errno
>> /nix/store/vbz4pivjk196dw5h8rmg68qxh5m44cn9-mirage-xen-posix-2.3.4/lib/mirage-xen-posix/libxenposix.a
>>                  U errno
>> 0000000000000e4a T __h_errno_location
>> /nix/store/24lm5990pxzp71nsgnffcmb2i7n41fck-mirage-xen-2.3.3/lib/mirage-xen/libxencamlbindings.a
>> 0000000000000004 C errno
>
> This is it (C = Common).
>
>> So there are only .a files (which ld ignores when creating a shared
>> library, doesn't it?), and only libxencamlbindings actually defines it
>> (the others just reference it), which doesn't seem like a library I'd
>> necessarily need to link against.
>>
>> I don't know if mirage-xen actually makes use of .so libraries, as
>> I've only seen it produce static executables. But even if it doesn't
>> need them, I'd need to somehow tell oasis not to build shared versions
>> of safepass (both the .so and the .cmxs).
>
> Correct: Mirage on Xen does not use shared libraries. I don't know how
> to prevent them from being built. The easiest option might be to
> remove the uses of errno from the library (it doesn't look like the
> OCaml bindings do anything with it anyway).

The ocaml bindings (bcrypt_stub.c) doesn't, but the C code compiled
with it (crypt_blowfish.c) does, so I can't just remove it sadly.

So this is a bit ridiculous, but I did manage to get it compiling with
the following hack in the build script:

  mkdir fake_libc
  echo '"fake_libc": -traverse' >> _tags
  touch fake_libc/libc.c
  gcc -shared -nostdlib -o fake_libc/libc.so fake_libc/libc.c

(and also added -L$PWD/fake_libc to oasis' CCLib field)

I initially put just `int errno;` in that file, but it still works
without it. Seems like all it needs is for something to override glibc
on the library search path which doesn't contain any conflicting
definitions. It would probably work if I could just specify -nostdlib
too, but that's not so easy with .cmxa files.

I haven't actually been able to check whether the compiled xen
executable runs without issue yet, but both safepass and passe (which
uses it) now compile in xen mode, so that's a good sign!

I wonder if it would be worth providing some sort of stub libc.so with
mirage for this purpose? It seems like a lot of build tools expect to
be able to build a .so, and it may be easier to let them do their
thing with something trivial to link against rather than trying to
force them to build only static libraries.

Thanks for your help Thomas, and let me know if you have any better
ideas - I'll also get in touch with the safepass author now that I
have something working, and see if they have other ideas too.

Cheers,
 - Tim.

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel