Re: [MirageOS-devel] Compiling C components for mirage-xen

[Thomas Leonard <talex5@xxxxxxxxx>]

On 11 January 2016 at 09:06, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
> On Mon, Jan 11, 2016 at 4:52 AM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
>> On 10 January 2016 at 10:15, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>>> On Fri, Jan 8, 2016 at 11:14 PM, Thomas Leonard <talex5@xxxxxxxxx> wrote:
>>>> On 8 January 2016 at 11:19, Tim Cuthbertson <tim@xxxxxxxxxxx> wrote:
>>>>> I'd like to use `Bcrypt` from the `safepass` opam module in passe[0].
>>>>> There is sadly no `safepass-xen` package, and trying to just use
>>>>> `safepass` results in link errors:
>>>>>
>>>>>
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>> `BF_crypt':
>>>>> (.text+0x1b97): undefined reference to `__errno_location'
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>> `BF_crypt':
>>>>> (.text+0x2112): undefined reference to `__errno_location'
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>> `_crypt_blowfish_rn':
>>>>> (.text+0x21ea): undefined reference to `__errno_location'
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>> `_crypt_gensalt_blowfish_rn':
>>>>> (.text+0x2575): undefined reference to `__errno_location'
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o): In function
>>>>> `_crypt_gensalt_blowfish_rn':
>>>>> (.text+0x25a6): undefined reference to `__errno_location'
>>>>> <...>/lib/safepass/libsafepass_stubs.a(crypt_blowfish.o):(.text+0x25b7):
>>>>> more undefined references to `__errno_location' follow
>>>>>
>>>>>
>>>>> I'm assuming I'd need to customise the build process tp produce an
>>>>> appropriate C library as with `gmp-xen`, `zarith-xen` etc. The trouble
>>>>> is twofold:
>>>>>
>>>>>  - I don't really know what the thing I'm doing is. Is mirage-xen a
>>>>> different architecture? A different ABI? Do C libraries need to be
>>>>> cross-compiled, or just compiled with particular flags so that they'll
>>>>> work on xen-minios instead of gnu/linux?
>>>>
>>>> C libraries need to be compiled with different headers and flags (in
>>>> particular, -mno-red-zone). Using libraries compiled for Linux will
>>>> often appear to work, but may fail when interrupts occur, so the build
>>>> makes it hard to use them by accident.
>>>>
>>>> However, OCaml binaries can be shared (ocamlopt doesn't use a red zone
>>>> and the OCaml ABI is the same).
>>>>
>>>>>  - The existing scripts seem to be pretty ad-hoc - zarith gets away
>>>>> mainly with particular CFLAGS
>>>>> (https://github.com/ocaml/opam-repository/blob/master/packages/zarith-xen/zarith-xen.1.4/files/mirage-install.sh),
>>>>
>>>> It's using pkg-config to get the important flags. The others are
>>>> optimisations (except -fno-builtin, which is either not needed or
>>>> should be supplied by Mini-OS).
>>>>
>>>>> while gmp looks like a full cross-compilation using `--target` (and
>>>>> then does some other sneaky things).
>>>>>
>>>>> safepass uses OASIS for its build system, which doesn't seem to have
>>>>> any cross-compilation functionality built in as far as I could tell.
>>>>> Can anyone give me pointers on what I'd need to do to safepass' build
>>>>> scripts to get this working in mirage-xen, or at the least some
>>>>> clarification on why it doesn't work as-is and how cross compilation
>>>>> works for C libraries used in mirage-xen?
>>>>>
>>>>> [0]: https://github.com/timbertson/passe
>>>>
>>>> The io-page library provides a fairly simple template:
>>>>
>>>> https://github.com/mirage/io-page/blob/master/_oasis
>>>>
>>>> It builds io_page_unix and io_page_xen libraries. The C flags are
>>>> added to the main META file with:
>>>>
>>>> XMETAExtraLines: xen_linkopts = "-lio_page_xen_stubs"
>>>>
>>>> There's a command to get the Xen flags:
>>>>
>>>> PostConfCommand: ocaml postconf.ml
>>>>
>>>> The Xen source file (stub_alloc_pages_xen.c) is just a symlink to the
>>>> Unix one (you need a separate file because OASIS can't compile the
>>>> same source path with different flags).
>>>
>>> Thanks, Thomas. - that helps quite a lot
>>>
>>> I think I've followed the same pattern now, but still haven't been
>>> able to get safepass building.
>>>
>>> I'm using the following for CCOpt:
>>>
>>> `pkg-config --cflags --static mirage-xen-ocaml`
>>>
>>> Now the linking phase fails with:
>>>
>>> + ocamlfind ocamlmklib -o src/safepass_stubs src/crypt_blowfish.o
>>> src/bcrypt_stub.o
>>> ld: errno: TLS definition in /nix/store/<...>-glibc-2.21/lib/libc.so.6
>>> section .tbss mismatches non-TLS reference in src/crypt_blowfish.o
>>> /nix/store/<...>-glibc-2.21/lib/libc.so.6: could not read symbols: Bad value
>>> collect2: error: ld returned 1 exit status
>>> Command exited with code 2.
>>> E: Failure("Command ''ocamlbuild' src/libsafepass_stubs.a
>>> src/dllsafepass_stubs.so src/safepass.cma src/safepass.cmxa
>>> src/safepass.a src/safepass.cmxs -tag debug' terminated with error
>>> code 10")
>>
>> Try running it as "ocamlfind ocamlmklib -verbose" to see what commands
>> it's actually using. For io-page, I get:
>>
>> $ ocamlfind ocamlmklib -verbose -o lib/io_page_xen_stubs
>> lib/stub_alloc_pages_xen.o
>> Effective set of compiler predicates: autolink,byte,native
>> + ocamlmklib -ocamlc ocamlc.opt -ocamlopt ocamlopt.opt -verbose -o
>> lib/io_page_xen_stubs lib/stub_alloc_pages_xen.o
>> + gcc -shared  -o lib/dllio_page_xen_stubs.so lib/stub_alloc_pages_xen.o
>> + ar rc lib/libio_page_xen_stubs.a  lib/stub_alloc_pages_xen.o; ranlib
>> lib/libio_page_xen_stubs.a
>>
>> So it doesn't seem to be using ld at all, just ar.
>
> Tracing it with strace, that `gcc -shared` process ends up invoking
> `ld` internally.
>
> I tried running ocamlmklib with the `-failsafe` option, and it does
> continue past this and produce the static .a library using `ar`. But
> if I try and put that flag into the larger build it doesn't get much
> further, because it hits the same issue when creating
> src/safepass.cmxs:
>
> + ocamlfind ocamlopt -shared -I src src/safepass.cmxa src/bcrypt.cmx
> -o src/safepass.cmxs
> ld: errno: TLS definition in libc.so.6 section .tbss mismatches
> non-TLS reference in src/libsafepass_stubs.a(crypt_blowfish.o)
> libc.so.6: could not read symbols: Bad value
> collect2: error: ld returned 1 exit status
>
>>> This seems to be something special about errno being thread-local in
>>> glibc. Googling turns up a lot of advice on fixing this problem, but
>>> all of it assumes you're just building on linux for linux.
>>> bcrypt_stubs.c does <errno.h>, which I believe is being provided by
>>> minios-xen-0.8/include/minios-xen/errno.h.
>>>
>>> However, at link time I don't know where errno is supposed to come
>>> from. It seems to be coming from glibc, but shouldn't it be coming
>>> from minios instead given that's where the errno header comes from? I
>>> poked around with `nm` in minios' built files but couldn't find
>>> anything that actually defined errno.
>>
>> It's from mirage-xen:
>>
>> https://github.com/mirage/mirage-platform/blob/c035439de703edbfdd096044afdbab85f4cbf61f/bindings/main.c#L30
>
> Which compiled libraries does that end up in though?
>
> I tried `nm`-ing all the built files from mirage-{xen,xen-posix,xen-ocaml} 
> with:
>
> $ find /nix/store/inxv9vdvnz8cjs6236fmalj0mnaq3sgx-mirage-xen-ocaml-2.3.4
> /nix/store/vbz4pivjk196dw5h8rmg68qxh5m44cn9-mirage-xen-posix-2.3.4
> /nix/store/24lm5990pxzp71nsgnffcmb2i7n41fck-mirage-xen-2.3.3 -type f
> -exec bash -c 'if nm {} 2>/dev/null | grep -q errno; then echo {}; nm
> {} | grep errno; fi' \;
> /nix/store/inxv9vdvnz8cjs6236fmalj0mnaq3sgx-mirage-xen-ocaml-2.3.4/lib/mirage-xen-ocaml/libxenasmrun.a
>                  U errno
>                  U errno
>                  U errno
>                  U errno
> /nix/store/vbz4pivjk196dw5h8rmg68qxh5m44cn9-mirage-xen-posix-2.3.4/lib/mirage-xen-posix/libxenposix.a
>                  U errno
> 0000000000000e4a T __h_errno_location
> /nix/store/24lm5990pxzp71nsgnffcmb2i7n41fck-mirage-xen-2.3.3/lib/mirage-xen/libxencamlbindings.a
> 0000000000000004 C errno

This is it (C = Common).

> So there are only .a files (which ld ignores when creating a shared
> library, doesn't it?), and only libxencamlbindings actually defines it
> (the others just reference it), which doesn't seem like a library I'd
> necessarily need to link against.
>
> I don't know if mirage-xen actually makes use of .so libraries, as
> I've only seen it produce static executables. But even if it doesn't
> need them, I'd need to somehow tell oasis not to build shared versions
> of safepass (both the .so and the .cmxs).

Correct: Mirage on Xen does not use shared libraries. I don't know how
to prevent them from being built. The easiest option might be to
remove the uses of errno from the library (it doesn't look like the
OCaml bindings do anything with it anyway).

>>>  -- other things I've tried:
>>>
>>> 1) I tried using `pkg-config --cflags mirage-xen-ocaml` for CCLib,
>>> however `ocamlfind ocamlmklib` didn't seem to like that - it didn't
>>> understand "-m elf_x86_64" nor
>>> "-T/nix/store/<...>-minios-xen-0.8/lib/minios-xen/libminios.lds"
>>>
>>> 2) I notice that minios-xen/errno.h has:
>>>
>>> #ifdef HAVE_LIBC
>>> #include <mini-os/sched.h>
>>> extern int errno;
>>> #define ERRNO
>>> #define errno (get_current()->reent._errno)
>>> #endif
>>>
>>> So just to see what happened, I tried adding -DHAVE_LIBC. But then I
>>> get errors because sys/reent.h can't be found, so that's probably not
>>> a useful path to explore.
>>>
>>> Any advice where to go from here?
>>>
>>> Thanks,
>>>  - Tim.
>>
>>
>>
>> --
>> Dr Thomas Leonard        http://roscidus.com/blog/
>> GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA



-- 
Dr Thomas Leonard        http://roscidus.com/blog/
GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel