Re: [MirageOS-devel] Questions from potential new MirageOS Ocaml user

[Hannes Mehnert <hannes@xxxxxxxxxxx>]

Dear Luther,


I share your view regarding cloud computing likely moving towards edge
computing, especially with cheap off-the-shelf ARM64 boards being
available now (please see https://mirage.io/wiki/arm64 for recent work
on getting MirageOS on ARM64 (using Solo5, _not_ Xen)).


On 14/04/2018 05:20, Luther Flippen wrote:
> 1) THIS IS A WELL-KNOWN/WELL-DISCUSSED CRITICISM. Ocaml has a GIL (no shared 
> memory multicore) and no type classes. Apparently there has been a promise of 
> multicore development in progress that has been a long time coming to 
> fruition, and that has bred skepticism in at least some. Does multicore 
> (versus parallel-distributed) become less relevant as the number of cores per 
> node grows? My understanding is, and I have heard it argued that, multicore 
> behavior approaches distributed behavior with increasing core number. Is this 
> true? Modular implicits are supposed to be on the way too (which are supposed 
> to be better than type classes?), but how far off are they for mature 
> production-quality use?


The current state is: there is no support for multicore in OCaml.  There
is no support for multicore in the mirage-xen layer, neither in Solo5
(https://github.com/solo5/solo5 - basically a lightweight monitor /
execution environment targeting KVM) -- but there is a fork of Solo5 to
support multicore embedded in https://github.com/RWTH-OS/HermitCore

Several people in Cambridge are working hard on the OCaml side of
multicore, I expect this to be merged mainline within the next 5 years!

Modular implicits is as well a feature I'm looking forward to have in
OCaml, it will likely clean up lots of code (and get rid of the
polymorphic compare), as will an algebraic effect system.  I expect
these to get mainline within the next decade.


> 2) This, I think, is a big missing piece for MirageOS itself (especially in 
> light of Ocaml's GIL and the "no-forking" nature of unikernels): If I use 
> MirageOS as my development platform for Ocaml, which is what I would prefer, 
> what is the parallel distributed computing implementation that MirageOS will 
> use? (Note that this is different from Jitsu producing a swarm of application 
> copies for IO demand-response, from what I understand of it.) I read where 
> they will base it on the join calculus, but there seems to be some question 
> as to what is beyond that specifically. I read JoCaml, CIEL, and Opis 
> mentioned as possibilities. I assume this will involve communicating 
> unikernels, spread over multiple cores when on a single machine, and possibly 
> scaling up to multiple machines beyond that, for a given particular 
> application running in parallel. As a scientific programmer I might often 
> want parallel computing capability for any given application I deploy. 
> Obviously I will be limited to course grain parallelism mostly, this 
> environment being distributed.


Each MirageOS unikernel (as described above) is limited to a single
core.  In order to process data on multiple CPUs at the same time, what
is needed is a task scheduler / load balancer / broker / coordinator
which organises N times the same unikernel, each running on their own
CPU.  This requires a serialisation format (or a fully-fledged remote
procedure call API), what we have available at the moment (likely
incomplete and I haven't used all of them):
- Xen only are vchans (used by mirage-framebuffer etc.): shared memory
between virtual machines
- serialisation formats (several as well support a ppx deriver (so that
a developer only has to annotate their types with `[@@ deriving yyy]`,
and not write the serialiser/deserialiser manually):
  - Marshal
http://caml.inria.fr/pub/docs/manual-ocaml/libref/Marshal.html (depends
on concrete OCaml version)
  - S-expressions (https://github.com/janestreet/sexplib,
https://github.com/dbuenzli/sexpm)
  - Json (https://github.com/dbuenzli/jsonm,
https://github.com/mjambon/yojson)
  - protobuf (https://github.com/alavrik/piqi,
https://github.com/orbitz/ocaml-protobuf,
https://github.com/mransan/ocaml-protoc, https://github.com/yallop/ocaml-pb)
  - XML (https://github.com/ocsigen/tyxml/,
https://github.com/dbuenzli/xmlm)
- cap'n proto RPC (https://github.com/mirage/capnp-rpc)
- next to TCP/IP stack on top of tap interfaces, there's as well
preliminary support for netmap (in the netmap branch of
https://github.com/solo5-netmap/solo5/)

Each MirageOS unikernel itself uses lwt
(http://ocsigen.org/lwt/dev/api/Lwt), an asynchronous task library (no
preemption).  This already implements various synchronisation
primitives, such as actors and mailboxes.


> 3) This is another potentially big missing piece from MirageOS: What are the 
> GPGPU programming capabilities available in the MirageOS ecosystem? Is the 
> SPOC library available and usable through MirageOS on hypervisors on which 
> MirageOS can run? If so can it be utilized in conjunction with the 
> distributed computing solution discussed in the previous question above at 
> the same time? In other words, can a unikernel in a pool of 
> parallel-distributed communicating unikernels access GPGPU programming 
> resources (via SPOC, clMAGMA, or some alternative) on the node on which it 
> resides?


I haven't seen any effort to include GPUs with either OCaml or MirageOS.
 For the latter, some sort of (PCI?-)passthrough is needed in the
monitor (Xen/Solo5) to allow the guest to access the GPU in question.


> 4) To what extent can the above two capabilities if present, GPGPU and 
> parallel-distributed programming, mitigate the lack of multicore capability, 
> especially regarding MirageOS? In this context, what are my options for fast 
> unboxed linear algebra computations, especially running on MirageOS? Would 
> Lapack routines be viable (which are usually fortran/C/C++)? Obviously shared 
> in-place memory manipulation of unboxed arrays is very efficient in this 
> context, as in multicore, but can new GPGPU capability compensate? For 
> example, the OpenCL BLAS and clMAGMA libraries for OpenCL, and the cuBLAS and 
> NVBLAS libraries for CUDA, come to mind, or something similarly able to do 
> linear algebra on the GPU. More generally, has the Xen/MirageOS community 
> looked into support for the scientific computing community? (I do not mean 
> that they would necessarily need/want to compete with the more-niche 
> sub-community of professional HPC for speed.) I am not just talking about Big 
> Data input and then visualization/exploring/manipulating the data by the way. 
> Some might want to run large simulations (physical, biological, etc) of some 
> sort for example.


Mort mentioned owl, I suspect "cross-compiling" (a MirageOS unikernel
(a) runs in ring0, (b) only has a minimal libc available) BLAS and
LAPACK shouldn't be too hard to get up and running since they unlikely
depend on many external symbols.  We do the same for crypto primitives,
libgmp, and openlibm (the julialang port of FreeBSD's libm).  I'm not
sure how much CUDA etc. owl supports, I know there were at least some
experiments in that direction.


> 5) Not a deal-breaker, but MirageOS needs a real user guide/manual, 
> preferably available in pdf, or better yet a book, but not just blogs as they 
> mostly have now. I would like more documentation beyond concept-introduction 
> papers too, emphasizing developer-needed knowledge in actually using MirageOS 
> day to day. Correct me if I am wrong by pointing out such references. 
> Ultimately a book that also addressed the above 4 issues as well would be 
> fantastic.


As Mindy pointed out, we started an initial table of contents at
https://github.com/mirage/mirage-handbook - please open an issue or PR
in that repository if you feel that anything is missing -- of course the
content is unfortunately not there yet.


I'd recommend, if you're willing to use MirageOS and OCaml in your
setting, to get started with some small application, and try to get owl
running as MirageOS unikernel soon after.  If you're stuck, please don't
hesitate to ask in any of our communication channels
https://mirage.io/community/  I'd be excited to see physics simulations
running as an orchestra of MirageOS unikernels! :)

hannes
_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/mirageos-devel