Re: [Xen-devel] Current LibXL Status

[George Dunlap <George.Dunlap@xxxxxxxxxxxxx>]

On Thu, Nov 19, 2015 at 11:23 AM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:
> create !
> title it libxl exit() on ENOMEM incompatible with gc'd languages
> thanks
>
> On Thu, 2015-11-19 at 10:55 +0000, Andrew Cooper wrote:
>> On 19/11/15 09:20, Ian Campbell wrote:
>> > On Wed, 2015-11-18 at 18:32 +0000, Martin Osterloh wrote:
>> >
>> > > I wanted to inquire about the current state of LibXL and in
>> > > particular
>> > > if there are any issues with using it in long-running processes.
>> > It is currently being used by libvirtd which I think has shaken out
>> > most of
>> > the issues with that environment.
>> >
>> > There are certain to be other bugs, but nothing show-stopping.
[snip]
>> Languages such as OCaml use -ENOMEM as a hint to run the garbage
>> collector some more.  I expect Haskell is the same.

So the implication you've made here, is that if libxl were to return
-ENOMEM to ocaml, that it would run the garbage collector and free up
some heap space, so that malloc() could succeed and the libxl
operation could proceed.

In preparation for writing up a descripion for an "outreach project"
(Outreachy/OPW/GSoC), I went and had a chat with some of the guys on
the xapi team (cc'd) about what such a thing might look like.

According to them, the ocaml garbage collector never frees memory.  It
grows its own internal heap as necessary, but it never reduces the
size of its heap.  So no -ENOMEM call or OOM callback can make a
failed malloc succeed.

One might then ask if libxl could simply allocate memory *from the
ocaml heap* itself.  It turns out that is also not tenable: Data in
the ocaml heap is stored in a heavily coded format.  (For example
integers are stored as (n*2+1), so that pointers can all be even and
non-pointers can all be odd.)

The only thing they said might be improved is:

1.  To know that libxl would never call exit() for any other reason
(which it seems is true)

2. To  have a callback in OOM conditions.  It's unlikely the process
as a whole could do anything but exit, but the ocaml runtime itself
might still have internal heap available, which would allow it to exit
more gracefully.

I think if Haskell or any library *is* capable of integrating with the
garbage collector, we'll have to wait until someone who understands
the language actually writes bindings and can ask for something they
know to be useful for that language.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel