Re: [Xen-devel] libxl and malloc failure (Re: Current LibXL Status)

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Ian Campbell writes ("Re: libxl and malloc failure (Re: [Xen-devel] Current 
LibXL Status)"):
> On Thu, 2016-02-18 at 18:15 +0000, Ian Jackson wrote:
> > Andrew Cooper writes ("Re: [Xen-devel] Current LibXL Status"):
> > > Languages such as OCaml use -ENOMEM as a hint to run the garbage
> > > collector some more.  I expect Haskell is the same.
> > 
> > This cannot possibly be true (if what you mean is that they use
> > ENOMEM[1] from malloc as such an indication).  It would make it
> > impossible to write a correct binding to a normal C library.
> > 
> > Typically C library which calls malloc will do so in the middle of its
> > execution.  Even if the library returns the resulting error up as
> > a distinguishable error code, you can't just make the same library
> > call again - it may have done half its work but not the other half.
> 
> FWIW there is some code in the Ocaml runtime which raises an Out_of_memory
> exception in response to ENOMEM in C land (and from ocaml code in some
> circumstances too).

ISTM that either there is something else which is turning ocaml out of
heap errors into ENOMEM, or that this code you describe is
fundamentally misconceived.

> In principal it ought to be possible to write an ocaml program which
> correctly dealt with those (and similar ones from other lower level
> bindings), for example the interactive command line interpreter catches
> Out_of_memory (but not Unix_error) and calls the gc before iterating. (I
> did see one reference in a book which suggested Out_of_memory wasn't
> catcheable, but that appears to be out of date from what I can tell).

But iterating in the cli would not recover the operation which had
failed.

In the general case, there is no recovery possible when a C language
library function returns ENOMEM.  Sometimes the internal state of the
library will now be unsuitable.  Certainly, a policy of always
retrying the operation after a gc would be very wrong indeed.

Ie if ocaml has the behaviour claimed by Andrew (ie, that the need for
a gc can be detected during malloc and reported as ENOMEM), all ocaml
programs would already suffer from occasional random failures of
operations being performed by code written in C.

Note that "need for a gc" is not an abnormal part of operation in a
consing garbage collected language like ocaml.  It is entirely
routine.  Ie these random failures would occur in normal, otherwise
entirely successful, operation.  They would occur even if the
program's actual memory use was modest.

> This of course relies on all the C bindings correctly turning errors into
> exceptions and the semantics of your particular daemon allowing for
> abandoning things half way through (which I suppose is a relative of some
> kind to crash-only s/w).

Even if the operations are properly abandonded, they have still been
erroneously made to fail.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel