Re: ocamldoc generation and packed files

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

The MLI files are actually quite often different from the implementation, as 
the interface can expose a more abstract view to other users.

For example:

foo.ml:

type t = int
let create () = 1
let add x = x + 1
let int_of_t x = x

foo.mli can provide a more abstract view that hides the internal implementation 
of t:

type t
val create: unit -> t
val add: t -> t
val int_of_t: t -> int

Now, users of Foo will never know that 'type t' is actually an integer 
internally, and have to go through the conversion function to make it into an 
integer.

In Mirage, there is a convenience target to generate a default inferred 
interface, via:

$ ocamlbuild foo.inferred.mli
$ cat _build/unix-direct/foo.inferred.mli

This can then be manually copied to the source directory to give a skeleton 
interface, and then edited to restrict the interface as desired. However, if 
you modify the implementation later on, you will need to manually update the 
.mli (as you are seeing).

MLI files are actually optional in normal OCaml, but are required in Mirage as 
a consequence of the build system we are using (the ocp-ocaml).  If an .mli 
file is not present, it is assumed to be blank (and thus, no interface will be 
exposed!). We could work around this, but it's good discipline to require an 
explicit interface file too...

Anil

On 19 Aug 2011, at 13:21, Haris Rotsos wrote:

> On a similar problem, is there a way using the makefile or the
> ocamlbuild command to regenerate the *.mli files? When I update a
> source file, I notice that the interface file is not updated leading
> to error in the compilation and I end up editing it by hand.
> 
> On 19 August 2011 12:27, Thomas Gazagnaire <thomas.gazagnaire@xxxxxxxxx> 
> wrote:
>> Well, actually it's seems quite easy to do, as the packed .annot file 
>> contains already the right location information. I can try to work out on a 
>> quick-and-dirty tool to fix your issue :-)
>> 
>> 
>> On Aug 19, 2011, at 1:21 PM, Thomas Gazagnaire wrote:
>> 
>>> I don't have concrete available solution for the .annot files. The two I 
>>> can see are :
>>> 
>>> * releasing the .ml -> .annot part of the compiler as a stand-alone tool 
>>> and use it to generate individual .annot files (I guess that's not very 
>>> different from running ocamlc -annot on every file)
>>> * having a tool to split out a packed .annot file into individual .annot 
>>> files. I guess it's not so difficult to do (just need to parse the 
>>> directive lines in the packed source file + its annot file and do some 
>>> comparison), but we don't have it yet.
>>> 
>>> Thomas
>>> 
>>> On Aug 19, 2011, at 11:58 AM, Anil Madhavapeddy wrote:
>>> 
>>>> As a followup to this, I've integrated ocaml-ocp into my tree and it's 
>>>> working really well. Thanks for releasing this Thomas!  There was an 
>>>> off-by-one in the cpp directives for line numbering (fixed in our tree).
>>>> 
>>>> It would be very useful to have some way for .annot files to work without 
>>>> compiling the individual sub-files; any thoughts on how this might work?
>>>> 
>>>> Anil
>>>> 
>>>> On 13 Aug 2011, at 20:39, Anil Madhavapeddy wrote:
>>>> 
>>>>> Sure, go for it if you get a chance.
>>>>> 
>>>>> The current packing "works", but the doc target cannot be built at the 
>>>>> same time as the normal code (because it causes the source pack to leave 
>>>>> .ml files in lib/std/foo.ml, which then causes the lib/std/foo.cm{xo} to 
>>>>> be rebuilt instead of the -packed copied .cmx to be used).
>>>>> 
>>>>> Basically, its a bit of a mess :-)
>>>>> 
>>>>> As a sneak preview, check out http://github.com/avsm/mirage-browser.git 
>>>>> (on index.html). This is a *very* rough jQuery-based module browser of 
>>>>> the Mirage OCamldoc that I'd like to have live on the website before the 
>>>>> CUFP tutorial.  The live search is pretty nice, and I'll flesh out the 
>>>>> rendering over the next few days.
>>>>> 
>>>>> This tool is also quite standalone as it uses the output of 
>>>>> odoc_json/ocamldoc, and so could be applied to the standard distribution 
>>>>> and also to the Citrix xapi-libs (if you're interested, David).
>>>>> 
>>>>> Killer feature: the js_of_ocaml interactive top-level should work in 
>>>>> here... :-)
>>>>> 
>>>>> Anil
>>>>> 
>>>>> On 13 Aug 2011, at 17:50, Thomas Gazagnaire wrote:
>>>>> 
>>>>>> No ocamlbuild integration yet but would be definitely useful to add. I
>>>>>> can have a look at integrating it to mirage build next week if it's
>>>>>> useful.
>>>>>> 
>>>>>> --
>>>>>> Thomas
>>>>>> 
>>>>>> 2011/8/13 Anil Madhavapeddy <anil@xxxxxxxxxx>:
>>>>>>> Thanks Thomas, that looks very useful and definitely better than the 
>>>>>>> script in tree at the moment.
>>>>>>> 
>>>>>>> One question: ocp-pack also requires that the list of files be passed 
>>>>>>> in dependency order, or else the resulting pack file will not compile.  
>>>>>>> Have you tried to integrate it as an ocamlbuild rule, so that it can 
>>>>>>> automatically sort the input modules before generates the packed file / 
>>>>>>> functor?
>>>>>>> 
>>>>>>> Anil
>>>>>>> 
>>>>>>> On 12 Aug 2011, at 20:01, Thomas Gazagnaire wrote:
>>>>>>> 
>>>>>>>> It will be anounced quite shortly on the ocaml mailing list, so I can
>>>>>>>> give you the link to ocp-pack :
>>>>>>>> 
>>>>>>>> http://www.ocamlpro.com/code/2011-08-10-ocaml-pack-functors.html
>>>>>>>> http://www.ocamlpro.com/files/ocp-pack-1.0.0.tar.gz
>>>>>>>> 
>>>>>>>> It should help you to pack modules easily and have a working doc.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Thomas
>>>>>>>> 
>>>>>>>> 2011/7/28 Anil Madhavapeddy <anil@xxxxxxxxxx>:
>>>>>>>>> That tool will be very useful! I've committed the 'make doc' target. 
>>>>>>>>> The output is still pretty dirty as we don't use the documentation 
>>>>>>>>> tags properly, but I'll go through adding .mli files and adding 
>>>>>>>>> proper documentation on the more stable interfaces.
>>>>>>>>> 
>>>>>>>>> It would be quite nice to eventually have a single documentation 
>>>>>>>>> output for all the backends, with an addition section saying 'only 
>>>>>>>>> present in Xen' or 'only present in Node'. I think that should be 
>>>>>>>>> possible by parsing the ocamldoc dump outputs, but something for the 
>>>>>>>>> future!
>>>>>>>>> 
>>>>>>>>> Anil
>>>>>>>>> 
>>>>>>>>> On 27 Jul 2011, at 16:56, Thomas Gazagnaire wrote:
>>>>>>>>> 
>>>>>>>>>> We have as well a program which pack together ML files (and is able 
>>>>>>>>>> to functorize packs as well...) it is not released yet, but I guess 
>>>>>>>>>> we can open-source it shortly.
>>>>>>>>>> 
>>>>>>>>>> But I think overriding the default rule for ocamldoc+pack in 
>>>>>>>>>> ocamlbuild is sufficient for now on so you should push your patch :-)
>>>>>>>>>> 
>>>>>>>>>> Thomas
>>>>>>>>>> 
>>>>>>>>>> On Jul 27, 2011, at 5:36 PM, Anil Madhavapeddy wrote:
>>>>>>>>>> 
>>>>>>>>>>> I'd really like to generate ocamldoc (HTML/PDF) of all the various 
>>>>>>>>>>> libraries so that it's easier to learn Mirage (and support editor 
>>>>>>>>>>> auto-completion, etc).
>>>>>>>>>>> 
>>>>>>>>>>> The big problem is that ocamldoc doesn't support packed modules, 
>>>>>>>>>>> and we use packing quite extensively (in Net, Http, Block, etc).
>>>>>>>>>>> 
>>>>>>>>>>> So I've hacked up an ocamlbuild target that concats together the 
>>>>>>>>>>> *source* ML files from an .mlpack and uses that to generate the 
>>>>>>>>>>> ocamldoc for the standard library, with one set of HTML files 
>>>>>>>>>>> generated per backend (Xen, Net-Direct, Net-Socket, and so on).
>>>>>>>>>>> 
>>>>>>>>>>> The issue is that these rules are a little grim: everywhere where 
>>>>>>>>>>> we have a .mlpack file at the moment, we need to override that rule 
>>>>>>>>>>> to generate a concatenated ML file that is used for ocamldoc (but 
>>>>>>>>>>> not for actual compilation, since line numbers get lost since those 
>>>>>>>>>>> aren't preserved when converting from ML files into a single big 
>>>>>>>>>>> one).
>>>>>>>>>>> 
>>>>>>>>>>> Does anyone have a better solution for ocamldoc and packed files? 
>>>>>>>>>>> All of the grimness here is hidden away in the ocamlbuild rules, so 
>>>>>>>>>>> I'm inclined to just commit this patch, and perhaps see about 
>>>>>>>>>>> adding -pack support into ocamldoc at a later stage (there's an 
>>>>>>>>>>> open bug in Mantis somewhere).
>>>>>>>>>>> 
>>>>>>>>>>> Dave, do you use pack in XAPI, and/or ocamldoc?  I wonder if 
>>>>>>>>>>> everyone else (like Core) also have their own swanky 'cat ML files 
>>>>>>>>>>> into one' script too...
>>>>>>>>>>> 
>>>>>>>>>>> -anil
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> Charalampos Rotsos
> PhD student
> The University of Cambridge
> Computer Laboratory
> William Gates Building
> JJ Thomson Avenue
> Cambridge
> CB3 0FD
> 
> Phone: +44-(0) 1223 767032
> Email: cr409@xxxxxxxxxxxx
>