Re: Test framework thoughts

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

On 14 Aug 2011, at 20:39, Thomas Gazagnaire wrote:

>> There are a few testing libraries that are easy to incorporate (e.g. oUnit 
>> or Kaputt), but I've been wondering if we can just use ocamlbuild to make 
>> things much easier.  It supports dynamic build dependencies (much as our 
>> CIEL framework also does), so a test can block mid-test and wait for another 
>> target to finish. 
> 
>> An individual test could be written as a module (say Foo), with a foo.test 
>> file that describes any dependencies and parameters, and ocamlbuild could 
>> just run them in parallel.  I'm still thinking it through, but wondering if 
>> anyone else has any thoughts on this. 
> 
> I don't see why using ocamlbuild can really help here.
> What I have started to do in another project is the following :
> 
> * The unit-tests of foo.ml are written in foo_test.ml, in keeping the same 
> structure (ie. if you have Bar.x inside foo.ml, then you have an easy way to 
> register the test "Bar.x"  in foo_test.ml :
> module Bar = struct
>  let x = Test.group "x" [ test (fun () -> <something which returns bool> ]
>  let test = Test.group "Bar" [ x ]
> end
> 
> * At the end of foo_test.ml, you have something like :
> Test.register [ Bar.test ]

Right, but the problem here is that you have to link in *all* the tests into 
one huge binary, which takes quite a while to compile. It's a similar problem 
to the tests in our ORM library.

On the other hand, if each test is a module, then ocamlbuild can link up a 
binary to run just that one test (with a target like 'ocamlbuild test1.run'), 
and that would pull in any dependencies automatically.

You could still have a 'run all.itarget' that would include all the tests for a 
full run, but this way I could just quickly define some oUnit tests for a 
module like JSON, and test it without having to worry about all the others 
tests running.

> For the complex settings to run some of the unit-tests, I guess you need to 
> have a library able to set-up correctly you environment, and use register 
> some post and pre-hooks if needed.

Indeed... some of the settings are very specific to the backend being used 
(like the Xen ones), so it's a bit tricky to do this from one big library.

The other thing with ocamlbuild is that it can run the tests in parallel 
without any additional help, as '-j' should just work if the dependencies are 
setup correctly.  Any errors would show up as a build failure, and the product 
file would be the performance results.

Anil