Re: Test framework thoughts

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

On 15 Aug 2011, at 05:02, Richard Mortier wrote:

> 
> On 14 Aug 2011, at 15:46, Anil Madhavapeddy wrote:
>> Right, but the problem here is that you have to link in *all* the tests into 
>> one huge binary, which takes quite a while to compile. It's a similar 
>> problem to the tests in our ORM library.
>> 
>> On the other hand, if each test is a module, then ocamlbuild can link up a 
>> binary to run just that one test (with a target like 'ocamlbuild 
>> test1.run'), and that would pull in any dependencies automatically.
> 
> not sure if this question makes sense but - is it reasonable to have a mirage 
> program that only succeeds for a subset of the backends?

Yes -- consider the unit tests for Xen block front, or the Node.js URL parser.

The emerging rule of thumb it that if a type or function is defined, it will be 
the same as all the other backends. But a type or function may not be defined 
in all backends.

So, OS.Clock.tm will always have the same timezone structure, but OS.Evtchn is 
only defined in the Xen backend (or actually soon, in the UNIX backend too, due 
to being able to use the /dev/evtchn userspace device).

This is another reason why it's difficult to link one big test binary together 
(although we could still have one binary per backend, of course).

> 
> neat; even better would be an ability to assert that the results file produce 
> had to meet a template configuration, so you couldn't validly commit code 
> that broke performance.

Yeah, I'm thinking that all of these pre- and post- conditions are just targets 
in the make system, so you might have a .perf output in JSON format, and could 
run assertion scripts over those.

Dave pointed out the other day that, particularly with storage backends these 
days, we don't really know what the optimal settings of all the various knobs 
are, and so running a fitness function to explore the parameter space to 
optimise a VM for a particular hardware setup would be quite easy with Mirage.

Anil