Proposal - lightweight auto test binaries

[Ian Jackson <iwj@xxxxxxxxxxxxxx>]

Andy and I were discussing some new tests he's written which (will)
live in xen.git and which we would like to wire into osstest.

We came up with the following proposal (I have refined some of the
details beyond what we discussed on IRC):

 * Tests will live in tools/tests/ ideally, but they might
   live in other places in the xen.git source tree.

 * When appropriately configured, the xen.git build system
   will ship them into dist/install/usr/local/....

 * There will be a specific directory containing tests which
   are suitable for running automatically:
      dist/install/usr/local/lib/xen/autotests

 * The pass/success reporting API will be encoded into the
   filename.  For now we define only one API:
     dist/install/usr/local/lib/xen/autotests/simple/*

The "simple" test API is as follows:

 * Every executable file (or symlink to one) in
     dist/install/usr/local/lib/xen/autotests/simple/
   will be run with no arguments.

 * Only files whose names contain only [-0-9a-z] will be run.
   Others will be ignored.  NB _ is not allowed.

 * A test may exit with one of the XTF exit statuses:
          0 # XTF SUCCESS
          3 # XTF SKIP
          4 # XTF ERROR
          5 # XTF FAILURE
          6 # XTF CRASH
   (osstest will treat anything other than 0 and 3 as "fail")
   Any other termination should be treated as CRASH.

 * If a test exits nonzero it should print some information about
   the situation to stderr.

 * Tests may print other information to stderr, which will be captured
   and logged.

 * Tests *must not* print anything to stdout.  Test runners *must*
   ignore stdout (though they may log it, possibly mixed in with
   stderr).  stdout is reserved for future extension.

 * Tests should exit with XTF ERROR if they are passed any arguments.

 * Tests will be run sequentially on a Xen dom0.  If the environment
   (eg Xen configuration) or hardware is not suitable they should
   report SKIP.

 * If a test program reports SUCCESS or SKIP, it *must* leave the
   host in a good state suitable for running more tests.  It
   *must not* leak resources (domains, xenstore entries, files).
   (It may of course advance the domid counter.)

 * If a test reports any other status, the host may not been in a good
   state any more.

 * Tests should support being run from their installed location
       /usr/local/lib/xen/autotests/
   If they need any additional files or data or anything, that
   *must* be provided somewhere in dist/install/ for installation
   on the host.

 * Tests may also support being run from the build tree, or from
   dist/install/ without copying into /usr/local.  They may require
   an environment variable to be set to find ancillary files, or
   perhaps look at their argv[0], or something.

 * The protocol document (basically, derived from this email) will
   live in xen.git (and might be shipped as
     dist/install/usr/local/lib/xen/autotests/README.md
   or something)

Future directions:

We think we may invent a new multiple-test API where the test program
can report the outcome of multiple tests.  That will have its own
directory in dist/install/usr/local/lib/xen/autotests/.

We may invent extensions which involve passing environent variables,
or command line arguments, to "simple" test programs.  The former will
be ignored by older test programs and the latter rejected.  The
precise details are beyond the scope of this mail.

Extension may also involve defining a meaning for the test's stdout.

Ian.