[Xen-devel] [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Build results done with one version of osstest are not necessarily
reuseable with a different version of osstest.  For example, the suite
may have changed.  The smoke tests try to reuse builds from
xen-unstable but if osstest changes incompatibly, the smoke tests
might repeatedly fail until a xen-unstable flight using the new
osstest completes.

(This issue is a problem for bisection too but that's less critical
and there is less of an easy answer.)

Address this by also considering the osstest revision when searching
for builds to reuse, so the smoke tests only reuse builds made with
the same version of osstest.  (If we are running with uncommitted
changes, we ignore that aspect and instead insist on only builds which
used the committed revision of osstest: this makes testing the
xen-unstable-smoke machinery marginally easier, and will make no
difference to production runs.)

This introduces a new problem, though: after an osstest push, there
will be no suitable builds until the next xen-unstable flight passes.
So each smoke test would run its own build.  This would delay the
smoke tests, and waste capacity.

To address this we permit the smoke tests to reuse (i) builds from a
suitable osstest flight (hopefully there will be an osstest flight
which justified the push of the osstest version we are running) or
(ii) previous builds done by a xen-unstable-smoke test (this is a
useful backstop).

sg-check-tested always reports the highest-numbered flight which
matches all the specified conditions, so overall this means that:

1. Normally, the most recent relevant build for each job will be a
xen-unstable build; xen-unstable-smoke will reuse those.  Recent
flights on the osstest branch will be unsuitable because they use
different osstest; and there will be no recent relevant builds on
xen-unstable-smoke because xen-unstable-smoke will prefer to reuse its
own old builds rather than make new ones.

2. After a normal osstest push (whether force-pushed manually on the
basis of a test flight, or automatically pushed), the xen-unstable
builds are unsuitable.  However, the osstest push _is_ suitable, and
its builds will be used.

3. After a manual force push of an untested osstest, there no suitable
builds on either xen-unstable or osstest.  The first
xen-unstable-smoke run will have to do all the builds.  However,
subsequent xen-unstable-smoke runs can just pick up those builds.
These same builds will be reused until a xen-unstable flight using the
new osstest produces a passing build.

(If the force pushed osstest causes a build to break, then
xen-unstable-smoke will keep retrying and failing that individual
build until a fix is pushed to osstest#production or xen#staging.)

We honour an environemnt variable SMOKE_HARNESS_REV to override the
automatic determination of the desired test harness revision.

Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
---
 cr-daily-branch |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index c7cc33b..364238c 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -281,9 +281,14 @@ flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING 
"$@"`
 
 case $branch in
 xen-unstable-smoke)
+       harness_rev=$(perl -e 'use Osstest; print get_harness_rev();')
+       : ${SMOKE_HARNESS_REV:=${harness_rev%+}}
+
        ./mg-adjust-flight-makexrefs -v $flight \
                '!build-amd64 !build-amd64-libvirt !build-armhf build-*' \
-               --debug --branch=xen-unstable --blessings=real
+               --debug --blessings=real                                \
+               --branch=xen-unstable,xen-unstable-smoke,osstest        \
+               --revision-osstest=$SMOKE_HARNESS_REV
        # Even adhoc or play flights ought to reuse only real
        # previous builds.
        ;;
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel