Re: [Xen-devel] [OSSTEST PATCH] README.email: Add `Worked example of relevant regression in previous flight'

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-11-27 at 15:38 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
> CC: Jan Beulich <JBeulich@xxxxxxxx>

Acked-by: Ian Campbell <ian.campbell@xxxxxxxxxx>

> ---
> ÂREADME.email |ÂÂÂ51 +++++++++++++++++++++++++++++++++++++++++++++++++++
> Â1 file changed, 51 insertions(+)
> 
> diff --git a/README.email b/README.email
> index 5de63dd..e14a816 100644
> --- a/README.email
> +++ b/README.email
> @@ -89,6 +89,9 @@ history.ÂÂHere are some examples:
> ÂÂÂÂÂÂÂjustifiable because they prevent other tests from running and
> ÂÂÂÂÂÂÂcan so conceal bugs.)
> Â
> +ÂÂÂÂÂÂSee `Worked example of relevant regression in previous flight',
> +ÂÂÂÂÂÂbelow.
> +
> ÂÂÂÂfail in 58948 pass in 58965
> ÂÂÂÂfail in 58948 like 37628
> Â
> @@ -159,3 +162,51 @@ X-Osstest-Versions-That:
> ÂÂÂÂÂÂtreeÂÂÂÂÂrevision
> Â
> Â`This' is the version being tested, and `That' is the baseline.
> +
> +
> +
> +Worked example of relevant regression in previous flight
> +--------------------------------------------------------
> +
> +Suppose two test steps A and B, which normally run in that order:
> +ÂÂjob test-foo
> +ÂÂÂÂÂÂÂAÂÂÂ./ts-do-some-thing
> +ÂÂÂÂÂÂÂBÂÂÂ./ts-do-another-thing
> +
> +Suppose failure of A prevents the execution of B.ÂÂ(This is the usual
> +case where step A precedes step B; normally later steps in a job
> +depend on the success of earlier steps, because after an earlier
> +failure the testbed state is not necessarily well-defined.)
> +
> +Now suppose A has an intermittent bug, but B is totally broken.
> +
> +With our current policy on intermittent bugs[1], we would allow a push
> +despite the bug in A.ÂÂBut we should not allow a push despite B: the
> +100% reproducible failure of B should prevent all pushes.
> +
> +But the bug in B only shows up when A happens to pass.ÂÂSo the
> +heisenbug compensator has to insist on seeing an actual pass of B
> +(which in this hypothetical situation, will not occur).
> +
> +Eg, consider these flights:
> +
> +ÂÂ100ÂÂis now masterÂÂA pass, B passÂÂÂÂÂÂpushed
> +ÂÂ200ÂÂstagingÂÂÂÂÂÂÂÂA pass, B failÂÂÂÂÂÂ`B REGR. vs 100'
> +ÂÂ201ÂÂstagingÂÂÂÂÂÂÂÂA fail, B not runÂÂÂ`B fail in 200 REGR. vs 100'
> +
> +In flight 201, the failure of A is indeed justifiable as a heisenbug
> +because it can be seen to succeed in flight 200.ÂÂIt is the problem
> +with B which is actually blocking the push - but that failure is only
> +visible in flight 200.
> +
> +If, contrary to the suppositions above, the failure of B is actually a
> +heisenbug, then hopefully eventually both A and then B will happen to
> +pass in the same run.ÂÂEven if that particular flight has other
> +problems, a future evaluation of a test of the same version can use
> +that flight's passes of A and B to justify, respectively, whatever
> +failures of A and/or B that it comes across.
> +
> +[1] In principle we could have a different policy: to try to reject
> +intermittent bugs.ÂÂBut it would require a lot of test resources
> +because all tests would have to be repeated a lot, and naturally
> +intermittent bugs would slip through anyway.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel