Re: [Xen-devel] [OSSTEST PATCH v2] sg-run-job: Coalesce a couple of repetitions

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Mon, 2015-11-30 at 16:49 +0000, Ian Jackson wrote:
> Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and move
> `guest-start.2' to after `guest-start.repeat' (reversing the contents
> of the latter so that the start comes before the stop).
> (guest-start.2 is still necessary because the start/stop test leaves
> the guest stopped, whereas the subsequent destroy test ought happen
> with the guest running.)
> 
> This change will allow the heisenbug compensator to see more of these
> failures as the same failures.
> 
> The overall effect includes a reduction of the number of localhost
> migrations from 11 to 10, but this is better than leaving a misleading
> testid containing the string `x10' (or changing the testid).
> 
> It is best to fold this way, keeping the testid of the step which
> previously had most of the regressions, because: the alternative,
> keeping the testid of the low-repetition step, would allow osstest to
> use previous lucky passes of the low-repetition step to justify
> current failures of the now-high-repetition step.
> 
> To check that the effect of the patch is as intended, I ran a before
> and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
> and diffed the sg-run-job transcripts and (b) looked in the db.
> 
> I also ran a real test (65261 in the Xen Project test lab) with a very
> similar version, which passed, and will re-run that before pushing.
> 
> (a):
> 
> Â c&p transcripts from mg-execute-flight email reports
> Â perl -i~ -pe 's/\b(38371|38370|65261|38395|38397)\b/FLIGHT/; s/^2015-
> 11-\d\d \S+ /TIME /' [tu]
> Â diff -u [tu] >v
> Â grep starting v
> 
> Â=>
> 
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-build-
> checkÂÂbuild-check(1)
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> saverestore host debian guest-saverestore.2
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> localmigrate host debian guest-localmigrate.2
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> localmigrate x10 host debian guest-localmigrate/x10
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> stop host debian guest-stop
> +TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-
> repeat-test 10 ts-guest-start host debian {;} ts-guest-stop host debian
> guest-start/debian.repeat
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> start host debian guest-start.2
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-
> repeat-test 10 ts-guest-stop host debian {;} ts-guest-start host debian
> guest-start/debian.repeat
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-
> destroy host debian guest-destroy
> ÂTIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-leak-
> check check host leak-check/check
> -TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-
> capture host capture-logs(24)
> +TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-
> capture host capture-logs(23)
> 
> (b)
> 
> osstestdb=> select * from (select job,stepno,step,status,testid from
> steps where flight=38370) before full outer join (select
> job,stepno,step,status,testid from steps where flight=38400) after using
> (testid) order by coalesce(before.stepno, after.stepno);
> ÂÂÂÂÂÂÂÂÂÂtestidÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂjobÂÂÂÂÂÂÂÂÂ| stepno
> |ÂÂÂÂÂÂÂÂÂÂÂÂÂstepÂÂÂÂÂÂÂÂÂÂÂÂÂ| status |ÂÂÂÂÂÂÂÂjobÂÂÂÂÂÂÂÂÂ| stepno
> |ÂÂÂÂÂÂÂÂÂÂÂÂÂstepÂÂÂÂÂÂÂÂÂÂÂÂÂ| status
> ---------------------------+--------------------+--------+---------------
> ---------------+--------+--------------------+--------+----------------
> --------------+--------
> Âbuild-check(1)ÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ1 | ts-build-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ1 | ts-build-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âhosts-allocateÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ2 | ts-hosts-
> allocateÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ2 | ts-hosts-
> allocateÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âhost-install(3)ÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ3 | ts-host-
> install-twiceÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ3 | ts-host-
> install-twiceÂÂÂÂÂÂÂÂ| pass
> Âhost-ping-check-nativeÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ4 | ts-host-ping-
> checkÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ4 | ts-host-ping-
> checkÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âxen-installÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ5 | ts-xen-
> installÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ5 | ts-xen-
> installÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âxen-bootÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ6 | ts-host-
> rebootÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ6 | ts-host-
> rebootÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âhost-ping-check-xenÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ7 | ts-host-ping-
> checkÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ7 | ts-host-ping-
> checkÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âleak-check/basis(8)ÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ8 | ts-leak-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ8 | ts-leak-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âdebian-installÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ9 | ts-debian-
> installÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂÂ9 | ts-debian-
> installÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âdebian-fixupÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ10 | ts-debian-
> fixupÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ10 | ts-debian-
> fixupÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âguest-startÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ11 | ts-guest-
> startÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ11 | ts-guest-
> startÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âmigrate-support-checkÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ12 | ts-migrate-
> support-checkÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ12 | ts-migrate-
> support-checkÂÂÂÂÂ| pass
> Âsaverestore-support-check | test-amd64-i386-xl |ÂÂÂÂÂ13 | ts-
> saverestore-support-check | passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ13 | ts-
> saverestore-support-check | pass
> Âguest-saverestoreÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ14 | ts-guest-
> saverestoreÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ14 | ts-guest-
> saverestoreÂÂÂÂÂÂÂÂÂ| pass
> Âguest-localmigrateÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ15 | ts-guest-
> localmigrateÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ15 | ts-guest-
> localmigrateÂÂÂÂÂÂÂÂ| pass
> Âguest-saverestore.2ÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ16 | ts-guest-
> saverestoreÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ16 | ts-guest-
> saverestoreÂÂÂÂÂÂÂÂÂ| pass
> Âguest-localmigrate.2ÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ17 | ts-guest-
> localmigrateÂÂÂÂÂÂÂÂ|
> passÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|
> Âguest-localmigrate/x10ÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ18 | ts-guest-
> localmigrateÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ17 | ts-guest-
> localmigrateÂÂÂÂÂÂÂÂ| pass
> Âguest-stopÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ19 | ts-guest-
> stopÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ18 | ts-guest-
> stopÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âguest-start.2ÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ20 | ts-guest-
> startÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ20 | ts-guest-
> startÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âguest-start/debian.repeat | test-amd64-i386-xl |ÂÂÂÂÂ21 | ts-repeat-
> testÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ19 | ts-repeat-
> testÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âguest-destroyÂÂÂÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ22 | ts-guest-
> destroyÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ21 | ts-guest-
> destroyÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âleak-check/checkÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ23 | ts-leak-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| passÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ22 | ts-leak-
> checkÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âcapture-
> logs(23)ÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
> ÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ23 | ts-logs-
> captureÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| pass
> Âcapture-logs(24)ÂÂÂÂÂÂÂÂÂÂ| test-amd64-i386-xl |ÂÂÂÂÂ24 | ts-logs-
> captureÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|
> passÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂ|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|
> (25 rows)
> 
> osstestdb=>
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>

Acked-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel