Re: [Xen-devel] [xen-unstable test] 57852: regressions - FAIL

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Mon, 2015-06-08 at 09:07 +0100, Jan Beulich wrote:
> >>> On 05.06.15 at 12:48, <ian.campbell@xxxxxxxxxx> wrote:
> > On Fri, 2015-06-05 at 10:18 +0100, Jan Beulich wrote:
> >> >>> On 05.06.15 at 11:07, <ian.campbell@xxxxxxxxxx> wrote:
> >> > WRT the move to the colo, flights in 5xxxx are in the new one, while
> >> > 3xxxx are in the old one,
> >> > http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64
> >> >  
> >> > -xl-qemuu-win7-amd64.xen-unstable.html
> >> > shows that things seemed ok for 8 consecutive runs after the move
> >> > (ignoring blockages).
> >> 
> >> And when it went live, all systems being in use now got immediately
> >> deployed?
> > 
> > All the flights in the new colo seem to have been on fiano[01].
> 
> So are there just two hosts to run all x86 tests on? I thought one
> of the purposes of the switch was to have a wider pool of test
> systems...

There are about a dozen, but when a test is failing osstest will have a
preference for the host on which it failed last time (i.e. failures
become sticky to the host), in order to catch host specific failures I
think.

I think it was just coincidence that the first group of runs which
passed were on fiano0, although perhaps the pool was smaller then since
the colo was in the process of being commissioned.

The stickiness does make it a bit harder to know if a failure is host
specific though, since you often don't get results for other systems.

> > But having looked at the page again the early success was all on fiano0
> > while the later failures were all on fiano1.
> 
> But that's for the unstable install failures only as it looks. At the
> example of flight 57955 (testing 4.2) a local migration failure was
> observed on fiano0. Which would seem to support your earlier
> assumption that the install and migration issues are likely unrelated
> (yet their coincidence still strikes me as odd).

http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.html
 has the cross branch history for this test case. With one exception (on 
chardonay0, in a linux-next test) all the fails were on fiano[01] and they were 
all on branches which would use xen-unstable as the Xen version (xen-unstable 
itself and linux-* + qemu-mainline which both use the current xen.git#master as 
their Xen).

I've got some adhoc results over the weekend, all can be found at
http://logs.test-lab.xenproject.org/osstest/logs/<NNNNN>/test-amd64-amd64-xl-qemuu-win7-amd64/info.html
 for flight <NNNNN>. All of them are using the binaries from 57852.

I messed up my first command line and ran them all on fiano0 by mistake,
so there are more results than I was planning for.

Flight  Host    Failed at       Install step duration
57940   fiano0  ts-guest-stop   1483
57945   fiano0  ts-guest-stop   1640
57953   fiano0  ts-guest-stop   1473
57958   fiano0  ts-guest-stop   1472
57962   fiano0  windows-install 7512
57973   fiano0  windows-install 7693
57080   fiano0  ts-guest-stop   1534
57986   fiano0  windows-install 7203
57933   fiano0  ts-guest-stop   1529
57997   fiano0  ts-guest-stop   1494
58004   fiano0  ts-guest-stop   1492

58011   fiano1  ts-guest-stop   1408
58012   fiano1  ts-guest-stop   1529
58017   fiano1  ts-guest-stop   1466
58023   fiano1  ts-guest-stop   1624
58028   fiano1  windows-install 7208
58038   fiano1  ts-guest-stop   1479
58043   fiano1  ts-guest-stop   1493

58053   fiano0  windows-install 7439
58062   fiano0  windows-install 1916
58063   fiano0  windows-install 1477

58067   fiano1  ts-guest-stop   1453
58071   fiano1  ts-guest-stop   1550
58077   fiano1  windows-install 7156

That's 6/14 (43%) failure rate on fiano0 and 2/10 (20%) on fiano1. Which
differs form the apparent xen-unstable failure rate. But I wouldn't take
this as evidence that the two systems differ significantly, despite how
the unstable results looked at first glance.

On successful install the test step takes 1450-1650s, with one outlier
at 1916. The failures take 7000-7500s (test case timeout is 7000, so
with slop that fits). So on success it takes <30mins and on fail it has
been given nearly 2hours.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel