Re: [Xen-devel] Analysis: switch osstest default kernel from 3.14 to 4.1 (Was: Re: [linux-4.1 test] 60785: tolerable FAIL - PUSHED)

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

TL;DR: There are issues, but IMHO switching can be justified.

On Thu, 2015-09-03 at 09:59 +0100, Ian Campbell wrote:
> TL;DR: There are issues which need fixing first...
> 
> On Fri, 2015-08-21 at 17:24 +0000, osstest service owner wrote:
> > flight 60785 linux-4.1 real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/60785/
> > 
> > Failures :-/ but no regressions.
> > 
> > Tests which are failing intermittently (not blocking):
> >  test-amd64-i386-xl-qemut-debianhvm-amd64-xsm 19 guest
> > -start/debianhvm.repeat fail in 60746 pass in 60785
> >  test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 16 guest
> > -start/debianhvm.repeat fail pass in 60746
> >  test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 13 guest
> > -localmigrate fail pass in 60746
> 
> I've been considering updating osstest to switch from the current baseline
> kernel (the one used for all non linux-* tests) from 3.14 to 4.1.
> 
> The last pass on 3.14 was 60666[0] a little while ago on 13/8 (due to a
> known issue[1]). As well as this pass of 4.1 today I therefore also
> considered 60654[2] on 12/8. Two reports (like this one) are attached
> comparing 3.16:60666 against 4.1:60654 and 4.1:60795 respectively.

Some time has passed and the issues with migration v2 are now fixed.


The most recent pass of 3.14 is (still, sigh) 60666[0].

The most recent fail was 62085[1] which suffers from the same known issue
WRT gntdev (which breaks qdisk based tests for qcow2 and vhd)

The latest 4.1 pass was 62053[2] from Friday.

Attached are two reports of 4.1:62053 vs 3.14:60666 and 3.14:62085.

The failures are:

vs. 60666 only:

 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 13 guest-localmigrate 
fail REGR. vs. 60666

vs. 60666 and 62085:

 test-amd64-amd64-xl-qemuu-ovmf-amd64 19 guest-start/debianhvm.repeat fail 
REGR. vs. 60666

For the first considering the history:
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/linux-3.14.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/linux-4.1.html

It seems pretty obvious that this is host specific, and in fact 60666
failed, just as guest-localmigrate.2 not guest-localmigrate like 62053 did.

For the second the history:
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/linux-3.14.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/linux-4.1.html

Suggests the issue is pretty intermittent. FWIW the push associated with
62085 justified this failure using 61947[3] where it passed with the exact
same kernel version.

So, I think it could be justified to switch our default kernel version to
linux-4.1. Probably after 4.6 is released though.

Ian.

[0] http://lists.xen.org/archives/html/xen-devel/2015-09/msg00304.html
[1] http://lists.xen.org/archives/html/xen-devel/2015-09/msg02441.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-09/msg02364.html
[3] http://lists.xen.org/archives/html/xen-devel/2015-09/msg01921.html

Attachment: 3.14:60666-vs-4.1:62053
Description: Text document

Attachment: 3.14:62085-vs-4.1:62053
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel