Re: [xen-unstable test] 169819: regressions - FAIL

[Juergen Gross <jgross@xxxxxxxx>]

On 02.05.22 12:49, Roger Pau Monné wrote:
> On Mon, May 02, 2022 at 08:51:40AM +0200, Juergen Gross wrote:
> > On 29.04.22 13:00, Roger Pau Monné wrote:
> > > On Fri, Apr 29, 2022 at 12:37:13PM +0200, Jan Beulich wrote:
> > > > On 29.04.2022 12:30, Roger Pau Monné wrote:
> > > > > On Fri, Apr 29, 2022 at 07:46:47AM +0000, osstest service owner wrote:
> > > > > > flight 169819 xen-unstable real [real]
> > > > > > flight 169843 xen-unstable real-retest [real]
> > > > > > http://logs.test-lab.xenproject.org/osstest/logs/169819/
> > > > > > http://logs.test-lab.xenproject.org/osstest/logs/169843/
> > > > > >
> > > > > > Regressions :-(
> > > > > >
> > > > > > Tests which did not succeed and are blocking,
> > > > > > including tests which could not be run:
> > > > > >    test-arm64-arm64-examine      8 reboot                   fail REGR. vs. 
> > > > > > 169775
> > > > > >    test-arm64-arm64-libvirt-xsm  8 xen-boot                 fail REGR. vs. 
> > > > > > 169775
> > > > > >    test-arm64-arm64-libvirt-raw  8 xen-boot                 fail REGR. vs. 
> > > > > > 169775
> > > > > >    test-arm64-arm64-xl-credit1   8 xen-boot                 fail REGR. vs. 
> > > > > > 169775
> > > > > >    test-arm64-arm64-xl-thunderx  8 xen-boot                 fail REGR. vs. 
> > > > > > 169775
> > > > > >    test-arm64-arm64-xl           8 xen-boot                 fail REGR. vs. 
> > > > > > 169775
> > > > > >
> > > > > > Tests which are failing intermittently (not blocking):
> > > > > >    test-amd64-amd64-xl-qemut-debianhvm-i386-xsm 12 debian-hvm-install fail pass 
> > > > > > in 169843-retest
> > > > > Looked into this one, and it's slightly concerning, guest seems to be
> > > > > stuck at installation:
> > > > >
> > > > > Select and install software  [  481.093857] watchdog: BUG: soft lockup - CPU#1 
> > > > > stuck for 23s! [ksoftirqd/1:17]
> > > > > [  509.093865] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  545.093820] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  573.093809] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  609.093855] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  637.093836] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  673.093957] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  701.093854] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  733.093805] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  761.093817] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  797.093898] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  825.093863] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  861.093865] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  889.093945] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  925.093974] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  953.093925] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [  985.093832] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1013.093855] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1049.094031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1077.093860] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1113.093938] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1141.093803] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1177.094051] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1205.093805] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1237.093955] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1265.094004] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1301.093835] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1329.094039] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1365.093883] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1393.094167] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1429.093857] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1457.093900] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1489.094026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1517.093997] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1553.093996] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1581.094064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1617.094076] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1645.093882] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1681.093896] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1709.094022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1741.093870] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1769.093854] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1805.094017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1833.093837] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1869.094043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1897.094101] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1933.093879] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1961.093933] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 1997.093952] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2025.093925] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2057.093895] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2085.094172] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2121.094018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2149.094021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2185.093931] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2213.093864] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2249.093951] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2277.093899] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2309.094054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2337.093956] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2373.094111] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2401.094132] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2437.094064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2465.094003] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2501.093959] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2529.094050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2561.094027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2589.094061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2625.093999] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2653.093956] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2689.094024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2717.094093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2753.093913] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2781.094095] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2813.093959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2841.094117] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2877.094041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2905.094009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2941.093893] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 2969.094085] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 3005.094061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 3033.093917] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > > [ 3065.094060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> > > > > [ksoftirqd/1:17]
> > > > >
> > > > > debina1 is one of the boxes I've put into production recently.
> > > > ISTR this issue having surfaced randomly before. With the randomness making
> > > > it difficult to investigate.
> > > Indeed, it seems to happen on other boxes also:
> > >
> > > http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/xen-unstable.html
> > >
> > > At least doesn't seem to be explicitly related to debinas.
> > How valuable is it to test systems using a kernel from 2019?
> >
> > Chances are high that bugs which might trigger are corrected already.
> > IMO trying to find probably known bugs is a waste of time.
> This is the kernel from oldstable Debian distribution, which is still
> supported (until June 2024 AFAICT).  I think it's not unlikely that
Yeah, but at least the install kernel doesn't seem to have received any
updates since 2.5 years now (compile date is reported to be 2019-09-20).
And supported doesn't mean that a kernel will run without any further
fixes until then, but that fixes are applied. And this seems not to have
happened here for quite some time.

> people are running those kernels in guests.  I wouldn't mind testing
> newer guest kernels, but someone would have to update osstest to use
> Debian 11 which ships with kernel 5.10.
I don't mind the base kernel version, just the lack of applied patches.

And BTW: this install kernel seems to lack any Xen support:

[    0.012917] Booting paravirtualized kernel on bare hardware


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature
Description: OpenPGP digital signature