[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xen-unstable test] 169819: regressions - FAIL


  • To: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • From: Jan Beulich <jbeulich@xxxxxxxx>
  • Date: Fri, 29 Apr 2022 12:37:13 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Epzp5HkR9KkZW4E2jKSKxyhqnnGDanuZ6PAHXAylqII=; b=amvsW8mwfINVI1pTKCDabU5Kk3v+G19nO9rixnw6nV38NbgFT+YHeMbdNtvfgQA/xYeNc6WiOKzVeUG6PfdNCZepJ/8nWGCbkUCEb/6EZKJ9ZG/DJehlBhF8Uu/07fzxwjhbV5uHfr1OHhwoa+jKoQUPrq6k7jTKNaguB7YkMFqIwdxRZGxsX7SyvU6WBqgZi47Y3squyzoFw3Qnl1Z2fJY5nwNzgILoH3mN7veTpm67/SQdcuSc4oxMid0bzvK+ksj2N2GanxBHEz8pi6JWgjCvyW02/kitlvw+xyCXf7btuGmwXTKc2oWKE47uDBvFSAcZF4Noh4UziadLWqKrUA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jbPCcu5XfafbjepqgsdFYWgH0HaFgNbSYBkyv6NJhM/0ycYBkLgbMh5kLNOaAqVtuv0f98oxvwp61gyDUigJC/BfH5izoBCGS1PZlZYeJCKrMmSxLsufr034jZbUVs+H3pkOcm+IpgbaOLqeY08Wfm7i17+TicYr+8nKNwsS4BSZ8IwDjyjQQeof08wkrZGnXeFlWhP0Ru7CEr0tcfeVCzxzaKiEeNdzwUI2ha5g0lq0WebDjXQEaOjasRyg7qzReyOalqYStYu/7xVwLEfV69UZ6sSAIsk3Vrd0E6ky9NiZiuV3KvukxyPtVPF2azZIb9GkL6ZgkyQFzWt2hxXKiA==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com;
  • Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx, Juergen Gross <jgross@xxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>, osstest service owner <osstest-admin@xxxxxxxxxxxxxx>
  • Delivery-date: Fri, 29 Apr 2022 10:37:23 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 29.04.2022 12:30, Roger Pau Monné wrote:
> On Fri, Apr 29, 2022 at 07:46:47AM +0000, osstest service owner wrote:
>> flight 169819 xen-unstable real [real]
>> flight 169843 xen-unstable real-retest [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/169819/
>> http://logs.test-lab.xenproject.org/osstest/logs/169843/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>>  test-arm64-arm64-examine      8 reboot                   fail REGR. vs. 
>> 169775
>>  test-arm64-arm64-libvirt-xsm  8 xen-boot                 fail REGR. vs. 
>> 169775
>>  test-arm64-arm64-libvirt-raw  8 xen-boot                 fail REGR. vs. 
>> 169775
>>  test-arm64-arm64-xl-credit1   8 xen-boot                 fail REGR. vs. 
>> 169775
>>  test-arm64-arm64-xl-thunderx  8 xen-boot                 fail REGR. vs. 
>> 169775
>>  test-arm64-arm64-xl           8 xen-boot                 fail REGR. vs. 
>> 169775
>>
>> Tests which are failing intermittently (not blocking):
>>  test-amd64-amd64-xl-qemut-debianhvm-i386-xsm 12 debian-hvm-install fail 
>> pass in 169843-retest
> 
> Looked into this one, and it's slightly concerning, guest seems to be
> stuck at installation:
> 
> Select and install software  [  481.093857] watchdog: BUG: soft lockup - 
> CPU#1 stuck for 23s! [ksoftirqd/1:17]
> [  509.093865] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [  545.093820] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  573.093809] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  609.093855] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [  637.093836] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [  673.093957] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  701.093854] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  733.093805] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  761.093817] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  797.093898] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [  825.093863] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  861.093865] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  889.093945] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [  925.093974] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  953.093925] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [  985.093832] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1013.093855] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1049.094031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1077.093860] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1113.093938] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1141.093803] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1177.094051] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1205.093805] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1237.093955] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1265.094004] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1301.093835] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1329.094039] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1365.093883] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1393.094167] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1429.093857] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1457.093900] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1489.094026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1517.093997] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1553.093996] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1581.094064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1617.094076] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1645.093882] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1681.093896] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1709.094022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1741.093870] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1769.093854] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1805.094017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1833.093837] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1869.094043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1897.094101] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1933.093879] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 1961.093933] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 1997.093952] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2025.093925] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2057.093895] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2085.094172] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2121.094018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2149.094021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2185.093931] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2213.093864] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2249.093951] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2277.093899] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2309.094054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2337.093956] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2373.094111] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2401.094132] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2437.094064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2465.094003] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2501.093959] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2529.094050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2561.094027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2589.094061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2625.093999] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2653.093956] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2689.094024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2717.094093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2753.093913] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2781.094095] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2813.093959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2841.094117] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2877.094041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2905.094009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 2941.093893] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 2969.094085] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! 
> [ksoftirqd/1:17]
> [ 3005.094061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 3033.093917] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> [ 3065.094060] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! 
> [ksoftirqd/1:17]
> 
> debina1 is one of the boxes I've put into production recently.

ISTR this issue having surfaced randomly before. With the randomness making
it difficult to investigate.

Jan




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.