[Xen-users] xen/linux bug?

[Adam Goryachev <mailinglists@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I've created a new debian stable domU on an existing debian stable dom0 
with xen 4.1 installed from packages.
I installed some applications/etc into the domU, but haven't had time to 
actually do anything, so it should be very much idle.
In fact, the domU has been up almost 12 days:
 11:51:52 up 11 days, 22:26,  1 user,  load average: 0.00, 0.01, 0.05
and xm list shows less than 1 hour CPU time:
pabx                                         6  2048     2 -b----   3325.7

I noticed when I looked at it today, there are a number of kernel errors 
(BUG).
The last line of the normal bootup (from dmesg output) through to the 
end of the first BUG are here:
[   18.216073] eth0: no IPv6 routers present
[12957.032892] hrtimer: interrupt took 180298877 ns
[571901.759780] sched: RT throttling activated
[943100.068753] BUG: soft lockup - CPU#1 stuck for 23s! [swapper/1:0]
[943100.068753] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache 
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys 
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache 
xen_blkfront xen_netfront
[943100.068753] CPU 1
[943100.068753] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache 
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys 
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache 
xen_blkfront xen_netfront
[943100.068753]
[943100.068753] Pid: 0, comm: swapper/1 Not tainted 3.2.0-4-amd64 #1 
Debian 3.2.60-1+deb7u1
[943100.068753] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>] 
hypercall_page+0x22a/0x1000
[943100.068753] RSP: e02b:ffff88007fd03e90  EFLAGS: 00000246
[943100.068753] RAX: 0000000000040001 RBX: ffffffff816040c0 RCX: 
ffffffff8100122a
[943100.068753] RDX: ffff88007fd03e30 RSI: 0000000000000000 RDI: 
0000000000000000
[943100.068753] RBP: ffff88007d371fd8 R08: 0000000000000005 R09: 
0000000000000004
[943100.068753] R10: 0000000000000020 R11: 0000000000000246 R12: 
0000000000000100
[943100.068753] R13: 0000000000000003 R14: 0000000000000008 R15: 
ffff88007d371fd8
[943100.068753] FS:  00007fac5856c7c0(0000) GS:ffff88007fd00000(0000) 
knlGS:0000000000000000
[943100.068753] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
[943100.068753] CR2: 00007fac568e8850 CR3: 0000000001605000 CR4: 
0000000000000660
[943100.068753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[943100.068753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[943100.068753] Process swapper/1 (pid: 0, threadinfo ffff88007d370000, 
task ffff88007d360780)
[943100.068753] Stack:
[943100.068753]  ffff88007fd0e980 0000000000000001 ffffffff81006790 
ffffffff81006d22
[943100.068753]  ffff88007fd0e980 0000000000000020 0000000000000004 
0000000000000005
[943100.068753]  0000000000000200 0000000000000001 ffff88007fd03e30 
0000000000000001
[943100.068753] Call Trace:
[943100.068753]  <IRQ>
[943100.068753]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753]  [<ffffffff81006d0f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[943100.068753]  [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[943100.068753]  [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[943100.068753]  [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[943100.068753]  [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[943100.068753]  [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[943100.068753]  [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[943100.068753]  [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[943100.068753]  [<ffffffff8135783e>] ? xen_do_hypervisor_callback+0x1e/0x30
[943100.068753]  <EOI>
[943100.068753]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753]  [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[943100.068753]  [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[943100.068753]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc cc 
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 00 
0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[943100.068753] Call Trace:
[943100.068753]  <IRQ>  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa
[943100.068753]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753]  [<ffffffff81006d0f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[943100.068753]  [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[943100.068753]  [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[943100.068753]  [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[943100.068753]  [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[943100.068753]  [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[943100.068753]  [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[943100.068753]  [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[943100.068753]  [<ffffffff8135783e>] ? xen_do_hypervisor_callback+0x1e/0x30
[943100.068753]  <EOI>  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753]  [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[943100.068753]  [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[943100.068753]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4

The second occurrence was similar:
[1024036.074678] BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0]
[1024036.074678] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache 
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys 
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache 
xen_blkfront xen_netfront
[1024036.074678] CPU 1
[1024036.074678] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache 
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys 
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache 
xen_blkfront xen_netfront
[1024036.074678]
[1024036.074678] Pid: 0, comm: swapper/1 Not tainted 3.2.0-4-amd64 #1 
Debian 3.2.60-1+deb7u1
[1024036.074678] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>] 
hypercall_page+0x22a/0x1000
[1024036.074678] RSP: e02b:ffff88007fd03e90  EFLAGS: 00000246
[1024036.074678] RAX: 0000000000040001 RBX: ffffffff816040c0 RCX: 
ffffffff8100122a
[1024036.074678] RDX: ffff88007fd03e30 RSI: 0000000000000000 RDI: 
0000000000000000
[1024036.074678] RBP: ffff88007d371fd8 R08: 0000000000000020 R09: 
0000000000000020
[1024036.074678] R10: 0000000000000020 R11: 0000000000000246 R12: 
0000000000000100
[1024036.074678] R13: 0000000000000001 R14: 0000000000000008 R15: 
ffff88007d371fd8
[1024036.074678] FS:  00007fac5856c7c0(0000) GS:ffff88007fd00000(0000) 
knlGS:0000000000000000
[1024036.074678] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
[1024036.074678] CR2: 00007fac568e8850 CR3: 0000000001605000 CR4: 
0000000000000660
[1024036.074678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[1024036.074678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[1024036.074678] Process swapper/1 (pid: 0, threadinfo ffff88007d370000, 
task ffff88007d360780)
[1024036.074678] Stack:
[1024036.074678]  ffff88007fd0e980 0000000000000001 ffffffff81006790 
ffffffff81006d22
[1024036.074678]  ffff88007fd0e980 0000000000000020 0000000000000020 
0000000000000020
[1024036.074678]  0000000000000200 0000000000000001 ffff88007fd03e30 
0000000000000001
[1024036.074678] Call Trace:
[1024036.074678]  <IRQ>
[1024036.074678]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678]  [<ffffffff81006d0f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[1024036.074678]  [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[1024036.074678]  [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[1024036.074678]  [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[1024036.074678]  [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[1024036.074678]  [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[1024036.074678]  [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[1024036.074678]  [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[1024036.074678]  [<ffffffff8135783e>] ? 
xen_do_hypervisor_callback+0x1e/0x30
[1024036.074678]  <EOI>
[1024036.074678]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678]  [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[1024036.074678]  [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[1024036.074678]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc 
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 
00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[1024036.074678] Call Trace:
[1024036.074678]  <IRQ>  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa
[1024036.074678]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678]  [<ffffffff81006d0f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[1024036.074678]  [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[1024036.074678]  [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[1024036.074678]  [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[1024036.074678]  [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[1024036.074678]  [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[1024036.074678]  [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[1024036.074678]  [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[1024036.074678]  [<ffffffff8135783e>] ? 
xen_do_hypervisor_callback+0x1e/0x30
[1024036.074678]  <EOI>  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678]  [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678]  [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678]  [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[1024036.074678]  [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[1024036.074678]  [<ffffffff81006cc9>] ? xen_irq_enable_direct_reloc+0x4/0x4

Note there are no NFS mounts or similar.
Memory stats now:
free
             total       used       free     shared    buffers cached
Mem:       2051044     262388    1788656          0     114992 79496
-/+ buffers/cache:      67900    1983144
Swap:      2171900          0    2171900

Disk info:
fdisk -l

Disk /dev/xvda: 53.7 GB, 53685415936 bytes
255 heads, 63 sectors/track, 6526 cylinders, total 104854328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ee580

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1            2048   100507647    50252800   83  Linux
/dev/xvda2       100509694   104853503     2171905    5  Extended
/dev/xvda5       100509696   104853503     2171904   82  Linux swap / 
Solaris
The disk is provided by dom0, which is a multipath iSCSI device.

Debian packages on dom0:
dpkg -l|grep xen
ii  libxen-4.1 4.1.4-3+deb7u1               amd64        Public libs for Xen
ii  libxenstore3.0 4.1.4-3+deb7u1               amd64        Xenstore 
communications library for Xen
ii  xen-hypervisor-4.1-amd64 4.1.4-3+deb7u1               amd64        
Xen Hypervisor on AMD64
ii  xen-linux-system-3.2.0-4-amd64 3.2.60-1+deb7u3              
amd64        Xen system with Linux 3.2 on 64-bit PCs (meta-package)
ii  xen-linux-system-amd64 3.2+46                       amd64        Xen 
system with Linux for 64-bit PCs (meta-package)
ii  xen-system-amd64 4.1.4-3+deb7u1               amd64        Xen 
System on AMD64 (meta-package)
ii  xen-utils-4.1 4.1.4-3+deb7u1               amd64        XEN 
administrative tools
ii  xen-utils-common 4.1.4-3+deb7u1               all          Xen 
administrative tools - common files
ii  xenstore-utils 4.1.4-3+deb7u1               amd64        Xenstore 
utilities for Xen
ii  linux-image-3.2.0-4-amd64 3.2.60-1+deb7u3              amd64        
Linux 3.2 for 64-bit PCs
Packages on domU
ii  linux-image-3.2.0-4-amd64 3.2.60-1+deb7u1                    
amd64        Linux 3.2 for 64-bit PCs
Can anyone provide any suggestions or information on how I might resolve 
this before I need to actually use it?
Thanks,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users