[Xen-users] Ubuntu 8.04 DomU crash

[Chris 'Xenon' Hanson <xenon@xxxxxxxxxxxxxx>]

  My Debian 5 Lenny Dom0 has been running great for a few weeks with two PV 
DomU's running
Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian's 
packages. Both
DomU's are webservers, and one runs qmail as well. The DomU that crashes is not 
under
heavy load.

  However, twice since moving to this new system, one of my DomU's has crashed 
bad and
needed to be taken down and restarted. This last time, I captured some of the 
info in the
console. There's lots more if anyone wants to see it.

  Does this look familiar to anyone? I'm wondering if the FS (ext3) is 
implicated, or just
an innocent bystander. I'm using LVM storage on a RAID1 done with Linux kernel 
software RAID.

  Thanks in advance for any advice.

[140027.089210]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[140027.089216] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620
[140027.089220] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140027.089225] DR6: ffff0ff0 DR7: 00000400
[140027.089227]  [<c01a6d3d>] try_to_free_buffers+0x2d/0x90
[140027.089235]  [<c0167c15>] shrink_page_list+0x4c5/0x600
[140027.089242]  [<c0166daf>] isolate_lru_pages+0x5f/0x1c0
[140027.089248]  [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0
[140027.089255]  [<c016819c>] shrink_zone+0x9c/0x100
[140027.089260]  [<c016883c>] kswapd+0x44c/0x490
[140027.089266]  [<c013bac0>] autoremove_wake_function+0x0/0x40
[140027.089274]  [<c011e260>] complete+0x40/0x60
[140027.089280]  [<c01683f0>] kswapd+0x0/0x490
[140027.089285]  [<c013b802>] kthread+0x42/0x70
[140027.089289]  [<c013b7c0>] kthread+0x0/0x70
[140027.089294]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[140027.089299]  =======================
[140027.190913] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725]
[140027.190921]
[140027.190925] Pid: 4725, comm: multilog Tainted: G      D (2.6.24-23-xen #1)
[140027.190928] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 0
[140027.190936] EIP is at _spin_lock+0x7/0x10
[140027.190939] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000
[140027.190943] ESI: 00098002 EDI: 00000000 EBP: 00000000 ESP: eb8d1c78
[140027.190946]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140027.190954] CR0: 8005003b CR2: 080521c0 CR3: 2c7a1000 CR4: 00002620
[140027.190959] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140027.190962] DR6: ffff0ff0 DR7: 00000400
[140027.190966]  [<c01a720b>] __find_get_block_slow+0x5b/0x150
[140027.190977]  [<ee094cde>] ext3_mark_iloc_dirty+0x1ce/0x350 [ext3]
[140027.190991]  [<c01a75f6>] __find_get_block+0xd6/0x200
[140027.190998]  [<c01a7743>] __getblk+0x23/0x2b0
[140027.191005]  [<ee094f50>] __ext3_get_inode_loc+0xf0/0x360 [ext3]
[140027.191016]  [<c020a5e2>] __next_cpu+0x12/0x20
[140027.191022]  [<ee05ba6f>] start_this_handle+0x8f/0x360 [jbd]
[140027.191032]  [<c011d2d7>] find_busiest_group+0x177/0x550
[140027.191039]  [<ee095207>] ext3_reserve_inode_write+0x27/0x80 [ext3]
[140027.191050]  [<ee095283>] ext3_mark_inode_dirty+0x23/0x50 [ext3]
[140027.191060]  [<ee05bdc2>] journal_start+0x82/0xb0 [jbd]
[140027.191069]  [<ee0983e9>] ext3_dirty_inode+0x79/0x90 [ext3]
[140027.191081]  [<c01a35c0>] __mark_inode_dirty+0x30/0x180
[140027.191087]  [<c015ef46>] __generic_file_aio_write_nolock+0x266/0x540
[140027.191095]  [<c0119532>] kmap_atomic_prot+0x152/0x190
[140027.191102]  [<c015f285>] generic_file_aio_write+0x65/0xe0
[140027.191109]  [<ee093690>] ext3_file_write+0x30/0xc0 [ext3]
[140027.191119]  [<c0185385>] do_sync_write+0xd5/0x120
[140027.191126]  [<c013bac0>] autoremove_wake_function+0x0/0x40
[140027.191133]  [<c012c002>] __do_softirq+0x92/0x130
[140027.191139]  [<c01852b0>] do_sync_write+0x0/0x120
[140027.191144]  [<c0185c89>] vfs_write+0xb9/0x170
[140027.191149]  [<c01863c1>] sys_write+0x41/0x70
[140027.191154]  [<c0105832>] syscall_call+0x7/0xb
[140027.191160]  =======================
[140031.736833] BUG: soft lockup - CPU#3 stuck for 11s! [apache2:10423]
[140031.736840]
[140031.736842] Pid: 10423, comm: apache2 Tainted: G      D (2.6.24-23-xen #1)
[140031.736846] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 3
[140031.736851] EIP is at _spin_lock+0x7/0x10
[140031.736853] EAX: ed1a96d8 EBX: ed1a9694 ECX: c1c9cc40 EDX: 00000000
[140031.736856] ESI: c1c9cc40 EDI: 00000000 EBP: e8787dd0 ESP: e8787c34
[140031.736860]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140031.736865] CR0: 80050033 CR2: 087b3010 CR3: 14a2e000 CR4: 00002620
[140031.736869] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140031.736873] DR6: ffff0ff0 DR7: 00000400
[140031.736875]  [<c01a6d3d>] try_to_free_buffers+0x2d/0x90
[140031.736884]  [<c0167c15>] shrink_page_list+0x4c5/0x600
[140031.736891]  [<c0166daf>] isolate_lru_pages+0x5f/0x1c0
[140031.736898]  [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0
[140031.736906]  [<c016819c>] shrink_zone+0x9c/0x100
[140031.736910]  [<c0168ce2>] try_to_free_pages+0x152/0x250
[140031.736916]  [<c0162f2b>] __alloc_pages+0x14b/0x390
[140031.736922]  [<c017130c>] handle_mm_fault+0x89c/0x1350
[140031.736928]  [<c0109030>] timer_interrupt+0x3a0/0x770
[140031.736934]  [<c013eaea>] hrtimer_run_queues+0xda/0x1e0
[140031.736939]  [<c0173d2b>] vma_adjust+0x10b/0x440
[140031.736944]  [<c03298c6>] do_page_fault+0x366/0xe90
[140031.736949]  [<c01744d4>] vma_merge+0x144/0x1d0
[140031.736954]  [<c0174a75>] do_brk+0x195/0x240
[140031.736961]  [<c0175026>] sys_brk+0xb6/0xf0
[140031.736966]  [<c0329560>] do_page_fault+0x0/0xe90
[140031.736972]  [<c0328205>] error_code+0x35/0x40
[140031.736977]  [<c0320000>] vcc_ioctl+0x1e0/0x2d0
[140031.736982]  =======================
[140035.315737] BUG: soft lockup - CPU#2 stuck for 11s! [apache2:7138]
[140035.315745]
[140035.315748] Pid: 7138, comm: apache2 Tainted: G      D (2.6.24-23-xen #1)
[140035.315752] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 2
[140035.315758] EIP is at _spin_lock+0x7/0x10
[140035.315761] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000
[140035.315764] ESI: 000a98d2 EDI: 00000000 EBP: 00000000 ESP: e4f05c80
[140035.315768]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140035.315774] CR0: 80050033 CR2: b6eb2000 CR3: 2b8d9000 CR4: 00002620
[140035.315779] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140035.315783] DR6: ffff0ff0 DR7: 00000400
[140035.315786]  [<c01a720b>] __find_get_block_slow+0x5b/0x150
[140035.315795]  [<ee0957ee>] ext3_get_blocks_handle+0x9e/0x9e0 [ext3]
[140035.315809]  [<c01a75f6>] __find_get_block+0xd6/0x200
[140035.315815]  [<c01a7743>] __getblk+0x23/0x2b0
[140035.315821]  [<c0130370>] run_timer_softirq+0x30/0x220
[140035.315827]  [<ee096204>] ext3_getblk+0xd4/0x1d0 [ext3]
[140035.315840]  [<ee09a994>] ext3_find_entry+0x254/0x650 [ext3]
[140035.315853]  [<c018cc57>] permission+0x97/0x120
[140035.315858]  [<c01dba7c>] security_inode_permission+0x1c/0x20
[140035.315864]  [<c018e979>] __link_path_walk+0x79/0xe10
[140035.315870]  [<c018187e>] cache_alloc_refill+0x6e/0x570
[140035.315876]  [<c020a677>] _atomic_dec_and_lock+0x47/0x70
[140035.315884]  [<ee09c51c>] ext3_lookup+0x3c/0x120 [ext3]
[140035.315896]  [<c0197b04>] d_alloc+0x114/0x1a0
[140035.315902]  [<c018d1b5>] __lookup_hash+0xd5/0x120
[140035.315907]  [<c01908c4>] open_namei+0x114/0x640
[140035.315912]  [<c0107ec5>] local_clock+0x55/0xa0
[140035.315918]  [<c0183ab3>] do_filp_open+0x33/0x60
[140035.315923]  [<c020a677>] _atomic_dec_and_lock+0x47/0x70
[140035.315930]  [<c0183772>] get_unused_fd_flags+0x52/0xd0
[140035.315936]  [<c0183b2c>] do_sys_open+0x4c/0xe0
[140035.315942]  [<c0183bfc>] sys_open+0x1c/0x20
[140035.315947]  [<c0105832>] syscall_call+0x7/0xb
[140035.315951]  [<c0320000>] vcc_ioctl+0x1e0/0x2d0
[140035.315957]  =======================
[140038.904832] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:130]
[140038.904839]
[140038.904842] Pid: 130, comm: kswapd0 Tainted: G      D (2.6.24-23-xen #1)
[140038.904846] EIP: 0061:[<c0327d25>] EFLAGS: 00000286 CPU: 1
[140038.904851] EIP is at _spin_lock+0x5/0x10
[140038.904854] EAX: ed1a96d8 EBX: ed1a9694 ECX: c18566e0 EDX: 00000000
[140038.904857] ESI: c18566e0 EDI: 00000000 EBP: ed7c1f7c ESP: ed7c1dcc
[140038.904860]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[140038.904866] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620
[140038.904871] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140038.904875] DR6: ffff0ff0 DR7: 00000400
[140038.904878]  [<c01a6d3d>] try_to_free_buffers+0x2d/0x90
[140038.904886]  [<c0167c15>] shrink_page_list+0x4c5/0x600
[140038.904893]  [<c0166daf>] isolate_lru_pages+0x5f/0x1c0
[140038.904899]  [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0
[140038.904906]  [<c016819c>] shrink_zone+0x9c/0x100
[140038.904911]  [<c016883c>] kswapd+0x44c/0x490
[140038.904917]  [<c013bac0>] autoremove_wake_function+0x0/0x40
[140038.904924]  [<c011e260>] complete+0x40/0x60
[140038.904929]  [<c01683f0>] kswapd+0x0/0x490
[140038.904933]  [<c013b802>] kthread+0x42/0x70
[140038.904938]  [<c013b7c0>] kthread+0x0/0x70
[140038.904943]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[140038.904948]  =======================
[140039.006416] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725]
[140039.006424]
[140039.006427] Pid: 4725, comm: multilog Tainted: G      D (2.6.24-23-xen #1)
[140039.006431] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 0
[140039.006436] EIP is at _spin_lock+0x7/0x10
[140039.006439] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000
[140039.006443] ESI: 00098002 EDI: 00000000 EBP: 00000000 ESP: eb8d1c78
[140039.006446]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140039.006451] CR0: 8005003b CR2: 080521c0 CR3: 2c7a1000 CR4: 00002620
[140039.006456] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140039.006460] DR6: ffff0ff0 DR7: 00000400
[140039.006462]  [<c01a720b>] __find_get_block_slow+0x5b/0x150
[140039.006470]  [<ee094cde>] ext3_mark_iloc_dirty+0x1ce/0x350 [ext3]
[140039.006484]  [<c01a75f6>] __find_get_block+0xd6/0x200
[140039.006491]  [<c01a7743>] __getblk+0x23/0x2b0
[140039.006498]  [<ee094f50>] __ext3_get_inode_loc+0xf0/0x360 [ext3]
[140039.006510]  [<c020a5e2>] __next_cpu+0x12/0x20
[140039.006516]  [<ee05ba6f>] start_this_handle+0x8f/0x360 [jbd]
[140039.006526]  [<c011d2d7>] find_busiest_group+0x177/0x550
[140039.006532]  [<ee095207>] ext3_reserve_inode_write+0x27/0x80 [ext3]
[140039.006543]  [<ee095283>] ext3_mark_inode_dirty+0x23/0x50 [ext3]
[140039.006554]  [<ee05bdc2>] journal_start+0x82/0xb0 [jbd]
[140039.006563]  [<ee0983e9>] ext3_dirty_inode+0x79/0x90 [ext3]
[140039.006574]  [<c01a35c0>] __mark_inode_dirty+0x30/0x180
[140039.006580]  [<c015ef46>] __generic_file_aio_write_nolock+0x266/0x540
[140039.006588]  [<c0119532>] kmap_atomic_prot+0x152/0x190
[140039.006596]  [<c015f285>] generic_file_aio_write+0x65/0xe0
[140039.006602]  [<ee093690>] ext3_file_write+0x30/0xc0 [ext3]
[140039.006613]  [<c0185385>] do_sync_write+0xd5/0x120
[140039.006620]  [<c013bac0>] autoremove_wake_function+0x0/0x40
[140039.006626]  [<c012c002>] __do_softirq+0x92/0x130
[140039.006632]  [<c01852b0>] do_sync_write+0x0/0x120
[140039.006638]  [<c0185c89>] vfs_write+0xb9/0x170
[140039.006642]  [<c01863c1>] sys_write+0x41/0x70
[140039.006647]  [<c0105832>] syscall_call+0x7/0xb
[140039.006653]  =======================
[140043.552310] BUG: soft lockup - CPU#3 stuck for 11s! [apache2:10423]
[140043.552316]
[140043.552319] Pid: 10423, comm: apache2 Tainted: G      D (2.6.24-23-xen #1)
[140043.552322] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 3
[140043.552326] EIP is at _spin_lock+0x7/0x10
[140043.552329] EAX: ed1a96d8 EBX: ed1a9694 ECX: c1c9cc40 EDX: 00000000
[140043.552332] ESI: c1c9cc40 EDI: 00000000 EBP: e8787dd0 ESP: e8787c34
[140043.552335]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140043.552341] CR0: 80050033 CR2: 087b3010 CR3: 14a2e000 CR4: 00002620
[140043.552346] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140043.552349] DR6: ffff0ff0 DR7: 00000400
[140043.552352]  [<c01a6d3d>] try_to_free_buffers+0x2d/0x90
[140043.552359]  [<c0167c15>] shrink_page_list+0x4c5/0x600
[140043.552366]  [<c0166daf>] isolate_lru_pages+0x5f/0x1c0
[140043.552372]  [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0
[140043.552380]  [<c016819c>] shrink_zone+0x9c/0x100
[140043.552385]  [<c0168ce2>] try_to_free_pages+0x152/0x250
[140043.552391]  [<c0162f2b>] __alloc_pages+0x14b/0x390
[140043.552398]  [<c017130c>] handle_mm_fault+0x89c/0x1350
[140043.552403]  [<c0109030>] timer_interrupt+0x3a0/0x770
[140043.552410]  [<c013eaea>] hrtimer_run_queues+0xda/0x1e0
[140043.552415]  [<c0173d2b>] vma_adjust+0x10b/0x440
[140043.552420]  [<c03298c6>] do_page_fault+0x366/0xe90
[140043.552425]  [<c01744d4>] vma_merge+0x144/0x1d0
[140043.552430]  [<c0174a75>] do_brk+0x195/0x240
[140043.552436]  [<c0175026>] sys_brk+0xb6/0xf0
[140043.552440]  [<c0329560>] do_page_fault+0x0/0xe90
[140043.552445]  [<c0328205>] error_code+0x35/0x40
[140043.552451]  [<c0320000>] vcc_ioctl+0x1e0/0x2d0
[140043.552456]  =======================
[140047.127138] BUG: soft lockup - CPU#2 stuck for 11s! [apache2:7138]
[140047.127157]
[140047.127162] Pid: 7138, comm: apache2 Tainted: G      D (2.6.24-23-xen #1)
[140047.127166] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 2
[140047.127171] EIP is at _spin_lock+0x7/0x10
[140047.127174] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000
[140047.127177] ESI: 000a98d2 EDI: 00000000 EBP: 00000000 ESP: e4f05c80
[140047.127180]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[140047.127186] CR0: 80050033 CR2: b6eb2000 CR3: 2b8d9000 CR4: 00002620
[140047.127190] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140047.127194] DR6: ffff0ff0 DR7: 00000400
[140047.127197]  [<c01a720b>] __find_get_block_slow+0x5b/0x150
[140047.127205]  [<ee0957ee>] ext3_get_blocks_handle+0x9e/0x9e0 [ext3]
[140047.127218]  [<c01a75f6>] __find_get_block+0xd6/0x200
[140047.127225]  [<c01a7743>] __getblk+0x23/0x2b0
[140047.127230]  [<c0130370>] run_timer_softirq+0x30/0x220
[140047.127236]  [<ee096204>] ext3_getblk+0xd4/0x1d0 [ext3]
[140047.127249]  [<ee09a994>] ext3_find_entry+0x254/0x650 [ext3]
[140047.127261]  [<c018cc57>] permission+0x97/0x120
[140047.127266]  [<c01dba7c>] security_inode_permission+0x1c/0x20
[140047.127272]  [<c018e979>] __link_path_walk+0x79/0xe10
[140047.127279]  [<c018187e>] cache_alloc_refill+0x6e/0x570
[140047.127285]  [<c020a677>] _atomic_dec_and_lock+0x47/0x70
[140047.127292]  [<ee09c51c>] ext3_lookup+0x3c/0x120 [ext3]
[140047.127303]  [<c0197b04>] d_alloc+0x114/0x1a0
[140047.127309]  [<c018d1b5>] __lookup_hash+0xd5/0x120
[140047.127314]  [<c01908c4>] open_namei+0x114/0x640
[140047.127320]  [<c0107ec5>] local_clock+0x55/0xa0
[140047.127325]  [<c0183ab3>] do_filp_open+0x33/0x60
[140047.127331]  [<c020a677>] _atomic_dec_and_lock+0x47/0x70
[140047.127337]  [<c0183772>] get_unused_fd_flags+0x52/0xd0
[140047.127343]  [<c0183b2c>] do_sys_open+0x4c/0xe0
[140047.127348]  [<c0183bfc>] sys_open+0x1c/0x20
[140047.127353]  [<c0105832>] syscall_call+0x7/0xb
[140047.127358]  [<c0320000>] vcc_ioctl+0x1e0/0x2d0
[140047.127363]  =======================
[140050.719988] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:130]
[140050.719997]
[140050.720001] Pid: 130, comm: kswapd0 Tainted: G      D (2.6.24-23-xen #1)
[140050.720005] EIP: 0061:[<c0327d2a>] EFLAGS: 00000286 CPU: 1
[140050.720011] EIP is at _spin_lock+0xa/0x10
[140050.720014] EAX: ed1a96d8 EBX: ed1a9694 ECX: c18566e0 EDX: 00000000
[140050.720017] ESI: c18566e0 EDI: 00000000 EBP: ed7c1f7c ESP: ed7c1dcc
[140050.720021]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[140050.720029] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620
[140050.720033] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[140050.720037] DR6: ffff0ff0 DR7: 00000400
[140050.720041]  [<c01a6d3d>] try_to_free_buffers+0x2d/0x90
[140050.720051]  [<c0167c15>] shrink_page_list+0x4c5/0x600
[140050.720060]  [<c0166daf>] isolate_lru_pages+0x5f/0x1c0
[140050.720065]  [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0
[140050.720073]  [<c016819c>] shrink_zone+0x9c/0x100
[140050.720077]  [<c016883c>] kswapd+0x44c/0x490
[140050.720084]  [<c013bac0>] autoremove_wake_function+0x0/0x40
[140050.720090]  [<c011e260>] complete+0x40/0x60
[140050.720096]  [<c01683f0>] kswapd+0x0/0x490
[140050.720101]  [<c013b802>] kthread+0x42/0x70
[140050.720106]  [<c013b7c0>] kthread+0x0/0x70
[140050.720112]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[140050.720118]  =======================
[140050.818054] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725]
[140050.818060]


-- 
Chris 'Xenon' Hanson, omo sanza lettere                  Xenon AlphaPixel.com
PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
"There is no Truth. There is only Perception. To Perceive is to Exist." - Xen

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users