[Xen-devel] 2.6.28 domU panics in blk_invoke_request_fn()

["Christopher S. Aker" <caker@xxxxxxxxxxxx>]

Below are two similar panics we've captured thus far with some light 
testing of 2.6.28 domUs across our Xen cluster.  The kernel binary and 
associated files can be found here:
http://theshore.net/~caker/xen/BUGS/2.6.28/

------------[ cut here ]------------
Kernel BUG at c03eed00 [verbose debug info unavailable]
invalid opcode: 0000 [#1] SMP
last sysfs file:
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.28-linode15 #1)
EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0
EIP is at do_blkif_request+0x2e0/0x360
EAX: 00000001 EBX: 00000000 ECX: d51650c0 EDX: d529c3f0
ESI: d5984288 EDI: d59842c8 EBP: 000003b8 ESP: c06cde20
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process swapper (pid: 0, ti=c06cc000 task=c0671340 task.ti=c06cc000)
Stack:
 00000005 d5984288 00000288 d5874ae0 d5940000 d53bf3c4 00000000 00000001
 d5940000 00000002 00000006 d5984000 00000000 d51650c0 d529c3cc ffffffff
 d5874ae0 d5940000 00000003 00000001 c03a7165 d5940000 c03eed96 00000000
Call Trace:
 [<c03a7165>] blk_invoke_request_fn+0x95/0x100
 [<c03eed96>] kick_pending_request_queues+0x16/0x30
 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0
 [<c0159500>] handle_IRQ_event+0x30/0x60
 [<c015b418>] handle_level_irq+0x78/0xf0
 [<c010ab17>] do_IRQ+0x77/0x90
 [<c0105d0a>] check_events+0x8/0xe
 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150
 [<c01091c7>] xen_do_upcall+0x7/0xc
 [<c01013a7>] _stext+0x3a7/0x1000
 [<c010547f>] xen_safe_halt+0xf/0x20
 [<c0103c50>] xen_idle+0x20/0x40
 [<c0106d2f>] cpu_idle+0x5f/0xb0
Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 24 
3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> 0b 
eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20
c7 04 24 0b
EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:c06cde20
Kernel panic - not syncing: Fatal exception in interrupt
------------[ cut here ]------------
WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0()
Modules linked in:
Pid: 0, comm: swapper Tainted: G      D    2.6.28-linode15 #1
Call Trace:
 [<c0128aaf>] warn_on_slowpath+0x5f/0x90
 [<c03b92c6>] memmove+0x36/0x40
 [<c03dd10a>] scrup+0x7a/0xe0
 [<c0140967>] atomic_notifier_call_chain+0x17/0x20
 [<c03dd18f>] notify_update+0x1f/0x30
 [<c03dd41a>] vt_console_print+0x20a/0x2d0
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0105d0a>] check_events+0x8/0xe
 [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0105d0a>] check_events+0x8/0xe
 [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1
 [<c01295b0>] vprintk+0x170/0x350
 [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0
 [<c0106000>] stop_self+0x0/0x30
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0105d0a>] check_events+0x8/0xe
 [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c03df146>] do_unblank_screen+0x16/0x130
 [<c014a474>] smp_call_function+0x14/0x20
 [<c0128b3e>] panic+0x4e/0x100
 [<c010ac6c>] oops_end+0x8c/0xa0
 [<c0109b80>] do_invalid_op+0x0/0xa0
 [<c0109bff>] do_invalid_op+0x7f/0xa0
 [<c03eed00>] do_blkif_request+0x2e0/0x360
 [<c011689e>] pvclock_clocksource_read+0x4e/0xe0
 [<c01059b3>] get_abs_timeout+0x13/0x30
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0105d0a>] check_events+0x8/0xe
 [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c056245a>] error_code+0x72/0x78
 [<c03eed00>] do_blkif_request+0x2e0/0x360
 [<c03a7165>] blk_invoke_request_fn+0x95/0x100
 [<c03eed96>] kick_pending_request_queues+0x16/0x30
 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0
 [<c0159500>] handle_IRQ_event+0x30/0x60
 [<c015b418>] handle_level_irq+0x78/0xf0
 [<c010ab17>] do_IRQ+0x77/0x90
 [<c0105d0a>] check_events+0x8/0xe
 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150
 [<c01091c7>] xen_do_upcall+0x7/0xc
 [<c01013a7>] _stext+0x3a7/0x1000
 [<c010547f>] xen_safe_halt+0xf/0x20
 [<c0103c50>] xen_idle+0x20/0x40
 [<c0106d2f>] cpu_idle+0x5f/0xb0
---[ end trace d5f11e988eae6396 ]---



And this one is from another user on a different host:

------------[ cut here ]------------
Kernel BUG at c03eed00 [verbose debug info unavailable]
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/class/net/tun1/type
Modules linked in: dahdi_dummy dahdi

Pid: 157, comm: kswapd0 Not tainted (2.6.28-linode15 #1)
EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0
EIP is at do_blkif_request+0x2e0/0x360
EAX: 00000001 EBX: 00000000 ECX: c1bc7240 EDX: d6132d20
ESI: d5966000 EDI: d5966040 EBP: 00000165 ESP: d61e1b90
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process kswapd0 (pid: 157, ti=d61e0000 task=d6106ef0 task.ti=d61e0000)
Stack:
 00000005 d5966000 00000000 d61da9e0 d596e000 d5488d0c 00000000 00000018
 d596e000 00000002 00000000 d5966000 00000000 c1bc7240 d613294c ffffffff
 d61da9e0 d596e000 00000006 00000018 c03a7165 d596e000 c03eed96 00000000
Call Trace:
 [<c03a7165>] blk_invoke_request_fn+0x95/0x100
 [<c03eed96>] kick_pending_request_queues+0x16/0x30
 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0
 [<c03a514d>] elv_next_request+0x1d/0x170
 [<c0159500>] handle_IRQ_event+0x30/0x60
 [<c015b418>] handle_level_irq+0x78/0xf0
 [<c010ab17>] do_IRQ+0x77/0x90
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150
 [<c01091c7>] xen_do_upcall+0x7/0xc
 [<c0101227>] _stext+0x227/0x1000
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0182687>] kmem_cache_alloc+0x57/0xb0
 [<c016140d>] mempool_alloc+0x2d/0xe0
 [<c016140d>] mempool_alloc+0x2d/0xe0
 [<c01774dc>] try_to_unmap_one+0x8c/0x240
 [<c01a737b>] bvec_alloc_bs+0x7b/0x140
 [<c01a7641>] bio_alloc_bioset+0x51/0xe0
 [<c01a773b>] bio_alloc+0xb/0x20
 [<c017ae20>] get_swap_bio+0x20/0xa0
 [<c017afdb>] swap_writepage+0x4b/0xc0
 [<c017b050>] end_swap_bio_write+0x0/0x80
 [<c01692cd>] shrink_page_list+0x51d/0x650
 [<c016550f>] determine_dirtyable_memory+0x1f/0x90
 [<c0165598>] get_dirty_limits+0x18/0x2e0
 [<c0169837>] shrink_zone+0x437/0x860
 [<c0142117>] getnstimeofday+0x37/0xe0
 [<c0562025>] _spin_lock+0x5/0x10
 [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0
 [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100
 [<c016a7f8>] kswapd+0x4e8/0x500
 [<c0168220>] isolate_pages_global+0x0/0x220
 [<c013c3c0>] autoremove_wake_function+0x0/0x40
 [<c011ec00>] complete+0x40/0x60
 [<c016a310>] kswapd+0x0/0x500
 [<c013c0a2>] kthread+0x42/0x70
 [<c013c060>] kthread+0x0/0x70
 [<c0109177>] kernel_thread_helper+0x7/0x10
Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 24 
3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> 0b 
eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20 c7 04 24 0b
EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:d61e1b90
Kernel panic - not syncing: Fatal exception in interrupt
------------[ cut here ]------------
WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0()
Modules linked in: dahdi_dummy dahdi
Pid: 157, comm: kswapd0 Tainted: G      D    2.6.28-linode15 #1
Call Trace:
 [<c0128aaf>] warn_on_slowpath+0x5f/0x90
 [<c03b92c6>] memmove+0x36/0x40
 [<c03dd10a>] scrup+0x7a/0xe0
 [<c0140967>] atomic_notifier_call_chain+0x17/0x20
 [<c03dd18f>] notify_update+0x1f/0x30
 [<c03dd41a>] vt_console_print+0x20a/0x2d0
 [<c0562183>] _spin_lock_irqsave+0x33/0x50
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c01291dc>] release_console_sem+0x19c/0x1e0
 [<c01295b0>] vprintk+0x170/0x350
 [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0
 [<c0106000>] stop_self+0x0/0x30
 [<c0562183>] _spin_lock_irqsave+0x33/0x50
 [<c01324a7>] lock_timer_base+0x27/0x60
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c03df146>] do_unblank_screen+0x16/0x130
 [<c014a474>] smp_call_function+0x14/0x20
 [<c0128b3e>] panic+0x4e/0x100
 [<c010ac6c>] oops_end+0x8c/0xa0
 [<c0109b80>] do_invalid_op+0x0/0xa0
 [<c0109bff>] do_invalid_op+0x7f/0xa0
 [<c03eed00>] do_blkif_request+0x2e0/0x360
 [<c01055d5>] get_runstate_snapshot+0x75/0x90
 [<c0105a2f>] xen_sched_clock+0x1f/0x80
 [<c014135b>] __update_sched_clock+0x2b/0x140
 [<c011689e>] pvclock_clocksource_read+0x4e/0xe0
 [<c0562183>] _spin_lock_irqsave+0x33/0x50
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c056245a>] error_code+0x72/0x78
 [<c03eed00>] do_blkif_request+0x2e0/0x360
 [<c03a7165>] blk_invoke_request_fn+0x95/0x100
 [<c03eed96>] kick_pending_request_queues+0x16/0x30
 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0
 [<c03a514d>] elv_next_request+0x1d/0x170
 [<c0159500>] handle_IRQ_event+0x30/0x60
 [<c015b418>] handle_level_irq+0x78/0xf0
 [<c010ab17>] do_IRQ+0x77/0x90
 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20
 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150
 [<c01091c7>] xen_do_upcall+0x7/0xc
 [<c0101227>] _stext+0x227/0x1000
 [<c0105427>] xen_force_evtchn_callback+0x17/0x30
 [<c0182687>] kmem_cache_alloc+0x57/0xb0
 [<c016140d>] mempool_alloc+0x2d/0xe0
 [<c016140d>] mempool_alloc+0x2d/0xe0
 [<c01774dc>] try_to_unmap_one+0x8c/0x240
 [<c01a737b>] bvec_alloc_bs+0x7b/0x140
 [<c01a7641>] bio_alloc_bioset+0x51/0xe0
 [<c01a773b>] bio_alloc+0xb/0x20
 [<c017ae20>] get_swap_bio+0x20/0xa0
 [<c017afdb>] swap_writepage+0x4b/0xc0
 [<c017b050>] end_swap_bio_write+0x0/0x80
 [<c01692cd>] shrink_page_list+0x51d/0x650
 [<c016550f>] determine_dirtyable_memory+0x1f/0x90
 [<c0165598>] get_dirty_limits+0x18/0x2e0
 [<c0169837>] shrink_zone+0x437/0x860
 [<c0142117>] getnstimeofday+0x37/0xe0
 [<c0562025>] _spin_lock+0x5/0x10
 [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0
 [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100
 [<c016a7f8>] kswapd+0x4e8/0x500
 [<c0168220>] isolate_pages_global+0x0/0x220
 [<c013c3c0>] autoremove_wake_function+0x0/0x40
 [<c011ec00>] complete+0x40/0x60
 [<c016a310>] kswapd+0x0/0x500
 [<c013c0a2>] kthread+0x42/0x70
 [<c013c060>] kthread+0x0/0x70
 [<c0109177>] kernel_thread_helper+0x7/0x10
---[ end trace 505b292c57e00b05 ]---

-Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel