Xen project Mailing List

Re: [Xen-devel] 2.6.35.7 pvops crash

To: Shaun Reitan <mailinglists@xxxxxxxxxxxxxxxx>

From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>

Date: Fri, 19 Nov 2010 09:30:01 -0800

Delivery-date: Fri, 19 Nov 2010 09:31:39 -0800

List-id: Xen developer discussion <xen-devel.lists.xensource.com>

On 11/19/2010 09:15 AM, Shaun Reitan wrote: > OK if I'm posting this to the wrong ML I'm sorry. I'm not sure if i > should be posting this to -devel or to -user, or maybe not to you guys > at all (maybe the kernel ML i should be sending this too?). Here's a > snip i was able to grab, sorry i didn't get it all. xen-devel is probably the right place for this, but its a bit unclear what's going on here. Unfortunately your report is missing the most interesting information. Can you go into more detail about what's going on here? This is a 32bit kernel, but what are the other details? What kernel version is it (is it a distro kernel?) dom0? domU? Which version of Xen? What was the system doing at the time? Thanks, J > > > <0> 00000000 d1df9a78 e8ddbb34 00000000 00000000 ec362580 c0105bd4 > 00400000 > <0> 00000000 00000001 c0000000 e424e010 ec362580 c2050b60 c147080c > cd5aeff8 > Call Trace: > [<c01893dd>] ? unmap_vmas+0x2dd/0x7e0 > [<c0105bd4>] ? check_events+0x8/0xc > [<c0105bcb>] ? xen_restore_fl_direct_end+0x0/0x1 > [<c0103630>] ? xen_mc_flush+0xe0/0x190 > [<c0105bd4>] ? check_events+0x8/0xc > [<c018e03c>] ? exit_mmap+0x8c/0x140 > [<c012d3fd>] ? mmput+0x3d/0xd0 > [<c01310e3>] ? exit_mm+0xd3/0xf0 > [<c05f0d60>] ? _raw_spin_lock_irq+0x10/0x20 > [<c0132647>] ? do_exit+0x167/0x6e0 > [<c05f0d37>] ? _raw_spin_lock_irqsave+0x27/0x40 > [<c0130b46>] ? kmsg_dump+0x76/0x140 > [<c013023b>] ? printk+0x1b/0x20 > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c010aa1f>] ? oops_end+0x9f/0xa0 > [<c0108d3f>] ? do_invalid_op+0x7f/0x90 > [<c0196d4f>] ? swap_count_continued+0x1bf/0x1d0 > [<c0105bcb>] ? xen_restore_fl_direct_end+0x0/0x1 > [<c05f0d81>] ? _raw_spin_unlock_irqrestore+0x11/0x20 > [<c0169afc>] ? __rcu_process_callbacks+0x6c/0x2d0 > [<c0169d95>] ? rcu_process_callbacks+0x35/0x40 > [<c0134546>] ? __do_softirq+0xb6/0x120 > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c0103f5d>] ? pte_pfn_to_mfn+0x6d/0x80 > [<c01041a9>] ? xen_set_pte+0x19/0x30 > [<c05f1676>] ? error_code+0x66/0x6c > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c0196d4f>] ? swap_count_continued+0x1bf/0x1d0 > [<c0196e87>] ? swap_entry_free+0x127/0x150 > [<c0197af7>] ? free_swap_and_cache+0x27/0xd0 > [<c0189661>] ? unmap_vmas+0x561/0x7e0 > [<c018e03c>] ? exit_mmap+0x8c/0x140 > [<c012d3fd>] ? mmput+0x3d/0xd0 > [<c01a6ad8>] ? flush_old_exec+0x238/0x570 > [<c01a1950>] ? do_sync_read+0x0/0xf0 > [<c01da7f9>] ? load_elf_binary+0x2d9/0x1810 > [<c017a27b>] ? __alloc_pages_nodemask+0xeb/0x560 > [<c03c27ef>] ? security_vm_enough_memory_mm+0x1f/0x40 > [<c01037e6>] ? __raw_callee_save_xen_pte_val+0x6/0x8 > [<c01884bb>] ? vm_normal_page+0x2b/0xd0 > [<c041200b>] ? strrchr+0xb/0x20 > [<c01d82e7>] ? load_misc_binary+0x87/0x3d0 > [<c018ad09>] ? __get_user_pages+0xe9/0x3e0 > [<c01a5d12>] ? get_arg_page+0x52/0xb0 > [<c01a5fdb>] ? search_binary_handler+0x9b/0x230 > [<c01a7733>] ? do_execve+0x1f3/0x250 > [<c010e683>] ? sys_execve+0x33/0x70 > [<c0107c22>] ? ptregs_execve+0x12/0x18 > [<c0107b58>] ? sysenter_do_call+0x12/0x28 > Code: b6 00 00 00 00 8b 54 24 08 8b 44 24 04 89 51 04 89 01 c3 66 90 > 64 a1 44 db 7d c0 85 c0 75 0c 64 c7 05 44 db 7d c0 01 00 00 00 c3 <0f> > 0b eb fe 90 90 90 90 90 90 fa fb 50 9d 9c 58 cf fb 0f 35 0f > EIP: [<c011c9b6>] paravirt_enter_lazy_mmu+0x16/0x20 SS:ESP 0069:e8ddba94 > ---[ end trace 684584b81fd0c02c ]--- > Fixing recursive fault but reboot is needed! > BUG: scheduling while atomic: httpd/14418/0x00000001 > Modules linked in: > Pid: 14418, comm: httpd Tainted: G D 2.6.35.7-1 #4 > Call Trace: > [<c05ef5e0>] ? schedule+0x550/0x610 > [<c0130013>] ? vprintk+0x193/0x3a0 > [<c0132b04>] ? do_exit+0x624/0x6e0 > [<c05f0d37>] ? _raw_spin_lock_irqsave+0x27/0x40 > [<c05f0cf5>] ? _raw_spin_trylock+0x5/0x10 > [<c0130b46>] ? kmsg_dump+0x76/0x140 > [<c013023b>] ? printk+0x1b/0x20 > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c010aa1f>] ? oops_end+0x9f/0xa0 > [<c0108d3f>] ? do_invalid_op+0x7f/0x90 > [<c011c9b6>] ? paravirt_enter_lazy_mmu+0x16/0x20 > [<c015104a>] ? tick_program_event+0x2a/0x40 > [<c0146ff7>] ? hrtimer_interrupt+0x147/0x240 > [<c0105bd4>] ? check_events+0x8/0xc > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c0105bd4>] ? check_events+0x8/0xc > [<c05f1676>] ? error_code+0x66/0x6c > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c011c9b6>] ? paravirt_enter_lazy_mmu+0x16/0x20 > [<c01893dd>] ? unmap_vmas+0x2dd/0x7e0 > [<c0105bd4>] ? check_events+0x8/0xc > [<c0105bcb>] ? xen_restore_fl_direct_end+0x0/0x1 > [<c0103630>] ? xen_mc_flush+0xe0/0x190 > [<c0105bd4>] ? check_events+0x8/0xc > [<c018e03c>] ? exit_mmap+0x8c/0x140 > [<c012d3fd>] ? mmput+0x3d/0xd0 > [<c01310e3>] ? exit_mm+0xd3/0xf0 > [<c05f0d60>] ? _raw_spin_lock_irq+0x10/0x20 > [<c0132647>] ? do_exit+0x167/0x6e0 > [<c05f0d37>] ? _raw_spin_lock_irqsave+0x27/0x40 > [<c0130b46>] ? kmsg_dump+0x76/0x140 > [<c013023b>] ? printk+0x1b/0x20 > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c010aa1f>] ? oops_end+0x9f/0xa0 > [<c0108d3f>] ? do_invalid_op+0x7f/0x90 > [<c0196d4f>] ? swap_count_continued+0x1bf/0x1d0 > [<c0105bcb>] ? xen_restore_fl_direct_end+0x0/0x1 > [<c05f0d81>] ? _raw_spin_unlock_irqrestore+0x11/0x20 > [<c0169afc>] ? __rcu_process_callbacks+0x6c/0x2d0 > [<c0169d95>] ? rcu_process_callbacks+0x35/0x40 > [<c0134546>] ? __do_softirq+0xb6/0x120 > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c0103f5d>] ? pte_pfn_to_mfn+0x6d/0x80 > [<c01041a9>] ? xen_set_pte+0x19/0x30 > [<c05f1676>] ? error_code+0x66/0x6c > [<c0108cc0>] ? do_invalid_op+0x0/0x90 > [<c0196d4f>] ? swap_count_continued+0x1bf/0x1d0 > [<c0196e87>] ? swap_entry_free+0x127/0x150 > [<c0197af7>] ? free_swap_and_cache+0x27/0xd0 > [<c0189661>] ? unmap_vmas+0x561/0x7e0 > [<c018e03c>] ? exit_mmap+0x8c/0x140 > [<c012d3fd>] ? mmput+0x3d/0xd0 > [<c01a6ad8>] ? flush_old_exec+0x238/0x570 > [<c01a1950>] ? do_sync_read+0x0/0xf0 > [<c01da7f9>] ? load_elf_binary+0x2d9/0x1810 > [<c017a27b>] ? __alloc_pages_nodemask+0xeb/0x560 > [<c03c27ef>] ? security_vm_enough_memory_mm+0x1f/0x40 > [<c01037e6>] ? __raw_callee_save_xen_pte_val+0x6/0x8 > [<c01884bb>] ? vm_normal_page+0x2b/0xd0 > [<c041200b>] ? strrchr+0xb/0x20 > [<c01d82e7>] ? load_misc_binary+0x87/0x3d0 > [<c018ad09>] ? __get_user_pages+0xe9/0x3e0 > [<c01a5d12>] ? get_arg_page+0x52/0xb0 > [<c01a5fdb>] ? search_binary_handler+0x9b/0x230 > [<c01a7733>] ? do_execve+0x1f3/0x250 > [<c010e683>] ? sys_execve+0x33/0x70 > [<c0107c22>] ? ptregs_execve+0x12/0x18 > [<c0107b58>] ? sysenter_do_call+0x12/0x28 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#3] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 31745, comm: mysqld Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[<c0123267>] EFLAGS: 00010282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c17f86c0 EDX: 80000004 > ESI: f57ff000 EDI: 0000001c EBP: 0000001c ESP: df171df4 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process mysqld (pid: 31745, ti=df170000 task=eb4b0030 task.ti=df170000) > Stack: > c20349c0 c05ef7c7 f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> df171eb8 0000001c 0000001c c012328c 00000163 80000000 b174ada0 > c017505b > <0> 00000000 c17f86c0 c891a978 0000001c 022cda20 c17f86c0 00000106 > c891a974 > Call Trace: > [<c05ef7c7>] ? io_schedule+0x77/0xa0 > [<c012328c>] ? kmap_atomic+0x1c/0x30 > [<c017505b>] ? file_read_actor+0xcb/0x100 > [<c017683a>] ? generic_file_aio_read+0x2ca/0x720 > [<c01a1a05>] ? do_sync_read+0xb5/0xf0 > [<c0169d95>] ? rcu_process_callbacks+0x35/0x40 > [<c03c1a7c>] ? security_file_permission+0xc/0x10 > [<c01a1aa6>] ? rw_verify_area+0x66/0xd0 > [<c01a2251>] ? vfs_read+0xa1/0x180 > [<c01a1950>] ? do_sync_read+0x0/0xf0 > [<c01a2781>] ? sys_read+0x41/0x70 > [<c0107b58>] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [<c0123267>] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:df171df4 > ---[ end trace 684584b81fd0c02d ]--- > note: mysqld[31745] exited with preempt_count 1 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#4] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 14396, comm: leechprotect Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[<c0123267>] EFLAGS: 00210282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c18b3bc0 EDX: 80000004 > ESI: f57ff000 EDI: 00001000 EBP: 000046fd ESP: ce9a3df4 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process leechprotect (pid: 14396, ti=ce9a2000 task=c4ecf7f0 > task.ti=ce9a2000) > Stack: > c05f1676 08cf2440 f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> ce9a3eb8 00001000 000046fd c012328c 00000163 80000000 08cf2440 > c017505b > <0> 00000000 c18b3bc0 e0dc8ef0 00001000 000046fd c18b3bc0 00000000 > e0dc8eec > Call Trace: > [<c05f1676>] ? error_code+0x66/0x6c > [<c012328c>] ? kmap_atomic+0x1c/0x30 > [<c017505b>] ? file_read_actor+0xcb/0x100 > [<c017683a>] ? generic_file_aio_read+0x2ca/0x720 > [<c01ded70>] ? dquot_file_open+0x0/0x40 > [<c01b1c3e>] ? dput+0x1e/0x120 > [<c01a1a05>] ? do_sync_read+0xb5/0xf0 > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c0105bd4>] ? check_events+0x8/0xc > [<c01053c7>] ? xen_force_evtchn_callback+0x17/0x30 > [<c03c1a7c>] ? security_file_permission+0xc/0x10 > [<c01a1aa6>] ? rw_verify_area+0x66/0xd0 > [<c0105bd4>] ? check_events+0x8/0xc > [<c01a2251>] ? vfs_read+0xa1/0x180 > [<c01a1950>] ? do_sync_read+0x0/0xf0 > [<c01a2781>] ? sys_read+0x41/0x70 > [<c0107b58>] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [<c0123267>] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:ce9a3df4 > ---[ end trace 684584b81fd0c02e ]--- > note: leechprotect[14396] exited with preempt_count 1 > ------------[ cut here ]------------ > kernel BUG at arch/x86/mm/highmem_32.c:45! > invalid opcode: 0000 [#5] SMP > last sysfs file: /sys/devices/vif-0/net/eth0/broadcast > Modules linked in: > > Pid: 13782, comm: cpanellogd Tainted: G D 2.6.35.7-1 #4 / > EIP: 0061:[<c0123267>] EFLAGS: 00010282 CPU: 4 > EIP is at kmap_atomic_prot+0x137/0x140 > EAX: 5b489063 EBX: db489067 ECX: c1ca5300 EDX: 80000004 > ESI: f57ff000 EDI: ccbaa500 EBP: eb4b3d14 ESP: eb4b3ca0 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 > Process cpanellogd (pid: 13782, ti=eb4b2000 task=d17257f0 > task.ti=eb4b2000) > Stack: > c01c2576 dcec2e3c f57ff000 00000298 00000053 00000163 80000000 80000000 > <0> 00001000 ccbaa500 eb4b3d14 c012328c 00000163 80000000 00000000 > c01c3b0e > <0> eaca9b38 00000001 eb4b3d14 00000000 c1ca5300 dcec2e3c 00001000 > 00000000 > Call Trace: > [<c01c2576>] ? __find_get_block_slow+0x46/0x140 > [<c012328c>] ? kmap_atomic+0x1c/0x30 > [<c01c3b0e>] ? __block_prepare_write+0x23e/0x380 > [<c01c3ce6>] ? block_write_begin_newtrunc+0x56/0xd0 > [<c0237c20>] ? ext3_get_block+0x0/0x100 > [<c01c41a3>] ? block_write_begin+0x53/0xc0 > [<c0237c20>] ? ext3_get_block+0x0/0x100 > [<c0280c6c>] ? journal_start+0x7c/0xa0 > [<c0239356>] ? ext3_write_begin+0x146/0x220 > [<c0237c20>] ? ext3_get_block+0x0/0x100 > [<c0174c72>] ? generic_file_buffered_write+0xe2/0x200 > [<c0175feb>] ? __generic_file_aio_write+0x26b/0x520 > [<c01a038f>] ? __dentry_open+0x17f/0x230 > [<c01a1187>] ? nameidata_to_filp+0x47/0x50 > [<c01ded70>] ? dquot_file_open+0x0/0x40 > [<c01abc0c>] ? do_last+0x55c/0x5a0 > [<c01762f2>] ? generic_file_aio_write+0x52/0xb0 > [<c01a1915>] ? do_sync_write+0xb5/0xf0 > [<c01a524e>] ? cp_new_stat64+0xee/0x100 > [<c03c1a7c>] ? security_file_permission+0xc/0x10 > [<c01a1aa6>] ? rw_verify_area+0x66/0xd0 > [<c01a20d3>] ? vfs_write+0xa3/0x180 > [<c01ae4c6>] ? do_fcntl+0x396/0x470 > [<c01a1860>] ? do_sync_write+0x0/0xf0 > [<c01a27f1>] ? sys_write+0x41/0x70 > [<c0107b58>] ? sysenter_do_call+0x12/0x28 > Code: e0 0c 29 44 24 08 8b 44 24 08 83 c4 1c 5b 5e 5f 5d c3 8b 74 24 > 14 8b 7c 24 18 23 35 d8 0c 77 c0 23 3d dc 0c 77 c0 e9 7a ff ff ff <0f> > 0b eb fe 90 8d 74 26 00 53 83 ec 08 8b 1d 10 05 80 c0 8b 0d > EIP: [<c0123267>] kmap_atomic_prot+0x137/0x140 SS:ESP 0069:eb4b3ca0 > ---[ end trace 684584b81fd0c02f ]--- > > -- > ShaunR > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@xxxxxxxxxxxxxxxxxxx > http://lists.xensource.com/xen-devel > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.