Xen project Mailing List

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

To: Pim van Riezen <pi+lists@xxxxxxxxxxxx>

From: Pasi Kärkkäinen <pasik@xxxxxx>

Date: Fri, 9 Apr 2010 19:48:43 +0300

Delivery-date: Fri, 09 Apr 2010 09:50:40 -0700

List-id: Xen user discussion <xen-users.lists.xensource.com>

On Fri, Apr 09, 2010 at 04:34:08PM +0200, Pim van Riezen wrote: > > On Apr 9, 2010, at 16:21 , Pasi Kärkkäinen wrote: > > > Can you please post the soft lockup messages again.. > > Here's a metric truckload of them. The lockups don't seem to happen > consistently in the same code path. It may also be worth mentioning that it > only seems to happen with guests that have more than 1 vcpu, but I'm not sure > of that distinction. > I guess you already posted these to xen-devel ? -- Pasi > Apr 9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s! > [gzip:27658] > Apr 9 08:21:01 host kernel: Modules linked in: > Apr 9 08:21:01 host kernel: CPU 1: > Apr 9 08:21:01 host kernel: Modules linked in: > Apr 9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:21:01 host kernel: RIP: e033:[<0000000000401958>] > [<0000000000401958>] 0x401958 > Apr 9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38 EFLAGS: 00000297 > Apr 9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002 RCX: > 0000000000657ce8 > Apr 9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002 RDI: > 000000000000da48 > Apr 9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034 R09: > 000000000000da48 > Apr 9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080 R12: > 0000000000658c06 > Apr 9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06 R15: > 0000000000000080 > Apr 9 08:21:01 host kernel: FS: 00007fa6088946e0(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 08:21:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000 CR4: > 0000000000002660 > Apr 9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:21:02 host kernel: Call Trace: > Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s! > [swapper:0] > Apr 9 08:29:39 host kernel: Modules linked in: > Apr 9 08:29:39 host kernel: CPU 1: > Apr 9 08:29:39 host kernel: Modules linked in: > Apr 9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 08:29:39 host kernel: FS: 00007fa913f42720(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 08:29:39 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000 CR4: > 0000000000002660 > Apr 9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:29:39 host kernel: Call Trace: > Apr 9 08:29:39 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 08:29:39 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 08:29:39 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s! > [swapper:0] > Apr 9 08:29:39 host kernel: Modules linked in: > Apr 9 08:29:39 host kernel: CPU 0: > Apr 9 08:29:40 host kernel: Modules linked in: > Apr 9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 08:29:40 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 08:29:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000 CR4: > 0000000000002660 > Apr 9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:29:40 host kernel: Call Trace: > Apr 9 08:29:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 08:29:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 08:29:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 08:29:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! > [ssh:2324] > Apr 9 08:34:34 host kernel: Modules linked in: > Apr 9 08:34:34 host kernel: CPU 1: > Apr 9 08:34:34 host kernel: Modules linked in: > Apr 9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>] > [<00007f1b0fcac76a>] 0x7f1b0fcac76a > Apr 9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480 EFLAGS: 00010202 > Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: > 0000000000000181 > Apr 9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013 RDI: > 000000000000000a > Apr 9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004 R09: > 0101010101010101 > Apr 9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690 R12: > 0000000000000010 > Apr 9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001 R15: > 0000000000000004 > Apr 9 08:34:34 host kernel: FS: 00007f1b10528710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000 CR4: > 0000000000002660 > Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:34:34 host kernel: Call Trace: > Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s! > [swapper:0] > Apr 9 08:34:34 host kernel: Modules linked in: > Apr 9 08:34:34 host kernel: CPU 0: > Apr 9 08:34:34 host kernel: Modules linked in: > Apr 9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 08:34:34 host kernel: FS: 00007fa913f42720(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000 CR4: > 0000000000002660 > Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:34:34 host kernel: Call Trace: > Apr 9 08:34:34 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 08:34:34 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 08:34:34 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 08:34:34 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s! > [gzip:30581] > Apr 9 08:39:49 host kernel: Modules linked in: > Apr 9 08:39:49 host kernel: CPU 1: > Apr 9 08:39:49 host kernel: Modules linked in: > Apr 9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>] > [<ffffffff811ede38>] fuse_request_send+0x60/0x22b > Apr 9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28 EFLAGS: 00000202 > Apr 9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0 RCX: > fffffffffffdfb5e > Apr 9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d RDI: > ffff88007eb1f710 > Apr 9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40 R09: > ffff8800593170d8 > Apr 9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f R12: > ffff88007eb1f400 > Apr 9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000 R15: > 0000000000001000 > Apr 9 08:39:49 host kernel: FS: 00007f7b12f606e0(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000 CR4: > 0000000000002660 > Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:39:49 host kernel: Call Trace: > Apr 9 08:39:49 host kernel: [<ffffffff811ede34>] ? > fuse_request_send+0x5c/0x22b > Apr 9 08:39:49 host kernel: [<ffffffff811f1655>] ? > fuse_wait_on_page_writeback+0x31/0x168 > Apr 9 08:39:49 host kernel: [<ffffffff810904b4>] ? > add_to_page_cache_lru+0x41/0x58 > Apr 9 08:39:49 host kernel: [<ffffffff811f186a>] ? fuse_send_write+0xb8/0xc2 > Apr 9 08:39:49 host kernel: [<ffffffff811f29d7>] ? > fuse_file_aio_write+0x497/0x62f > Apr 9 08:39:49 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113 > Apr 9 08:39:49 host kernel: [<ffffffff8100f6af>] ? __switch_to+0x1ad/0x297 > Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? > autoremove_wake_function+0x0/0x2e > Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? > finish_task_switch+0x27/0x6e > Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109 > Apr 9 08:39:49 host kernel: [<ffffffff810bb4b2>] ? vfs_write+0xad/0x149 > Apr 9 08:39:49 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e > Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? > system_call_fastpath+0x16/0x1b > Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s! > [sshfs:2332] > Apr 9 08:39:49 host kernel: Modules linked in: > Apr 9 08:39:49 host kernel: CPU 0: > Apr 9 08:39:49 host kernel: Modules linked in: > Apr 9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted > 2.6.32.9xls-domU #2 > Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>] > [<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d > Apr 9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0 EFLAGS: 00000206 > Apr 9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080 RCX: > 0000000101b312ce > Apr 9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0 RDI: > ffff88007d1dc1c0 > Apr 9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f R09: > 0000000101b312ce > Apr 9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a R12: > ffff880068b8f1c0 > Apr 9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020 R15: > ffff880068b8f1f8 > Apr 9 08:39:49 host kernel: FS: 00007f1b10528710(0063) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000 CR4: > 0000000000002660 > Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 08:39:49 host kernel: Call Trace: > Apr 9 08:39:49 host kernel: <IRQ> [<ffffffff81419140>] ? > tcp_rcv_established+0x587/0x87b > Apr 9 08:39:49 host kernel: [<ffffffff8141f7f1>] ? tcp_v4_do_rcv+0x2c/0x1c9 > Apr 9 08:39:49 host kernel: [<ffffffff81046156>] ? local_bh_enable+0x22/0x8f > Apr 9 08:39:49 host kernel: [<ffffffff8141fe05>] ? tcp_v4_rcv+0x477/0x6f8 > Apr 9 08:39:49 host kernel: [<ffffffff814064f7>] ? > ip_local_deliver_finish+0xf2/0x1c7 > Apr 9 08:39:49 host kernel: [<ffffffff81406126>] ? ip_rcv_finish+0x326/0x340 > Apr 9 08:39:49 host kernel: [<ffffffff814063c0>] ? ip_rcv+0x280/0x2c5 > Apr 9 08:39:49 host kernel: [<ffffffff8135d39d>] ? xennet_poll+0xb06/0xc68 > Apr 9 08:39:49 host kernel: [<ffffffff8100e39f>] ? > xen_restore_fl_direct_end+0x0/0x1 > Apr 9 08:39:49 host kernel: [<ffffffff81334afc>] ? > do_blkif_request+0x341/0x350 > Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff813c469b>] ? net_rx_action+0x77/0x179 > Apr 9 08:39:49 host kernel: [<ffffffff810462b4>] ? __do_softirq+0xdd/0x1a1 > Apr 9 08:39:49 host kernel: [<ffffffff81011bec>] ? call_softirq+0x1c/0x30 > Apr 9 08:39:49 host kernel: [<ffffffff81012edf>] ? do_softirq+0x3f/0x7c > Apr 9 08:39:49 host kernel: [<ffffffff812ff975>] ? > xen_evtchn_do_upcall+0x173/0x190 > Apr 9 08:39:49 host kernel: [<ffffffff81011c3e>] ? > xen_do_hypervisor_callback+0x1e/0x30 > Apr 9 08:39:49 host kernel: <EOI> [<ffffffff8100e3b2>] ? > check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? > hypercall_page+0x22a/0x1001 > Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? > hypercall_page+0x22a/0x1001 > Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff8100e359>] ? > xen_irq_enable_direct_end+0x0/0x7 > Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? > finish_task_switch+0x27/0x6e > Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109 > Apr 9 08:39:49 host kernel: [<ffffffff8103d336>] ? > default_wake_function+0x0/0x9 > Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? > cap_socket_recvmsg+0x0/0x3 > Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? > cap_socket_recvmsg+0x0/0x3 > Apr 9 08:39:49 host kernel: [<ffffffff814d6e59>] ? > schedule_timeout+0x23/0x1d9 > Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? > cap_socket_recvmsg+0x0/0x3 > Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? > _spin_unlock_irqrestore+0xc/0xd > Apr 9 08:39:49 host kernel: [<ffffffff81456ebe>] ? > unix_stream_recvmsg+0x2c4/0x644 > Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? > autoremove_wake_function+0x0/0x2e > Apr 9 08:39:49 host kernel: [<ffffffff81457231>] ? > unix_stream_recvmsg+0x637/0x644 > Apr 9 08:39:49 host kernel: [<ffffffff813b6303>] ? sock_aio_read+0x103/0x117 > Apr 9 08:39:49 host kernel: [<ffffffff810bac91>] ? do_sync_read+0xce/0x113 > Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? > xen_hypervisor_callback+0x1b/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? > autoremove_wake_function+0x0/0x2e > Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? > cap_socket_recvmsg+0x0/0x3 > Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? > _spin_unlock_irqrestore+0xc/0xd > Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? > xen_hypervisor_callback+0x1b/0x20 > Apr 9 08:39:49 host kernel: [<ffffffff810bb6f0>] ? vfs_read+0xbd/0x146 > Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? > hypercall_page+0x22a/0x1001 > Apr 9 08:39:49 host kernel: [<ffffffff810bb835>] ? sys_read+0x45/0x6e > Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? > system_call_fastpath+0x16/0x1b > Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s! > [swapper:0] > Apr 9 09:11:36 host kernel: Modules linked in: > Apr 9 09:11:36 host kernel: CPU 1: > Apr 9 09:11:36 host kernel: Modules linked in: > Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 09:11:36 host kernel: FS: 00007f44bc51f710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 09:11:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:11:36 host kernel: Call Trace: > Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s! > [swapper:0] > Apr 9 09:11:36 host kernel: Modules linked in: > Apr 9 09:11:36 host kernel: CPU 0: > Apr 9 09:11:36 host kernel: Modules linked in: > Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 09:11:36 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 09:11:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:11:36 host kernel: Call Trace: > Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 09:11:36 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! > [ssh:2324] > Apr 9 09:20:24 host kernel: Modules linked in: > Apr 9 09:20:24 host kernel: CPU 1: > Apr 9 09:20:24 host kernel: Modules linked in: > Apr 9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>] > [<00007f1b0e259f20>] 0x7f1b0e259f20 > Apr 9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338 EFLAGS: 00000206 > Apr 9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0 RCX: > 00000000000000a0 > Apr 9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004 RDI: > 00007f1b0e539c30 > Apr 9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001 R09: > 0000000000000001 > Apr 9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0 R12: > 0000000000000000 > Apr 9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111 R15: > 0000000000000071 > Apr 9 09:20:24 host kernel: FS: 00007f1b10528710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000 CR4: > 0000000000002660 > Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:20:24 host kernel: Call Trace: > Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s! > [swapper:0] > Apr 9 09:20:24 host kernel: Modules linked in: > Apr 9 09:20:24 host kernel: CPU 0: > Apr 9 09:20:24 host kernel: Modules linked in: > Apr 9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 09:20:24 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000 CR4: > 0000000000002660 > Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:20:24 host kernel: Call Trace: > Apr 9 09:20:24 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 09:20:24 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 09:20:24 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 09:20:24 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s! > [gzip:4496] > Apr 9 09:24:36 host kernel: Modules linked in: > Apr 9 09:24:36 host kernel: CPU 1: > Apr 9 09:24:36 host kernel: Modules linked in: > Apr 9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:24:36 host kernel: RIP: e033:[<0000000000409804>] > [<0000000000409804>] 0x409804 > Apr 9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8 EFLAGS: 00000202 > Apr 9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000 RCX: > 00007f529598c610 > Apr 9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6 RDI: > 00000000006534fb > Apr 9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000 R09: > 7464697720223333 > Apr 9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246 R12: > 00000000006522a0 > Apr 9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 09:24:36 host kernel: FS: 00007f5295e2a6e0(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 09:24:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000 CR4: > 0000000000002660 > Apr 9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:24:37 host kernel: Call Trace: > Apr 9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s! > [swapper:0] > Apr 9 09:24:37 host kernel: Modules linked in: > Apr 9 09:24:37 host kernel: CPU 0: > Apr 9 09:24:37 host kernel: Modules linked in: > Apr 9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 09:24:38 host kernel: FS: 00007f2f6a31a710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 09:24:38 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000 CR4: > 0000000000002660 > Apr 9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 09:24:38 host kernel: Call Trace: > Apr 9 09:24:38 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 09:24:38 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 09:24:38 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 09:24:38 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s! > [swapper:0] > Apr 9 10:05:51 host kernel: Modules linked in: > Apr 9 10:05:51 host kernel: CPU 1: > Apr 9 10:05:51 host kernel: Modules linked in: > Apr 9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 10:05:51 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:05:51 host kernel: Call Trace: > Apr 9 10:05:51 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 10:05:51 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 10:05:51 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s! > [httpd:9922] > Apr 9 10:05:51 host kernel: Modules linked in: > Apr 9 10:05:51 host kernel: CPU 0: > Apr 9 10:05:51 host kernel: Modules linked in: > Apr 9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>] > [<00007ff39a44a36d>] 0x7ff39a44a36d > Apr 9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118 EFLAGS: 00000a86 > Apr 9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60 RCX: > c0c176e403831574 > Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: > 000000000420e43b > Apr 9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004 R09: > 00007fff4b9ad270 > Apr 9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188 R12: > 000000000320e6d0 > Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368 R15: > 0000000000000000 > Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 10:05:51 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000 CR4: > 0000000000002660 > Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:05:51 host kernel: Call Trace: > Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s! > [swapper:0] > Apr 9 10:50:04 host kernel: Modules linked in: > Apr 9 10:50:04 host kernel: CPU 1: > Apr 9 10:50:04 host kernel: Modules linked in: > Apr 9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 10:50:04 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 10:50:04 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000 CR4: > 0000000000002660 > Apr 9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:50:04 host kernel: Call Trace: > Apr 9 10:50:04 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 10:50:04 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 10:50:04 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s! > [httpd:16797] > Apr 9 10:50:04 host kernel: Modules linked in: > Apr 9 10:50:04 host kernel: CPU 0: > Apr 9 10:50:04 host kernel: Modules linked in: > Apr 9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>] > [<00007ff39a49ae55>] 0x7ff39a49ae55 > Apr 9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8 EFLAGS: 00000246 > Apr 9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0 RCX: > 222879726575713e > Apr 9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c RDI: > 00007fff4b9b1d00 > Apr 9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001 R09: > 0101010101010101 > Apr 9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be R12: > 00007fff4b9b1d00 > Apr 9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0 R15: > 0000000003695df0 > Apr 9 10:50:05 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 10:50:05 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000 CR4: > 0000000000002660 > Apr 9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:50:05 host kernel: Call Trace: > Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s! > [exim:17977] > Apr 9 10:57:01 host kernel: Modules linked in: > Apr 9 10:57:01 host kernel: CPU 1: > Apr 9 10:57:01 host kernel: Modules linked in: > Apr 9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>] > [<ffffffff8100922a>] hypercall_page+0x22a/0x1001 > Apr 9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0 EFLAGS: 00000246 > Apr 9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: > ffffffff8100922a > Apr 9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000 RDI: > 0000000000000000 > Apr 9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710 R09: > 0000000000000003 > Apr 9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246 R12: > 0000000000000004 > Apr 9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0 R15: > ffff88005a51b3c0 > Apr 9 10:57:01 host kernel: FS: 00007fd75c048710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000 CR4: > 0000000000002660 > Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:57:01 host kernel: Call Trace: > Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 10:57:01 host kernel: [<ffffffff8100e359>] ? > xen_irq_enable_direct_end+0x0/0x7 > Apr 9 10:57:01 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d > Apr 9 10:57:01 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30 > Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s! > [netstat:17966] > Apr 9 10:57:01 host kernel: Modules linked in: > Apr 9 10:57:01 host kernel: CPU 0: > Apr 9 10:57:01 host kernel: Modules linked in: > Apr 9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted > 2.6.32.9xls-domU #2 > Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>] > [<ffffffff814d8226>] _spin_lock+0xa/0x15 > Apr 9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0 EFLAGS: 00000202 > Apr 9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180 RCX: > 0000000000000012 > Apr 9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8 RDI: > ffff88007ff77188 > Apr 9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000 R09: > 0000000000300000 > Apr 9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e R12: > ffff88000ff859c0 > Apr 9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38 R15: > 0000000000000002 > Apr 9 10:57:01 host kernel: FS: 00007fb435545710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000 CR4: > 0000000000002660 > Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 10:57:01 host kernel: Call Trace: > Apr 9 10:57:01 host kernel: [<ffffffff810cb01b>] ? __d_lookup+0x7a/0x105 > Apr 9 10:57:01 host kernel: [<ffffffff810c251e>] ? do_lookup+0x2a/0x1b3 > Apr 9 10:57:01 host kernel: [<ffffffff810c4498>] ? > __link_path_walk+0x348/0xdcd > Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 10:57:01 host kernel: [<ffffffff810c514b>] ? path_walk+0x66/0xca > Apr 9 10:57:01 host kernel: [<ffffffff810c51cf>] ? do_path_lookup+0x20/0x77 > Apr 9 10:57:01 host kernel: [<ffffffff810c5ae9>] ? user_path_at+0x48/0x79 > Apr 9 10:57:01 host kernel: [<ffffffff8100e39f>] ? > xen_restore_fl_direct_end+0x0/0x1 > Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? > mntput_no_expire+0x23/0xed > Apr 9 10:57:01 host kernel: [<ffffffff8129b703>] ? > _atomic_dec_and_lock+0x33/0x50 > Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? > mntput_no_expire+0x23/0xed > Apr 9 10:57:01 host kernel: [<ffffffff810bdf4d>] ? sys_readlinkat+0x25/0x8d > Apr 9 10:57:01 host kernel: [<ffffffff81010ac2>] ? > system_call_fastpath+0x16/0x1b > Apr 9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s! > [swapper:0] > Apr 9 11:00:36 host kernel: Modules linked in: > Apr 9 11:00:36 host kernel: CPU 1: > Apr 9 11:00:36 host kernel: Modules linked in: > Apr 9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 11:00:36 host kernel: FS: 00007f719f5fa6e0(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 11:00:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000 CR4: > 0000000000002660 > Apr 9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 11:00:37 host kernel: Call Trace: > Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s! > [swapper:0] > Apr 9 11:00:37 host kernel: Modules linked in: > Apr 9 11:00:37 host kernel: CPU 0: > Apr 9 11:00:37 host kernel: Modules linked in: > Apr 9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 11:00:37 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 11:00:37 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000 CR4: > 0000000000002660 > Apr 9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 11:00:37 host kernel: Call Trace: > Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 11:00:37 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s! > [httpd:19360] > Apr 9 11:08:50 host kernel: Modules linked in: > Apr 9 11:08:50 host kernel: CPU 0: > Apr 9 11:08:50 host kernel: Modules linked in: > Apr 9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>] > [<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc > Apr 9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80 EFLAGS: 00000246 > Apr 9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8 RCX: > ffff88005675dce4 > Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057 RDI: > ffff88002678c900 > Apr 9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4 R09: > 75646f72705f6d76 > Apr 9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683 R12: > 0000000000000057 > Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80 R15: > ffff88005675dd60 > Apr 9 11:08:50 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 11:08:50 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000 CR4: > 0000000000002660 > Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 11:08:50 host kernel: Call Trace: > Apr 9 11:08:50 host kernel: [<ffffffff814573c8>] ? > unix_stream_sendmsg+0x18a/0x31f > Apr 9 11:08:50 host kernel: [<ffffffff813b61ec>] ? sock_aio_write+0xf6/0x10a > Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 11:08:50 host kernel: [<ffffffff8100e39f>] ? > xen_restore_fl_direct_end+0x0/0x1 > Apr 9 11:08:50 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113 > Apr 9 11:08:50 host kernel: [<ffffffff810c5af3>] ? user_path_at+0x52/0x79 > Apr 9 11:08:50 host kernel: [<ffffffff810be0b0>] ? cp_new_stat+0xe9/0xfc > Apr 9 11:08:50 host kernel: [<ffffffff810561b8>] ? > autoremove_wake_function+0x0/0x2e > Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 11:08:50 host kernel: [<ffffffff810bb4c5>] ? vfs_write+0xc0/0x149 > Apr 9 11:08:50 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e > Apr 9 11:08:50 host kernel: [<ffffffff81010ac2>] ? > system_call_fastpath+0x16/0x1b > Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s! > [swapper:0] > Apr 9 11:08:50 host kernel: Modules linked in: > Apr 9 11:08:50 host kernel: CPU 1: > Apr 9 11:08:50 host kernel: Modules linked in: > Apr 9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 11:08:50 host kernel: FS: 00007f0e429cb6e0(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 11:08:50 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 11:08:50 host kernel: Call Trace: > Apr 9 11:08:50 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 11:08:50 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 11:08:50 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s! > [httpd:26903] > Apr 9 12:00:40 host kernel: Modules linked in: > Apr 9 12:00:40 host kernel: CPU 1: > Apr 9 12:00:40 host kernel: Modules linked in: > Apr 9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>] > [<00007ff39a427cd2>] 0x7ff39a427cd2 > Apr 9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20 EFLAGS: 00000202 > Apr 9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30 RCX: > 0000000000000013 > Apr 9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50 RDI: > 0000000003bd6418 > Apr 9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60 R09: > 00007ff39a783d80 > Apr 9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: > 00007ff39aaaa3e0 > Apr 9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50 R15: > 000000000000015e > Apr 9 12:00:40 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000 CR4: > 0000000000002660 > Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:00:40 host kernel: Call Trace: > Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s! > [swapper:0] > Apr 9 12:00:40 host kernel: Modules linked in: > Apr 9 12:00:40 host kernel: CPU 0: > Apr 9 12:00:40 host kernel: Modules linked in: > Apr 9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 12:00:40 host kernel: FS: 00007f74fde746e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:00:40 host kernel: Call Trace: > Apr 9 12:00:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 12:00:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 12:00:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 12:00:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s! > [swapper:0] > Apr 9 12:19:45 host kernel: Modules linked in: > Apr 9 12:19:45 host kernel: CPU 1: > Apr 9 12:19:45 host kernel: Modules linked in: > Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 12:19:45 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000 CR4: > 0000000000002660 > Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:19:45 host kernel: Call Trace: > Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s! > [swapper:0] > Apr 9 12:19:45 host kernel: Modules linked in: > Apr 9 12:19:45 host kernel: CPU 0: > Apr 9 12:19:45 host kernel: Modules linked in: > Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 12:19:45 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:19:45 host kernel: Call Trace: > Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 12:19:45 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s! > [swapper:0] > Apr 9 12:37:26 host kernel: Modules linked in: > Apr 9 12:37:26 host kernel: CPU 1: > Apr 9 12:37:26 host kernel: Modules linked in: > Apr 9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 12:37:26 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:37:26 host kernel: Call Trace: > Apr 9 12:37:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 12:37:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 12:37:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s! > [httpd:31684] > Apr 9 12:37:26 host kernel: Modules linked in: > Apr 9 12:37:26 host kernel: CPU 0: > Apr 9 12:37:26 host kernel: Modules linked in: > Apr 9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>] > [<00007ff39a41f7d3>] 0x7ff39a41f7d3 > Apr 9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330 EFLAGS: 00000246 > Apr 9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0 RCX: > 0000000000000001 > Apr 9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e RDI: > 0000000000000000 > Apr 9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60 R09: > 0000000000000003 > Apr 9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: > 0000000003a2b297 > Apr 9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0 R15: > 00007ff39a783e80 > Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 12:37:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000 CR4: > 0000000000002660 > Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 12:37:26 host kernel: Call Trace: > Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s! > [swapper:0] > Apr 9 13:44:52 host kernel: Modules linked in: > Apr 9 13:44:52 host kernel: CPU 1: > Apr 9 13:44:52 host kernel: Modules linked in: > Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 13:44:52 host kernel: FS: 00007fd75c048710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 13:44:52 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 13:44:52 host kernel: Call Trace: > Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s! > [swapper:0] > Apr 9 13:44:52 host kernel: Modules linked in: > Apr 9 13:44:52 host kernel: CPU 0: > Apr 9 13:44:52 host kernel: Modules linked in: > Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 13:44:52 host kernel: FS: 00007f0e429cb6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 13:44:52 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000 CR4: > 0000000000002660 > Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 13:44:52 host kernel: Call Trace: > Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 13:44:52 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s! > [swapper:0] > Apr 9 13:55:43 host kernel: Modules linked in: > Apr 9 13:55:43 host kernel: CPU 1: > Apr 9 13:55:43 host kernel: Modules linked in: > Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 13:55:43 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000 CR4: > 0000000000002660 > Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 13:55:43 host kernel: Call Trace: > Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s! > [swapper:0] > Apr 9 13:55:43 host kernel: Modules linked in: > Apr 9 13:55:43 host kernel: CPU 0: > Apr 9 13:55:43 host kernel: Modules linked in: > Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 13:55:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 13:55:43 host kernel: Call Trace: > Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 13:55:43 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s! > [swapper:0] > Apr 9 14:16:21 host kernel: Modules linked in: > Apr 9 14:16:21 host kernel: CPU 1: > Apr 9 14:16:21 host kernel: Modules linked in: > Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>] > [<ffffffff8100922a>] hypercall_page+0x22a/0x1001 > Apr 9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8 EFLAGS: 00000246 > Apr 9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810 RCX: > ffffffff8100922a > Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000000 > Apr 9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000 R09: > ffffffff8100e3b2 > Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000 R15: > 0000000000001000 > Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 14:16:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 14:16:21 host kernel: Call Trace: > Apr 9 14:16:21 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 14:16:21 host kernel: [<ffffffff8100e359>] ? > xen_irq_enable_direct_end+0x0/0x7 > Apr 9 14:16:21 host kernel: [<ffffffff8103b070>] ? > finish_task_switch+0x27/0x6e > Apr 9 14:16:21 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109 > Apr 9 14:16:21 host kernel: [<ffffffff814d87ba>] ? error_exit+0x2a/0x60 > Apr 9 14:16:21 host kernel: [<ffffffff8101149d>] ? > retint_restore_args+0x5/0x6 > Apr 9 14:16:21 host kernel: [<ffffffff8100fe5f>] ? cpu_idle+0x80/0x82 > Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s! > [swapper:0] > Apr 9 14:16:21 host kernel: Modules linked in: > Apr 9 14:16:21 host kernel: CPU 0: > Apr 9 14:16:21 host kernel: Modules linked in: > Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 14:16:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: > 0000000000002660 > Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 14:16:21 host kernel: Call Trace: > Apr 9 14:16:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 14:16:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 14:16:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 14:16:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s! > [swapper:0] > Apr 9 15:40:21 host kernel: Modules linked in: > Apr 9 15:40:21 host kernel: CPU 1: > Apr 9 15:40:21 host kernel: Modules linked in: > Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 15:40:21 host kernel: FS: 00007fd75c048710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 15:40:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 15:40:21 host kernel: Call Trace: > Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s! > [swapper:0] > Apr 9 15:40:21 host kernel: Modules linked in: > Apr 9 15:40:21 host kernel: CPU 0: > Apr 9 15:40:21 host kernel: Modules linked in: > Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 15:40:21 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 15:40:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 15:40:21 host kernel: Call Trace: > Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 15:40:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s! > [swapper:0] > Apr 9 16:06:21 host kernel: Modules linked in: > Apr 9 16:06:21 host kernel: CPU 1: > Apr 9 16:06:21 host kernel: Modules linked in: > Apr 9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 16:06:21 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 16:06:25 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:06:26 host kernel: Call Trace: > Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s! > [swapper:0] > Apr 9 16:06:26 host kernel: Modules linked in: > Apr 9 16:06:26 host kernel: CPU 0: > Apr 9 16:06:26 host kernel: Modules linked in: > Apr 9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 16:06:26 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 16:06:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000 CR4: > 0000000000002660 > Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:06:26 host kernel: Call Trace: > Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:06:26 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s! > [httpd:32342] > Apr 9 16:17:43 host kernel: Modules linked in: > Apr 9 16:17:43 host kernel: CPU 1: > Apr 9 16:17:43 host kernel: Modules linked in: > Apr 9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>] > [<ffffffff8100922a>] hypercall_page+0x22a/0x1001 > Apr 9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0 EFLAGS: 00000246 > Apr 9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: > ffffffff8100922a > Apr 9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000 RDI: > 0000000000000000 > Apr 9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240 R09: > 000000310f6de059 > Apr 9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000006 > Apr 9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0 R15: > ffff88007d093b40 > Apr 9 16:17:43 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 16:17:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000 CR4: > 0000000000002660 > Apr 9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:17:43 host kernel: Call Trace: > Apr 9 16:17:43 host kernel: [<ffffffff8100dd39>] ? > xen_force_evtchn_callback+0x9/0xa > Apr 9 16:17:43 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20 > Apr 9 16:17:43 host kernel: [<ffffffff8100e359>] ? > xen_irq_enable_direct_end+0x0/0x7 > Apr 9 16:17:43 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d > Apr 9 16:17:43 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30 > Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s! > [swapper:0] > Apr 9 16:17:43 host kernel: Modules linked in: > Apr 9 16:17:43 host kernel: CPU 0: > Apr 9 16:17:43 host kernel: Modules linked in: > Apr 9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 16:17:44 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 16:17:44 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000 CR4: > 0000000000002660 > Apr 9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:17:44 host kernel: Call Trace: > Apr 9 16:17:44 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:17:44 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:17:44 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:17:44 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s! > [swapper:0] > Apr 9 16:20:22 host kernel: Modules linked in: > Apr 9 16:20:22 host kernel: CPU 1: > Apr 9 16:20:22 host kernel: Modules linked in: > Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 16:20:22 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 16:20:22 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:20:22 host kernel: Call Trace: > Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s! > [swapper:0] > Apr 9 16:20:22 host kernel: Modules linked in: > Apr 9 16:20:22 host kernel: CPU 0: > Apr 9 16:20:22 host kernel: Modules linked in: > Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246 > Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: > ffffffff810093aa > Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > ffffffffffffffff > Apr 9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: > 0000000000000000 > Apr 9 16:20:22 host kernel: FS: 00007f07704fe6e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 16:20:22 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000 CR4: > 0000000000002660 > Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:20:22 host kernel: Call Trace: > Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:20:22 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f > Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s! > [swapper:0] > Apr 9 16:22:29 host kernel: Modules linked in: > Apr 9 16:22:29 host kernel: CPU 1: > Apr 9 16:22:29 host kernel: Modules linked in: > Apr 9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>] > [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001 > Apr 9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246 > Apr 9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: > ffffffff810093aa > Apr 9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: > 0000000000000001 > Apr 9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: > ffffffff8100e3b2 > Apr 9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: > 0000000000000000 > Apr 9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > Apr 9 16:22:29 host kernel: FS: 00007ff39e495710(0000) > GS:ffff8800022c7000(0000) knlGS:0000000000000000 > Apr 9 16:22:29 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b > Apr 9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000 CR4: > 0000000000002660 > Apr 9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:22:29 host kernel: Call Trace: > Apr 9 16:22:29 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15 > Apr 9 16:22:29 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40 > Apr 9 16:22:29 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82 > Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s! > [syslogd:1848] > Apr 9 16:22:29 host kernel: Modules linked in: > Apr 9 16:22:29 host kernel: CPU 0: > Apr 9 16:22:29 host kernel: Modules linked in: > Apr 9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted > 2.6.32.9xls-domU #2 > Apr 9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>] > [<00007f74fd9e4ad5>] 0x7f74fd9e4ad5 > Apr 9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0 EFLAGS: 00000246 > Apr 9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000 RCX: > ffffffff810092eb > Apr 9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0 RDI: > 0000000000000000 > Apr 9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000 R09: > 0000000000000000 > Apr 9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246 R12: > 00007fff22fa01b0 > Apr 9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120 R15: > 00007f74fde74690 > Apr 9 16:22:30 host kernel: FS: 00007f74fde746e0(0000) > GS:ffff8800022ac000(0000) knlGS:0000000000000000 > Apr 9 16:22:30 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > Apr 9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000 CR4: > 0000000000002660 > Apr 9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > Apr 9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > Apr 9 16:22:30 host kernel: Call Trace: > _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.