Xen project Mailing List

2010/1/31 Dana Rawding <dana@xxxxxxxxxxx>

Hi all,

I've been experiencing a rash of CPU lockups on a number of domU's recently. It's been happening on two different servers. About a year ago I had this problem every once in a while but it was not frequent. I was running Ubuntu with Xen 3.1 and 2.6.24-18 back then. I'm now running Xen 3.3 and 2.6.24-26.

What I have noticed is that just prior to the lockups the domU's had high cpu loads. The domU that I have the most problems with is a Zimbra server. My guess is that a rash of spam comes through and cpu loads get high, then the cpu's lock up. Originally I had it running with 1 cpu but have since upped it 2 then 3 cpu's.

I have been collecting the lockup messages and have posed a few below. Any ideas? Recommendations?

Thanks,
Dana

[138077.172283] =======================
[138075.147398] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:97]
[138075.147411]
[138075.147419] Pid: 97, comm: kswapd0 Tainted: G D (2.6.24-26-xen #1)
[138075.147426] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 0
[138075.147441] EIP is at _spin_lock+0x7/0x10
[138075.147447] EAX: c1da48ec EBX: 00000000 ECX: 220c7000 EDX: 00000000
[138075.147453] ESI: 8b804067 EDI: c1da48ec EBP: 00000f28 ESP: ed707dec
[138075.147459] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[138075.147471] CR0: 8005003b CR2: 080f0010 CR3: 2213b000 CR4: 00000660
[138075.147482] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[138075.147488] DR6: ffff0ff0 DR7: 00000400
[138075.147495] [<c01773cb>] page_check_address+0x1cb/0x3c0
[138075.147514] [<c0119868>] xen_invlpg_mask+0x38/0x40
[138075.147529] [<c017762e>] page_referenced_one+0x6e/0x190
[138075.147541] [<c017875c>] page_referenced+0xec/0x130
[138075.147552] [<c01671cf>] shrink_active_list+0x18f/0x5c0
[138075.147567] [<c016826d>] shrink_zone+0xdd/0x100
[138075.147578] [<c01688cc>] kswapd+0x44c/0x490
[138075.147589] [<c013bb00>] autoremove_wake_function+0x0/0x40
[138075.147603] [<c011e270>] complete+0x40/0x60
[138075.147614] [<c0168480>] kswapd+0x0/0x490
[138075.147625] [<c013b842>] kthread+0x42/0x70
[138075.147635] [<c013b800>] kthread+0x0/0x70
[138075.147646] [<c0105bb7>] kernel_thread_helper+0x7/0x10
[138075.147658] =======================
[138088.987826] BUG: soft lockup - CPU#1 stuck for 11s! [java:23215]
[138088.987841]
[138088.987846] Pid: 23215, comm: java Tainted: G D (2.6.24-26-xen #1)
[138088.987850] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 1
[138088.987862] EIP is at _spin_lock+0x7/0x10
[138088.987866] EAX: c1da48ec EBX: 00000000 ECX: c1da48e0 EDX: 00000ca8
[138088.987870] ESI: 8b804067 EDI: 00000000 EBP: e20c7ca8 ESP: e226be04
[138088.987873] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[138088.987883] CR0: 80050033 CR2: 940ef020 CR3: 2211f000 CR4: 00000660
[138088.987891] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[138088.987896] DR6: ffff0ff0 DR7: 00000400
[138088.987901] [<c016d88d>] unmap_vmas+0x43d/0xae0
[138088.987922] [<c011959c>] kmap_atomic+0x1c/0x30
[138088.987941] [<c01192fd>] kunmap_atomic+0x3d/0x60
[138088.987957] [<c0173ee8>] vma_adjust+0x1c8/0x440
[138088.987967] [<c0173765>] unmap_region+0x95/0x120
[138088.987975] [<c0174387>] do_munmap+0x147/0x1f0
[138088.987983] [<c0174c90>] mmap_region+0x70/0x450
[138088.987991] [<c01db3b7>] security_file_mmap+0x27/0x30
[138088.988001] [<c0175472>] do_mmap_pgoff+0x312/0x330
[138088.988008] [<c010a02b>] sys_mmap2+0xbb/0xd0
[138088.988016] [<c0105832>] syscall_call+0x7/0xb
[138088.988023] [<c0320000>] svc_accept+0x150/0x410
[138088.988032] =======================

[66916.451144] BUG: soft lockup - CPU#0 stuck for 11s! [java:2758]
[66928.193453] BUG: soft lockup - CPU#1 stuck for 11s! [java:3419]

[336990.703192] BUG: soft lockup - CPU#1 stuck for 11s! [ps:32586]
[336990.703206]
[336990.703214] Pid: 32586, comm: ps Tainted: G D (2.6.24-26-xen #1)
[336990.703221] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 1
[336990.703235] EIP is at _spin_lock+0x7/0x10
[336990.703241] EAX: c1dbc72c EBX: 00000000 ECX: c1dbc720 EDX: 00000007
[336990.703247] ESI: 57b51067 EDI: 00000001 EBP: e2cb93c8 ESP: e2033e4c
[336990.703253] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[336990.703266] CR0: 80050033 CR2: 08079004 CR3: 23651000 CR4: 00000660
[336990.703275] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[336990.703282] DR6: ffff0ff0 DR7: 00000400
[336990.703288] [<c0171646>] handle_mm_fault+0xae6/0x1360
[336990.703307] [<c020e057>] rb_insert_color+0x77/0xe0
[336990.703325] [<c032a27e>] do_page_fault+0x35e/0xe70
[336990.703337] [<c01745d4>] vma_merge+0x144/0x1d0
[336990.703349] [<c0174b75>] do_brk+0x195/0x240
[336990.703362] [<c0175126>] sys_brk+0xb6/0xf0
[336990.703374] [<c0329f20>] do_page_fault+0x0/0xe70
[336990.703384] [<c0328bc5>] error_code+0x35/0x40
[336990.703396] =======================
[337005.938292] BUG: soft lockup - CPU#2 stuck for 11s! [zmlocalconfig:11371]
[337005.938306]
[337005.938312] Pid: 11371, comm: zmlocalconfig Tainted: G D (2.6.24-26-xen #1)
[337005.938318] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 2
[337005.938330] EIP is at _spin_lock+0x7/0x10
[337005.938335] EAX: ec64a870 EBX: ec64a870 ECX: 00000002 EDX: ec64a871
[337005.938339] ESI: 00000000 EDI: c03fe800 EBP: c1261e38 ESP: c1261d7c
[337005.938343] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[337005.938357] CR0: 8005003b CR2: 08128000 CR3: 25d8e000 CR4: 00000660
[337005.938364] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[337005.938370] DR6: ffff0ff0 DR7: 00000400
[337005.938376] [<c01771f0>] page_lock_anon_vma+0x20/0x30
[337005.938391] [<c01786fd>] page_referenced+0x8d/0x130
[337005.938401] [<c01671cf>] shrink_active_list+0x18f/0x5c0
[337005.938411] [<c0164286>] get_dirty_limits+0x16/0x200
[337005.938421] [<ee04b38e>] mb_cache_shrink_fn+0x1e/0x100 [mbcache]
[337005.938435] [<c016826d>] shrink_zone+0xdd/0x100
[337005.938444] [<c0168d72>] try_to_free_pages+0x152/0x250
[337005.938453] [<c0162fcb>] __alloc_pages+0x14b/0x390
[337005.938463] [<c01855c5>] do_sync_read+0xd5/0x120
[337005.938475] [<c0163247>] __get_free_pages+0x37/0x50
[337005.938483] [<c0124496>] copy_process+0xa6/0x1210
[337005.938493] [<c0197c34>] d_alloc+0x114/0x1a0
[337005.938503] [<c0125830>] do_fork+0x40/0x260
[337005.938511] [<c0210f00>] copy_to_user+0x30/0x60
[337005.938523] [<c0103226>] sys_clone+0x36/0x40
[337005.938530] [<c0105832>] syscall_call+0x7/0xb
[337005.938542] =======================
[336990.803889] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:103]
[336990.803907]
[336990.803915] Pid: 103, comm: kswapd0 Tainted: G D (2.6.24-26-xen #1)
[336990.803922] EIP: 0061:[<c03286ea>] EFLAGS: 00000286 CPU: 0
[336990.803940] EIP is at _spin_lock+0xa/0x10
[336990.803948] EAX: c1dbc86c EBX: 00000000 ECX: 22cc3000 EDX: 00000000
[336990.803955] ESI: 57b47067 EDI: c1dbc86c EBP: 00000ff0 ESP: ed725dec
[336990.803961] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[336990.803976] CR0: 8005003b CR2: b791e6d9 CR3: 23e3b000 CR4: 00000660
[336990.803986] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[336990.803992] DR6: ffff0ff0 DR7: 00000400
[336990.804001] [<c01773cb>] page_check_address+0x1cb/0x3c0
[336990.804026] [<c017762e>] page_referenced_one+0x6e/0x190
[336990.804039] [<c017875c>] page_referenced+0xec/0x130
[336990.804049] [<c01671cf>] shrink_active_list+0x18f/0x5c0
[336990.804064] [<c0210556>] memmove+0x36/0x40
[336990.804079] [<c0164286>] get_dirty_limits+0x16/0x200
[336990.804089] [<c0139857>] call_rcu+0x97/0xa0
[336990.804102] [<ee04b38e>] mb_cache_shrink_fn+0x1e/0x100 [mbcache]
[336990.804120] [<c016826d>] shrink_zone+0xdd/0x100
[336990.804132] [<c01688cc>] kswapd+0x44c/0x490
[336990.804145] [<c013bb00>] autoremove_wake_function+0x0/0x40
[336990.804160] [<c011e270>] complete+0x40/0x60
[336990.804172] [<c0168480>] kswapd+0x0/0x490
[336990.804183] [<c013b842>] kthread+0x42/0x70
[336990.804194] [<c013b800>] kthread+0x0/0x70
[336990.804206] [<c0105bb7>] kernel_thread_helper+0x7/0x10
[336990.804218] =======================

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

Re: [Xen-users] BUG: soft lockup