|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] Re; System Crash ...
Hi, I've just has a rather nasty system crash but it's generated lots of nice logging .. Anyone know what may have caused it and whether there's anything I can do to stop it happening again ? (Xen3.1/Stock Ubuntu) Jan 18 14:00:31 nodec kernel: [ 362.689029] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.689044] Jan 18 14:00:31 nodec kernel: [ 362.689045] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.689048] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.689092] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.689108] [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160 Jan 18 14:00:31 nodec kernel: [ 362.689126] [__kmalloc+207/224] __kmalloc+0xcf/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.689139] [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160 Jan 18 14:00:31 nodec kernel: [ 362.689171] [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110 Jan 18 14:00:31 nodec kernel: [ 362.689190] [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.689211] [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240 Jan 18 14:00:31 nodec kernel: [ 362.689235] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.689259] [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.689271] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.689285] [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.689300] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.689327] [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60 Jan 18 14:00:31 nodec kernel: [ 362.689352] [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190 Jan 18 14:00:31 nodec kernel: [ 362.689375] [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0 Jan 18 14:00:31 nodec kernel: [ 362.689391] [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190 Jan 18 14:00:31 nodec kernel: [ 362.689405] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.689427] [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60 Jan 18 14:00:31 nodec kernel: [ 362.689440] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.689456] [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60 Jan 18 14:00:31 nodec kernel: [ 362.689488] [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90 Jan 18 14:00:31 nodec kernel: [ 362.689504] [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90 Jan 18 14:00:31 nodec kernel: [ 362.689516] [skb_clone+50/784] skb_clone+0x32/0x310 Jan 18 14:00:31 nodec kernel: [ 362.689534] [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100 Jan 18 14:00:31 nodec kernel: [ 362.689562] [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130 Jan 18 14:00:31 nodec kernel: [ 362.689587] [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.689607] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.689622] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.689638] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.689672] [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800 Jan 18 14:00:31 nodec kernel: [ 362.689688] [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380 Jan 18 14:00:31 nodec kernel: [ 362.689706] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.689727] [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130 Jan 18 14:00:31 nodec kernel: [ 362.689741] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.689755] [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130 Jan 18 14:00:31 nodec kernel: [ 362.689787] [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0 Jan 18 14:00:31 nodec kernel: [ 362.689802] [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470 Jan 18 14:00:31 nodec kernel: [ 362.689832] [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580 Jan 18 14:00:31 nodec kernel: [ 362.689864] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.689874] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.689881] [__wake_up+67/112] __wake_up+0x43/0x70 Jan 18 14:00:31 nodec kernel: [ 362.689891] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.689900] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.689906] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.689913] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.689922] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.689934] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.689938] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.689958] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.689971] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.689979] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.689991] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.690004] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.690012] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.690020] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.690033] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.690073] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.690089] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.690125] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.690139] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.690150] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.690163] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.690170] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.690178] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.690185] [system_call+0/139] system_call+0x0/0x8b 69,1 Top Jan 18 14:00:31 nodec kernel: [ 362.690192] Jan 18 14:00:31 nodec kernel: [ 362.690194] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.690196] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.690199] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.690203] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.690208] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.690212] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.690218] Active:58006 inactive:40628 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.690220] free:272 slab:7704 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.690230] DMA free:1088kB min:2896kB low:3620kB high:4344kB active:232024kB inactive:162512kB present:525200kB pages_scanned:0 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.690237] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.690242] DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1108kB Jan 18 14:00:31 nodec kernel: [ 362.690265] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.690269] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.690272] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.690277] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.693502] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.693507] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.693509] 31576 pages shared Jan 18 14:00:31 nodec kernel: [ 362.693511] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.693565] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.693570] Jan 18 14:00:31 nodec kernel: [ 362.693571] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.693576] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.693630] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.693641] [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160 Jan 18 14:00:31 nodec kernel: [ 362.693659] [__kmalloc+207/224] __kmalloc+0xcf/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.693677] [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160 Jan 18 14:00:31 nodec kernel: [ 362.693705] [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110 Jan 18 14:00:31 nodec kernel: [ 362.693723] [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.693744] [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240 Jan 18 14:00:31 nodec kernel: [ 362.693766] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.693787] [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.693801] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.693816] [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0 Jan 18 14:00:31 nodec kernel: [ 362.693831] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.693858] [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60 Jan 18 14:00:31 nodec kernel: [ 362.693879] [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190 Jan 18 14:00:31 nodec kernel: [ 362.693898] [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0 Jan 18 14:00:31 nodec kernel: [ 362.693913] [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190 Jan 18 14:00:31 nodec kernel: [ 362.693923] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.693942] [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60 Jan 18 14:00:31 nodec kernel: [ 362.693953] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.693966] [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60 Jan 18 14:00:31 nodec kernel: [ 362.693988] [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90 Jan 18 14:00:31 nodec kernel: [ 362.694000] [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90 Jan 18 14:00:31 nodec kernel: [ 362.694011] [skb_clone+50/784] skb_clone+0x32/0x310 Jan 18 14:00:31 nodec kernel: [ 362.694026] [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100 Jan 18 14:00:31 nodec kernel: [ 362.694054] [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130 Jan 18 14:00:31 nodec kernel: [ 362.694080] [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.694101] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.694115] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.694131] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.694165] [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800 Jan 18 14:00:31 nodec kernel: [ 362.694182] [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380 Jan 18 14:00:31 nodec kernel: [ 362.694201] [nf_iterate+112/160] nf_iterate+0x70/0xa0 Jan 18 14:00:31 nodec kernel: [ 362.694224] [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130 Jan 18 14:00:31 nodec kernel: [ 362.694234] [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.694242] [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130 Jan 18 14:00:31 nodec kernel: [ 362.694257] [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0 Jan 18 14:00:31 nodec kernel: [ 362.694264] [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470 Jan 18 14:00:31 nodec kernel: [ 362.694280] [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580 Jan 18 14:00:31 nodec kernel: [ 362.694300] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.694309] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.694314] [__wake_up+67/112] __wake_up+0x43/0x70 Jan 18 14:00:31 nodec kernel: [ 362.694323] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.694331] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.694337] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.694343] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.694351] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.694363] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.694367] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.694388] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.694401] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.694408] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.694420] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.694434] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.694442] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.694451] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.694462] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.694477] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.694484] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.694500] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.694522] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.694545] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.694573] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.694586] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.694607] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.694623] [system_call+0/139] system_call+0x0/0x8b Jan 18 14:00:31 nodec kernel: [ 362.694644] Jan 18 14:00:31 nodec kernel: [ 362.694648] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.694652] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.694659] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.694665] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 57 Jan 18 14:00:31 nodec kernel: [ 362.694674] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.694680] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.694688] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.694691] free:297 slab:7704 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.694700] DMA free:1188kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.694708] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.694713] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1288kB Jan 18 14:00:31 nodec kernel: [ 362.694737] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.694742] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.694747] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.694752] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.697874] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.697881] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.697883] 31576 pages shared Jan 18 14:00:31 nodec kernel: [ 362.697887] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.698059] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.698064] Jan 18 14:00:31 nodec kernel: [ 362.698065] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.698068] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.698119] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.698144] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.698158] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.698177] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.698191] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.698209] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.698227] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.698251] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.698294] [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580 Jan 18 14:00:31 nodec kernel: [ 362.698332] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.698348] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.698355] [__wake_up+67/112] __wake_up+0x43/0x70 Jan 18 14:00:31 nodec kernel: [ 362.698369] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.698383] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.698392] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.698403] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.698414] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.698432] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.698437] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.698486] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.698512] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.698526] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.698551] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.698582] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.698603] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.698621] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.698652] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.698686] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.698705] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.698745] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.698777] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.698797] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.698820] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.698838] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.698857] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.698874] [system_call+0/139] system_call+0x0/0x8b Jan 18 14:00:31 nodec kernel: [ 362.698893] Jan 18 14:00:31 nodec kernel: [ 362.698896] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.698900] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.698905] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.698911] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 57 Jan 18 14:00:31 nodec kernel: [ 362.698920] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.698927] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.698935] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.698937] free:265 slab:7736 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.698948] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.698954] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.698959] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB Jan 18 14:00:31 nodec kernel: [ 362.698983] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.698989] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.698992] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.698995] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.701567] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.701573] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.701575] 31576 pages shared Jan 18 14:00:31 nodec kernel: [ 362.701577] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.701614] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.701617] Jan 18 14:00:31 nodec kernel: [ 362.701618] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.701620] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.701656] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.701668] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.701676] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.701684] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.701692] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.701701] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.701709] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.701724] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.701743] [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580 Jan 18 14:00:31 nodec kernel: [ 362.701754] [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0 Jan 18 14:00:31 nodec kernel: [ 362.701770] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.701781] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.701786] [__wake_up+67/112] __wake_up+0x43/0x70 Jan 18 14:00:31 nodec kernel: [ 362.701796] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.701804] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.701810] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.701818] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.701826] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.701839] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.701843] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.701863] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.701877] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.701884] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.701896] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.701910] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.701919] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.701927] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.701938] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.701954] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.701962] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.701978] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.701992] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.702002] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.702014] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.702023] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.702030] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.702038] [system_call+0/139] system_call+0x0/0x8b Jan 18 14:00:31 nodec kernel: [ 362.702047] Jan 18 14:00:31 nodec kernel: [ 362.702049] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.702050] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.702055] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.702059] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 57 Jan 18 14:00:31 nodec kernel: [ 362.702064] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.702067] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.702073] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.702075] free:265 slab:7736 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.702080] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.702084] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.702088] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB Jan 18 14:00:31 nodec kernel: [ 362.702106] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.702109] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.702112] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.702114] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.704303] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.704308] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.704310] 31576 pages shared Jan 18 14:00:31 nodec kernel: [ 362.704312] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.704565] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.704569] Jan 18 14:00:31 nodec kernel: [ 362.704570] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.704571] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.704607] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.704619] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.704628] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.704636] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.704642] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.704652] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.704660] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.704674] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.704692] [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580 Jan 18 14:00:31 nodec kernel: [ 362.704712] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.704723] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.704734] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.704744] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.704750] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.704758] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.704767] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.704779] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.704784] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.704805] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.704820] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.704828] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.704840] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.704854] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.704863] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.704871] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.704882] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.704898] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.704906] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.704924] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.704938] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.704949] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.704963] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.704971] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.704980] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.704987] [system_call+0/139] system_call+0x0/0x8b Jan 18 14:00:31 nodec kernel: [ 362.704996] Jan 18 14:00:31 nodec kernel: [ 362.704998] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.705001] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.705004] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.705010] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 57 Jan 18 14:00:31 nodec kernel: [ 362.705013] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.705018] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.705023] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.705024] free:265 slab:7736 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.705030] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.705035] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.705039] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB Jan 18 14:00:31 nodec kernel: [ 362.705053] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.705057] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.705059] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.705061] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.707207] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.707211] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.707212] 31576 pages shared Jan 18 14:00:31 nodec kernel: [ 362.707214] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.707262] glusterfs: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.707265] Jan 18 14:00:31 nodec kernel: [ 362.707266] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.707268] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.707305] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.707313] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.707321] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.707327] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.707335] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.707343] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.707359] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.707377] [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580 Jan 18 14:00:31 nodec kernel: [ 362.707390] [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0 Jan 18 14:00:31 nodec kernel: [ 362.707408] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.707417] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.707427] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.707436] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.707442] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.707450] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.707459] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.707471] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.707476] <EOI> [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10 Jan 18 14:00:31 nodec kernel: [ 362.707497] [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0 Jan 18 14:00:31 nodec kernel: [ 362.707510] [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140 Jan 18 14:00:31 nodec kernel: [ 362.707519] [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.707531] [error_exit+0/121] error_exit+0x0/0x79 Jan 18 14:00:31 nodec kernel: [ 362.707544] [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140 Jan 18 14:00:31 nodec kernel: [ 362.707552] [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70 Jan 18 14:00:31 nodec kernel: [ 362.707561] [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280 Jan 18 14:00:31 nodec kernel: [ 362.707572] [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0 Jan 18 14:00:31 nodec kernel: [ 362.707586] [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50 Jan 18 14:00:31 nodec kernel: [ 362.707595] [sock_aio_read+372/384] sock_aio_read+0x174/0x180 Jan 18 14:00:31 nodec kernel: [ 362.707612] [do_sync_read+217/288] do_sync_read+0xd9/0x120 Jan 18 14:00:31 nodec kernel: [ 362.707627] [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 18 14:00:31 nodec kernel: [ 362.707638] [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850 Jan 18 14:00:31 nodec kernel: [ 362.707650] [vfs_read+382/400] vfs_read+0x17e/0x190 Jan 18 14:00:31 nodec kernel: [ 362.707658] [sys_read+83/144] sys_read+0x53/0x90 Jan 18 14:00:31 nodec kernel: [ 362.707667] [system_call+134/139] system_call+0x86/0x8b Jan 18 14:00:31 nodec kernel: [ 362.707675] [system_call+0/139] system_call+0x0/0x8b Jan 18 14:00:31 nodec kernel: [ 362.707682] Jan 18 14:00:31 nodec kernel: [ 362.707682] Jan 18 14:00:31 nodec kernel: [ 362.707685] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.707687] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.707690] CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 52 Jan 18 14:00:31 nodec kernel: [ 362.707696] CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 57 Jan 18 14:00:31 nodec kernel: [ 362.707699] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 49 Jan 18 14:00:31 nodec kernel: [ 362.707704] CPU 3: Hot: hi: 186, btch: 31 usd: 97 Cold: hi: 62, btch: 15 usd: 9 Jan 18 14:00:31 nodec kernel: [ 362.707709] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.707710] free:265 slab:7736 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.707716] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.707721] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.707725] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB Jan 18 14:00:31 nodec kernel: [ 362.707741] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.707744] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.707747] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.707749] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.709971] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.709977] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.709978] 31578 pages shared Jan 18 14:00:31 nodec kernel: [ 362.709982] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.881155] swapper: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.881164] Jan 18 14:00:31 nodec kernel: [ 362.881165] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.881168] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.881228] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.881251] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.881266] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.881280] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.881295] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.881312] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.881328] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.881355] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.881392] [_end+128027760/2130180472] :e1000:e1000_clean_rx_irq+0x368/0x580 Jan 18 14:00:31 nodec kernel: [ 362.881441] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.881463] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.881488] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.881504] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.881519] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.881532] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.881549] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.881578] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.881585] <EOI> [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.881639] [xen_idle+73/144] xen_idle+0x49/0x90 Jan 18 14:00:31 nodec kernel: [ 362.881647] [cpu_idle+120/224] cpu_idle+0x78/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.881656] [start_kernel+693/864] start_kernel+0x2b5/0x360 Jan 18 14:00:31 nodec kernel: [ 362.881663] [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150 Jan 18 14:00:31 nodec kernel: [ 362.881673] Mem-info: Jan 18 14:00:31 nodec kernel: [ 362.881677] DMA per-cpu: Jan 18 14:00:31 nodec kernel: [ 362.881684] CPU 0: Hot: hi: 186, btch: 31 usd: 22 Cold: hi: 62, btch: 15 usd: 60 Jan 18 14:00:31 nodec kernel: [ 362.881692] CPU 1: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 48 Jan 18 14:00:31 nodec kernel: [ 362.881699] CPU 2: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 58 Jan 18 14:00:31 nodec kernel: [ 362.881705] CPU 3: Hot: hi: 186, btch: 31 usd: 74 Cold: hi: 62, btch: 15 usd: 0 Jan 18 14:00:31 nodec kernel: [ 362.881715] Active:54759 inactive:43432 dirty:0 writeback:0 unstable:0 Jan 18 14:00:31 nodec kernel: [ 362.881718] free:267 slab:8169 mapped:2557 pagetables:675 bounce:0 Jan 18 14:00:31 nodec kernel: [ 362.881726] DMA free:1068kB min:2896kB low:3620kB high:4344kB active:219036kB inactive:173728kB present:525200kB pages_scanned:0 all_unreclaimable? no Jan 18 14:00:31 nodec kernel: [ 362.881733] lowmem_reserve[]: 0 0 0 Jan 18 14:00:31 nodec kernel: [ 362.881741] DMA: 19*4kB 6*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1068kB Jan 18 14:00:31 nodec kernel: [ 362.881760] Swap cache: add 0, delete 0, find 0/0, race 0+0 Jan 18 14:00:31 nodec kernel: [ 362.881764] Free swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.881766] Total swap = 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.881768] Free swap: 2097144kB Jan 18 14:00:31 nodec kernel: [ 362.885123] 133120 pages of RAM Jan 18 14:00:31 nodec kernel: [ 362.885130] 20456 reserved pages Jan 18 14:00:31 nodec kernel: [ 362.885133] 31592 pages shared Jan 18 14:00:31 nodec kernel: [ 362.885135] 0 pages swap cached Jan 18 14:00:31 nodec kernel: [ 362.885149] swapper: page allocation failure. order:3, mode:0x4020 Jan 18 14:00:31 nodec kernel: [ 362.885155] Jan 18 14:00:31 nodec kernel: [ 362.885156] Call Trace: Jan 18 14:00:31 nodec kernel: [ 362.885159] <IRQ> [__alloc_pages+806/848] __alloc_pages+0x326/0x350 Jan 18 14:00:31 nodec kernel: [ 362.885208] [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0 Jan 18 14:00:31 nodec kernel: [ 362.885231] [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0 Jan 18 14:00:31 nodec kernel: [ 362.885244] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.885260] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.885274] [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0 Jan 18 14:00:31 nodec kernel: [ 362.885292] [__alloc_skb+111/352] __alloc_skb+0x6f/0x160 Jan 18 14:00:31 nodec kernel: [ 362.885311] [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40 Jan 18 14:00:31 nodec kernel: [ 362.885337] [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0 Jan 18 14:00:31 nodec kernel: [ 362.885376] [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580 Jan 18 14:00:31 nodec kernel: [ 362.885423] [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320 Jan 18 14:00:31 nodec kernel: [ 362.885442] [net_rx_action+250/688] net_rx_action+0xfa/0x2b0 Jan 18 14:00:31 nodec kernel: [ 362.885467] [__do_softirq+138/288] __do_softirq+0x8a/0x120 Jan 18 14:00:31 nodec kernel: [ 362.885482] [call_softirq+28/48] call_softirq+0x1c/0x30 Jan 18 14:00:31 nodec kernel: [ 362.885496] [do_softirq+165/256] do_softirq+0xa5/0x100 Jan 18 14:00:31 nodec kernel: [ 362.885510] [do_IRQ+98/192] do_IRQ+0x62/0xc0 Jan 18 14:00:31 nodec kernel: [ 362.885524] [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150 Jan 18 14:00:31 nodec kernel: [ 362.885554] [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30 Jan 18 14:00:31 nodec kernel: [ 362.885561] <EOI> [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.885615] [xen_idle+73/144] xen_idle+0x49/0x90 Jan 18 14:00:31 nodec kernel: [ 362.885624] [cpu_idle+120/224] cpu_idle+0x78/0xe0 Jan 18 14:00:31 nodec kernel: [ 362.885637] [start_kernel+693/864] start_kernel+0x2b5/0x360 Jan 18 14:00:31 nodec kernel: [ 362.885652] [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150 etc ... _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |