[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Re; System Crash ...



Hi,

I've just has a rather nasty system crash but it's generated lots of nice logging ..

Anyone know what may have caused it and whether there's anything I can do to stop it happening again ?

(Xen3.1/Stock Ubuntu)

Jan 18 14:00:31 nodec kernel: [  362.689029] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.689044]
Jan 18 14:00:31 nodec kernel: [  362.689045] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.689048]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.689092]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.689108]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689126]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689139]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689171]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.689190]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689211]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.689235]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689259]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689271]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689285]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689300]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689327]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.689352]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689375]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.689391]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689405]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689427]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689440]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689456]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689488]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.689504]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.689516]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.689534]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.689562]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.689587]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689622]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689638]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689672]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.689688]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.689706]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689727]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689741]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689755]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689787]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.689802]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.689832]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.689864]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.689874]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.689881]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.689891]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.689900]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.689906]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.689913]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.689922]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.689934]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.689938]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.689958]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.689971]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.689979]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689991]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.690004]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.690012]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.690020]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.690033]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.690073]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.690089]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.690125]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.690139]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.690150]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.690163]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.690170]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.690178]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.690185]  [system_call+0/139] system_call+0x0/0x8b
                                                                                                                                                                                                69,1          Top
Jan 18 14:00:31 nodec kernel: [  362.690192]
Jan 18 14:00:31 nodec kernel: [  362.690194] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.690196] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.690199] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.690203] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690208] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.690212] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690218] Active:58006 inactive:40628 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.690220]  free:272 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.690230] DMA free:1088kB min:2896kB low:3620kB high:4344kB active:232024kB inactive:162512kB present:525200kB pages_scanned:0 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.690237] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.690242] DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1108kB
Jan 18 14:00:31 nodec kernel: [  362.690265] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.690269] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690272] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690277] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.693502] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.693507] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.693509] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.693511] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.693565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.693570]
Jan 18 14:00:31 nodec kernel: [  362.693571] Call Trace:

Jan 18 14:00:31 nodec kernel: [  362.693576]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.693630]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.693641]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693659]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.693677]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693705]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.693723]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693744]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.693766]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693787]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693801]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693816]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693831]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693858]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.693879]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693898]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.693913]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693923]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693942]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693953]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693966]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693988]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.694000]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.694011]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.694026]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.694054]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.694080]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694101]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694115]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694131]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694165]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.694182]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.694201]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.694224]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694234]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694242]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694257]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.694264]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.694280]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.694300]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.694309]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.694314]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.694323]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.694331]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.694337]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.694343]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.694351]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.694363]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.694367]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.694388]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.694401]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.694408]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.694420]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.694434]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.694442]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.694451]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.694462]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.694477]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.694484]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.694500]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.694522]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.694545]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.694573]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.694586]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.694607]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694623]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694644]
Jan 18 14:00:31 nodec kernel: [  362.694648] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.694652] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.694659] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.694665] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.694674] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.694680] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.694688] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.694691]  free:297 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.694700] DMA free:1188kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.694708] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.694713] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1288kB
Jan 18 14:00:31 nodec kernel: [  362.694737] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.694742] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694752] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.697874] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.697881] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.697883] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.697887] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.698059] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.698064]
Jan 18 14:00:31 nodec kernel: [  362.698065] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.698068]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.698119]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.698144]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.698158]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698177]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698191]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.698209]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.698227]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698251]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.698294]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.698332]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.698348]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.698355]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.698369]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.698383]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.698392]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.698403]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.698414]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.698432]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.698437]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.698486]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.698512]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.698526]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.698551]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.698582]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.698603]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.698621]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.698652]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.698686]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.698705]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.698745]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.698777]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.698797]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.698820]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.698838]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.698857]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698874]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698893]
Jan 18 14:00:31 nodec kernel: [  362.698896] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.698900] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.698905] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.698911] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.698920] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.698927] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.698935] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.698937]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.698948] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.698954] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.698959] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.698983] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.698989] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698992] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698995] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.701567] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.701573] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.701575] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.701577] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.701614] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.701617]
Jan 18 14:00:31 nodec kernel: [  362.701618] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.701620]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.701656]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.701668]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.701676]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701684]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701692]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.701701]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.701709]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701724]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.701743]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.701754]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.701770]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.701781]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.701786]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.701796]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.701804]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.701810]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.701818]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.701826]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.701839]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.701843]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.701863]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.701877]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.701884]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.701896]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.701910]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.701919]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.701927]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.701938]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.701954]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.701962]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.701978]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.701992]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.702002]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.702014]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.702023]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.702030]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702038]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702047]
Jan 18 14:00:31 nodec kernel: [  362.702049] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.702050] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.702055] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.702059] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.702064] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.702067] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.702073] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.702075]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.702080] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.702084] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.702088] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.702106] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.702109] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702112] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702114] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.704303] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.704308] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.704310] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.704312] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.704565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.704569]
Jan 18 14:00:31 nodec kernel: [  362.704570] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.704571]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.704607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.704619]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.704628]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704636]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704642]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.704652]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.704660]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704674]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.704692]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.704712]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.704723]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.704734]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.704744]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.704750]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.704758]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.704767]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.704779]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.704784]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.704805]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.704820]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.704828]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.704840]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.704854]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.704863]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.704871]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.704882]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.704898]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.704906]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.704924]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.704938]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.704949]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.704963]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.704971]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.704980]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704987]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704996]
Jan 18 14:00:31 nodec kernel: [  362.704998] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.705001] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.705004] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.705010] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.705013] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.705018] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.705023] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.705024]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.705030] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.705035] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.705039] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.705053] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.705057] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705059] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705061] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707207] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.707211] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.707212] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.707214] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.707262] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.707265]
Jan 18 14:00:31 nodec kernel: [  362.707266] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.707268]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.707305]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.707313]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707321]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707327]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.707335]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.707343]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707359]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.707377]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.707390]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.707408]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.707417]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.707427]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.707436]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.707442]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.707450]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.707459]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.707471]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.707476]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.707497]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.707510]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.707519]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.707531]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.707544]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.707552]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.707561]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.707572]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.707586]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.707595]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.707612]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.707627]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.707638]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.707650]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.707658]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.707667]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707675]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707685] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.707687] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.707690] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.707696] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.707699] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.707704] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.707709] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.707710]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.707716] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.707721] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.707725] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.707741] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.707744] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707749] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.709971] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.709977] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.709978] 31578 pages shared
Jan 18 14:00:31 nodec kernel: [  362.709982] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.881155] swapper: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.881164]
Jan 18 14:00:31 nodec kernel: [  362.881165] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.881168]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.881228]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.881251]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.881266]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881280]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881295]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.881312]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.881328]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881355]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.881392]  [_end+128027760/2130180472] :e1000:e1000_clean_rx_irq+0x368/0x580
Jan 18 14:00:31 nodec kernel: [  362.881441]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.881463]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.881488]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.881504]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.881519]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.881532]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.881549]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.881578]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.881585]  <EOI>  [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0
Jan 18 14:00:31 nodec kernel: [  362.881639]  [xen_idle+73/144] xen_idle+0x49/0x90
Jan 18 14:00:31 nodec kernel: [  362.881647]  [cpu_idle+120/224] cpu_idle+0x78/0xe0
Jan 18 14:00:31 nodec kernel: [  362.881656]  [start_kernel+693/864] start_kernel+0x2b5/0x360
Jan 18 14:00:31 nodec kernel: [  362.881663]  [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150
Jan 18 14:00:31 nodec kernel: [  362.881673] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.881677] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.881684] CPU    0: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:  60
Jan 18 14:00:31 nodec kernel: [  362.881692] CPU    1: Hot: hi:  186, btch:  31 usd: 161   Cold: hi:   62, btch:  15 usd:  48
Jan 18 14:00:31 nodec kernel: [  362.881699] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  58
Jan 18 14:00:31 nodec kernel: [  362.881705] CPU    3: Hot: hi:  186, btch:  31 usd:  74   Cold: hi:   62, btch:  15 usd:   0
Jan 18 14:00:31 nodec kernel: [  362.881715] Active:54759 inactive:43432 dirty:0 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.881718]  free:267 slab:8169 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.881726] DMA free:1068kB min:2896kB low:3620kB high:4344kB active:219036kB inactive:173728kB present:525200kB pages_scanned:0 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.881733] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.881741] DMA: 19*4kB 6*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1068kB
Jan 18 14:00:31 nodec kernel: [  362.881760] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.881764] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.881766] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.881768] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.885123] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.885130] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.885133] 31592 pages shared
Jan 18 14:00:31 nodec kernel: [  362.885135] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.885149] swapper: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.885155]
Jan 18 14:00:31 nodec kernel: [  362.885156] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.885159]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.885208]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.885231]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.885244]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885260]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885274]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.885292]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.885311]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885337]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.885376]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.885423]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.885442]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.885467]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.885482]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.885496]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.885510]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.885524]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.885554]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.885561]  <EOI>  [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0
Jan 18 14:00:31 nodec kernel: [  362.885615]  [xen_idle+73/144] xen_idle+0x49/0x90
Jan 18 14:00:31 nodec kernel: [  362.885624]  [cpu_idle+120/224] cpu_idle+0x78/0xe0
Jan 18 14:00:31 nodec kernel: [  362.885637]  [start_kernel+693/864] start_kernel+0x2b5/0x360
Jan 18 14:00:31 nodec kernel: [  362.885652]  [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150
etc ...



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.