|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] page allocation failures
HI We are seeing some fairly similar issues involving the network stack and driver at the dom0 and some failures to allocate pages, sometimes it is the swapper, some other timesÂksoftirqd or evenÂrsyslogd In many hosts we have seen this issue, seems to happen shortly after the hosts starts and there is some network heavy load, does not seem to bring the server down or repeat once it is happened. Sadly we cannot encounter at the moment exact steps to reproduce the issue. We are running 3.14.33-1.el6xen.x86_64 and xrn 4.2.5 in centos 6. We have found some old issue atÂhttps://access.redhat.com/solutions/90883Âand that describes the problem, issues look like they have been patched ages ago, but we wonder is there is some kind of regression, unsure how to track the exact patch to check if the kernel we have has it or not. We have been toying with the vm settings but there are mix feelings about changing vm.zone_reclaim_mode http://rhaas.blogspot.co.nz/2014/06/linux-disables-vmzonereclaimmode-by.html http://www.poempelfox.de/blog/2010/03/19/ Documentation about the parameters: https://www.kernel.org/doc/Documentation/sysctl/vm.txt Any of you have you seen this issue before? Regards Juan.- HOST 1#################################################### 2015-04-20T20:59:04+00:00 kernel Â- - Âswapper/0: page allocation failure: order:0, mode:0x120 2015-04-20T20:59:04+00:00 kernel Â- - ÂCPU: 0 PID: 0 Comm: swapper/0 Tainted: GF      Â3.14.33-1.el6xen.x86_64 #1 2015-04-20T20:59:04+00:00 kernel Â- - ÂHardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 4.6.5 02/08/2012 2015-04-20T20:59:04+00:00 kernel Â- - Â0000000000000000 ffff88003c603968 ffffffff8161e672 0000000000000010 2015-04-20T20:59:04+00:00 kernel Â- - Â0000000000000120 ffff88003c6039f8 ffffffff811415db ffff8800020366e2 2015-04-20T20:59:04+00:00 kernel Â- - Âffff88003d2ebb38 00ff880015a0ea68 ffff880027291e58 0000000000000000 2015-04-20T20:59:04+00:00 kernel Â- - ÂCall Trace: 2015-04-20T20:59:04+00:00 kernel Â- - Â<IRQ> Â[<ffffffff8161e672>] dump_stack+0x49/0x5f 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811415db>] warn_alloc_failed+0xeb/0x150 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811447bb>] __alloc_pages_nodemask+0x74b/0xaa0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff816226d6>] ? _raw_spin_unlock_irqrestore+0x16/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810afe83>] ? __wake_up+0x53/0x70 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811897a6>] alloc_pages_current+0xb6/0x170 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa03c9a9a>] xenvif_tx_build_gops+0x59a/0xf40 [xen_netback] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045e678>] ? br_forward_finish+0x58/0x60 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045e588>] ? __br_forward+0x98/0xc0 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045f470>] ? br_handle_frame+0x260/0x260 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045e60d>] ? br_forward+0x5d/0x70 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045f5c5>] ? br_handle_frame_finish+0x155/0x370 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045f3c0>] ? br_handle_frame+0x1b0/0x260 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa045f210>] ? br_handle_local_finish+0x50/0x50 [bridge] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff815586e1>] ? __netif_receive_skb_core+0x271/0x7a0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8100abb0>] ? xen_clocksource_read+0x20/0x30 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8100acaf>] ? xen_pvclock_gtod_notify+0x1f/0xc0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81558ebd>] ? netif_receive_skb_internal+0x2d/0x90 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa03ca49e>] xenvif_tx_action+0x5e/0x730 [xen_netback] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810c0429>] ? handle_irq_event_percpu+0xc9/0x200 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa03cc85c>] xenvif_poll+0x3c/0xec [xen_netback] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81559271>] net_rx_action+0x111/0x2a0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8107253c>] __do_softirq+0xfc/0x2b0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810727fd>] irq_exit+0xbd/0xd0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff813b88e5>] xen_evtchn_do_upcall+0x35/0x50 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8162d0be>] xen_do_hypervisor_callback+0x1e/0x30 2015-04-20T20:59:04+00:00 kernel Â- - Â<EOI> Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8100a5b0>] ? xen_safe_halt+0x10/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8101e844>] ? default_idle+0x24/0xc0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8101e1e6>] ? arch_cpu_idle+0x26/0x30 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810bfa7e>] ? cpu_startup_entry+0xae/0x260 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81613dc7>] ? rest_init+0x77/0x80 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff819332c0>] ? start_kernel+0x43b/0x448 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81932ca3>] ? repair_env_string+0x5b/0x5b 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81932603>] ? x86_64_start_reservations+0x2a/0x2c 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81936688>] ? xen_start_kernel+0x5bd/0x5bf 2015-04-20T20:59:04+00:00 kernel Â- - ÂCPU: 0 PID: 0 Comm: swapper/0 Tainted: GF      Â3.14.33-1.el6xen.x86_64 #1 2015-04-20T20:59:04+00:00 kernel Â- - ÂHardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 4.6.5 02/08/2012 2015-04-20T20:59:04+00:00 kernel Â- - Â0000000000000000 ffff88003c603b08 ffffffff8161e672 0000000000000010 2015-04-20T20:59:04+00:00 kernel Â- - Â0000000000000020 ffff88003c603b98 ffffffff811415db ffffffff816226d6 2015-04-20T20:59:04+00:00 kernel Â- - Âffff88003d2ebb38 01ffffff810afe83 ffffffff815b8196 ffffffff81879a70 2015-04-20T20:59:04+00:00 kernel Â- - ÂCall Trace: 2015-04-20T20:59:04+00:00 kernel Â- - Â<IRQ> Â[<ffffffff8161e672>] dump_stack+0x49/0x5f 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811415db>] warn_alloc_failed+0xeb/0x150 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff816226d6>] ? _raw_spin_unlock_irqrestore+0x16/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff815b8196>] ? tcp_v4_rcv+0x696/0x700 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811447bb>] __alloc_pages_nodemask+0x74b/0xaa0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810c0429>] ? handle_irq_event_percpu+0xc9/0x200 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff815bbe32>] ? tcp4_gro_receive+0xf2/0x140 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff811897a6>] alloc_pages_current+0xb6/0x170 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81545d67>] __netdev_alloc_frag+0x97/0x150 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8154a496>] __netdev_alloc_skb+0x96/0x120 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81559b31>] ? napi_gro_receive+0x31/0x100 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffffa00edf4d>] igb_poll+0x3dd/0x1040 [igb] 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810c0429>] ? handle_irq_event_percpu+0xc9/0x200 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81559271>] net_rx_action+0x111/0x2a0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8107253c>] __do_softirq+0xfc/0x2b0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810727fd>] irq_exit+0xbd/0xd0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff813b88e5>] xen_evtchn_do_upcall+0x35/0x50 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8162d0be>] xen_do_hypervisor_callback+0x1e/0x30 2015-04-20T20:59:04+00:00 kernel Â- - Â<EOI> Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8100a5b0>] ? xen_safe_halt+0x10/0x20 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8101e844>] ? default_idle+0x24/0xc0 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff8101e1e6>] ? arch_cpu_idle+0x26/0x30 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff810bfa7e>] ? cpu_startup_entry+0xae/0x260 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81613dc7>] ? rest_init+0x77/0x80 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff819332c0>] ? start_kernel+0x43b/0x448 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81932ca3>] ? repair_env_string+0x5b/0x5b 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81932603>] ? x86_64_start_reservations+0x2a/0x2c 2015-04-20T20:59:04+00:00 kernel Â- - Â[<ffffffff81936688>] ? xen_start_kernel+0x5bd/0x5bf HOST 2#################################################### 2015-03-06T01:12:42.134210+00:00 kernel Â- - ksoftirqd/0: page allocation failure: order:0, mode:0x20 2015-03-06T01:12:42.134542+00:00 kernel Â- - CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: GF      Â3.14.33-1.el6xen.x86_64 #1 2015-03-06T01:12:42.134549+00:00 kernel Â- - Hardware name: Supermicro X8DT3/X8DT3, BIOS 2.1   03/17/2012 2015-03-06T01:12:42.135167+00:00 kernel Â- - Â0000000000000000 ffff88005c2d99e8 ffffffff8161e672 0000000000000010 2015-03-06T01:12:42.135176+00:00 kernel Â- - Â0000000000000020 ffff88005c2d9a78 ffffffff811415db ffff88005a690080 2015-03-06T01:12:42.135181+00:00 kernel Â- - Âffff88007d45ab38 01ff88005c2d9a98 ffffffff815b8196 ffff88005c2d9af0 2015-03-06T01:12:42.135184+00:00 kernel Â- - Call Trace: 2015-03-06T01:12:42.135191+00:00 kernel Â- - Â[<ffffffff8161e672>] dump_stack+0x49/0x5f 2015-03-06T01:12:42.135195+00:00 kernel Â- - Â[<ffffffff811415db>] warn_alloc_failed+0xeb/0x150 2015-03-06T01:12:42.135200+00:00 kernel Â- - Â[<ffffffff815b8196>] ? tcp_v4_rcv+0x696/0x700 2015-03-06T01:12:42.135214+00:00 kernel Â- - Â[<ffffffff811447bb>] __alloc_pages_nodemask+0x74b/0xaa0 2015-03-06T01:12:42.135220+00:00 kernel Â- - Â[<ffffffff81592ae8>] ? ip_local_deliver_finish+0xa8/0x230 2015-03-06T01:12:42.135225+00:00 kernel Â- - Â[<ffffffff815bbe32>] ? tcp4_gro_receive+0xf2/0x140 2015-03-06T01:12:42.135229+00:00 kernel Â- - Â[<ffffffff811897a6>] alloc_pages_current+0xb6/0x170 2015-03-06T01:12:42.135234+00:00 kernel Â- - Â[<ffffffff81545d67>] __netdev_alloc_frag+0x97/0x150 2015-03-06T01:12:42.135238+00:00 kernel Â- - Â[<ffffffff8154a496>] __netdev_alloc_skb+0x96/0x120 2015-03-06T01:12:42.135242+00:00 kernel Â- - Â[<ffffffff81559b31>] ? napi_gro_receive+0x31/0x100 2015-03-06T01:12:42.135246+00:00 kernel Â- - Â[<ffffffffa00b3f4d>] igb_poll+0x3dd/0x1040 [igb] 2015-03-06T01:12:42.135462+00:00 kernel Â- - Â[<ffffffff81004f20>] ? xen_mc_flush+0xb0/0x1b0 2015-03-06T01:12:42.135472+00:00 kernel Â- - Â[<ffffffff81559271>] net_rx_action+0x111/0x2a0 2015-03-06T01:12:42.135476+00:00 kernel Â- - Â[<ffffffff8107253c>] __do_softirq+0xfc/0x2b0 2015-03-06T01:12:42.135480+00:00 kernel Â- - Â[<ffffffff81072728>] run_ksoftirqd+0x38/0x50 2015-03-06T01:12:42.135484+00:00 kernel Â- - Â[<ffffffff8109766d>] smpboot_thread_fn+0xfd/0x190 2015-03-06T01:12:42.135519+00:00 kernel Â- - swapper/0: page allocation failure: order:0, mode:0x20 2015-03-06T01:12:42.135523+00:00 kernel Â- - CPU: 0 PID: 0 Comm: swapper/0 Tainted: GF      Â3.14.33-1.el6xen.x86_64 #1 2015-03-06T01:12:42.135527+00:00 kernel Â- - Hardware name: Supermicro X8DT3/X8DT3, BIOS 2.1   03/17/2012 2015-03-06T01:12:42.135531+00:00 kernel Â- - Â0000000000000000 ffff88007c803b08 ffffffff8161e672 0000000000000010 2015-03-06T01:12:42.135534+00:00 kernel Â- - Â0000000000000020 ffff88007c803b98 ffffffff811415db 0000000025194c00 2015-03-06T01:12:42.135734+00:00 kernel Â- - Âffff88007d45ab38 01ff88007c803b38 ffffffff81085e94 ffff88007c803b78 2015-03-06T01:12:42.135742+00:00 kernel Â- - Call Trace: 2015-03-06T01:12:42.135747+00:00 kernel Â- - Â<IRQ> Â[<ffffffff8161e672>] dump_stack+0x49/0x5f 2015-03-06T01:12:42.135752+00:00 kernel Â- - Â[<ffffffff811415db>] warn_alloc_failed+0xeb/0x150 2015-03-06T01:12:42.135756+00:00 kernel Â- - Â[<ffffffff81085e94>] ? wake_up_worker+0x24/0x30 2015-03-06T01:12:42.135760+00:00 kernel Â- - Â[<ffffffff811447bb>] __alloc_pages_nodemask+0x74b/0xaa0 2015-03-06T01:12:42.135764+00:00 kernel Â- - Â[<ffffffff8108832a>] ? queue_work_on+0x3a/0x50 2015-03-06T01:12:42.135768+00:00 kernel Â- - Â[<ffffffff813f9b5e>] ? credit_entropy_bits+0x22e/0x2e0 2015-03-06T01:12:42.135783+00:00 kernel Â- - Â[<ffffffff811897a6>] alloc_pages_current+0xb6/0x170 2015-03-06T01:12:42.135789+00:00 kernel Â- - Â[<ffffffff81545d67>] __netdev_alloc_frag+0x97/0x150 2015-03-06T01:12:42.135793+00:00 kernel Â- - Â[<ffffffff8154a496>] __netdev_alloc_skb+0x96/0x120 2015-03-06T01:12:42.135798+00:00 kernel Â- - Â[<ffffffffa00b3f4d>] igb_poll+0x3dd/0x1040 [igb] 2015-03-06T01:12:42.135802+00:00 kernel Â- - Â[<ffffffff810c0429>] ? handle_irq_event_percpu+0xc9/0x200 2015-03-06T01:12:42.135804+00:00 kernel Â- - Â[<ffffffff81559271>] net_rx_action+0x111/0x2a0 2015-03-06T01:12:42.135807+00:00 kernel Â- - Â[<ffffffff8107253c>] __do_softirq+0xfc/0x2b0 2015-03-06T01:12:42.135817+00:00 kernel Â- - Â[<ffffffff810727fd>] irq_exit+0xbd/0xd0 2015-03-06T01:12:42.135820+00:00 kernel Â- - Â[<ffffffff813b88e5>] xen_evtchn_do_upcall+0x35/0x50 2015-03-06T01:12:42.135823+00:00 kernel Â- - Â[<ffffffff8162d0be>] xen_do_hypervisor_callback+0x1e/0x30 2015-03-06T01:12:42.135826+00:00 kernel Â- - Â<EOI> Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-03-06T01:12:42.135828+00:00 kernel Â- - Â[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 2015-03-06T01:12:42.135831+00:00 kernel Â- - Â[<ffffffff8100a5b0>] ? xen_safe_halt+0x10/0x20 2015-03-06T01:12:42.135833+00:00 kernel Â- - Â[<ffffffff8101e844>] ? default_idle+0x24/0xc0 2015-03-06T01:12:42.135836+00:00 kernel Â- - Â[<ffffffff8101e1e6>] ? arch_cpu_idle+0x26/0x30 2015-03-06T01:12:42.135844+00:00 kernel Â- - Â[<ffffffff810bfa7e>] ? cpu_startup_entry+0xae/0x260 2015-03-06T01:12:42.135848+00:00 kernel Â- - Â[<ffffffff81613dc7>] ? rest_init+0x77/0x80 2015-03-06T01:12:42.135850+00:00 kernel Â- - Â[<ffffffff819332c0>] ? start_kernel+0x43b/0x448 2015-03-06T01:12:42.135853+00:00 kernel Â- - Â[<ffffffff81932ca3>] ? repair_env_string+0x5b/0x5b 2015-03-06T01:12:42.135856+00:00 kernel Â- - Â[<ffffffff81932603>] ? x86_64_start_reservations+0x2a/0x2c 2015-03-06T01:12:42.135860+00:00 kernel Â- - Â[<ffffffff81936688>] ? xen_start_kernel+0x5bd/0x5bf 2015-03-06T01:12:42.135488+00:00 kernel Â- - Â[<ffffffff81097570>] ? smpboot_create_threads+0x80/0x80 2015-03-06T01:12:42.135492+00:00 kernel Â- - Â[<ffffffff8109047e>] kthread+0xce/0xf0 2015-03-06T01:12:42.135496+00:00 kernel Â- - Â[<ffffffff810903b0>] ? kthread_freezable_should_stop+0x70/0x70 2015-03-06T01:12:42.135510+00:00 kernel Â- - Â[<ffffffff8162b6fc>] ret_from_fork+0x7c/0xb0 2015-03-06T01:12:42.135514+00:00 kernel Â- - Â[<ffffffff810903b0>] ? kthread_freezable_should_stop+0x70/0x70 _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |