|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [Xen-users] unexpected Out Of Memory (OOM)
On Wed, 2013-08-07 at 02:02 +0200, Olivier Bonvalet wrote: > Hi, > > I have a memory problem on several PV domU, I *suppose* since the > upgrade to 3.10.4 kernel (and still occur with 3.10.5). > I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy > with a vanilla 3.9.11 kernel. > > The problem : once a domU throw oom-killer, then it loop until all > processes will be killed, then the kernel abort and shutdown the VM. > > I'm not sure how to read the OOM-killer messages about memory state, so > I copy/paste a list of them here, about a 2GB RAM domU : Does /proc/meminfo on the VM show that it actually has 2GB? > [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, > oom_score_adj=0 > [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2 This appears to show a 2.6.50 kernel, whatever that is, and not 3.10.x? > [56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 > 0000000000000000 > [56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a > ffff88007773bba0 > [56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da > 000000000000000b > [56313.921443] Call Trace: > [56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc > [56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21 > [56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed > [56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14 > [56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133 > [56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14 > [56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3 > [56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b > [56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a > [56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40 > [56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a > [56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74 > [56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0 > [56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70 > [56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b > [56313.921662] Mem-Info: > [56313.921671] Node 0 DMA per-cpu: > [56313.921676] CPU 0: hi: 0, btch: 1 usd: 0 > [56313.921680] Node 0 DMA32 per-cpu: > [56313.921684] CPU 0: hi: 186, btch: 31 usd: 83 > [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1 > [56313.921693] active_file:153 inactive_file:849 isolated_file:0 > [56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0 > [56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545 > [56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0 > [56313.921693] free_cma:0 > [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB > active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB > pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [56313.921726] lowmem_reserve[]: 0 1973 1973 1973 > [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB > active_anon:90052kB inactive_anon:4132kB active_file:612kB > inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB > mapped:14452kB shmem:33308kB slab_reclaimable:404740kB > slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 > all_unreclaimable? yes > [56313.921749] lowmem_reserve[]: 0 0 0 0 > [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) > 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = > 7936kB > [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB > 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB > [56313.921791] 9341 total pagecache pages > [56313.921794] 0 pages in swap cache > [56313.921797] Swap cache stats: add 0, delete 0, find 0/0 > [56313.921799] Free swap = 0kB > [56313.921802] Total swap = 0kB > [56313.927968] 524287 pages RAM > [56313.927976] 12090 pages reserved > [56313.927978] 307132 pages shared > [56313.927980] 430905 pages non-shared > [56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [56313.927996] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [56313.928001] [ 815] 0 815 13018 114 20 0 > 0 rsyslogd > [56313.928006] [ 853] 104 853 12487 102 27 0 > 0 zabbix_agentd > [56313.928011] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [56313.928015] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [56313.928020] [ 911] 104 911 12487 207 27 0 > 0 zabbix_agentd > [56313.928025] [ 912] 104 912 12487 101 27 0 > 0 zabbix_agentd > [56313.928029] [ 913] 104 913 12487 101 27 0 > 0 zabbix_agentd > [56313.928034] [ 914] 104 914 12487 101 27 0 > 0 zabbix_agentd > [56313.928038] [ 915] 104 915 13005 109 29 0 > 0 zabbix_agentd > [56313.928043] [ 1016] 0 1016 86937 2033 155 0 > 0 apache2 > [56313.928048] [ 1026] 0 1026 4692 40 14 0 > 0 atd > [56313.928057] [ 1054] 0 1054 5747 69 16 0 > 0 cron > [56313.928062] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [56313.928067] [ 1104] 106 1104 9630 147 23 0 > 0 ntpd > [56313.928072] [ 1117] 8 1117 2947 45 12 0 > 0 nullmailer-send > [56313.928077] [ 1130] 0 1130 18989 374 35 0 > 0 searchd > [56313.928081] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [56313.928086] [ 1152] 105 1152 10265 1088 24 0 > 0 unbound > [56313.928090] [ 1172] 0 1172 1496 25 9 0 > 0 getty > [56313.928150] [ 1173] 0 1173 1496 24 8 0 > 0 getty > [56313.928157] [31557] 33 31557 88162 5277 161 0 > 0 apache2 > [56313.928163] [31998] 33 31998 88044 5689 162 0 > 0 apache2 > [56313.928168] [ 1031] 33 1031 87879 4726 161 0 > 0 apache2 > [56313.928172] [ 1579] 33 1579 87493 5027 158 0 > 0 apache2 > [56313.928177] [ 2901] 33 2901 87622 4707 158 0 > 0 apache2 > [56313.928182] [ 3447] 33 3447 82447 4550 150 0 > 0 apache2 > [56313.928186] [ 4559] 33 4559 87714 4945 160 0 > 0 apache2 > [56313.928191] [ 4560] 33 4560 82914 4906 150 0 > 0 apache2 > [56313.928195] [ 4561] 33 4561 87859 3840 159 0 > 0 apache2 > [56313.928200] [ 6205] 33 6205 87422 3053 154 0 > 0 apache2 > [56313.928204] [ 6776] 0 6776 10460 89 25 0 > 0 cron > [56313.928229] [ 6780] 0 6780 1002 23 8 0 > 0 sh > [56313.928236] [ 6784] 0 6784 7322 109 20 0 > 0 wget > [56313.928241] [ 6847] 0 6847 10460 89 25 0 > 0 cron > [56313.928246] [ 6856] 0 6856 1002 23 8 0 > 0 sh > [56313.928253] [ 6862] 0 6862 7322 108 20 0 > 0 wget > [56313.928258] [ 6924] 0 6924 10460 89 25 0 > 0 cron > [56313.928263] [ 6928] 0 6928 1002 24 8 0 > 0 sh > [56313.928267] [ 6932] 0 6932 7322 107 20 0 > 0 wget > [56313.928272] [ 6948] 33 6948 82020 2026 135 0 > 0 apache2 > [56313.928277] [ 6991] 33 6991 86937 1966 137 0 > 0 apache2 > [56313.928281] [ 6992] 33 6992 86937 1966 137 0 > 0 apache2 > [56313.928285] [ 6993] 0 6993 10460 106 25 0 > 0 cron > [56313.928290] [ 6994] 0 6994 10460 106 25 0 > 0 cron > [56313.928294] [ 6995] 0 6995 10460 106 25 0 > 0 cron > [56313.928298] [ 6999] 1337 6999 10459 183 23 0 > 0 cron > [56313.928302] [ 7000] 0 7000 10459 182 23 0 > 0 cron > [56313.928306] [ 7001] 0 7001 10460 182 23 0 > 0 cron > [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or > sacrifice child > [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, > anon-rss:10848kB, file-rss:11908kB > [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, > oom_score_adj=0 > [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2 > [56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 > ffff8800793078e8 > [56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a > ffff880001802120 > [56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 > 000000000000000a > [56317.522142] Call Trace: > [56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5 > [56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b > [56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77 > [56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d > [56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141 > [56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f > [56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77 > [56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92 > [56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31 > [56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105 > [56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141 > [56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab > [56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b > [56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f > [56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d > [56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e > [56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 > [56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed > [56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291 > [56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20 > [56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d > [56317.522479] [<ffffffff8107f3fc>] ? > timekeeping_get_ns.constprop.10+0xd/0x31 > [56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72 > [56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3 > [56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d > [56317.522512] [<ffffffff8107f3fc>] ? > timekeeping_get_ns.constprop.10+0xd/0x31 > [56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55 > [56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [56317.522526] Mem-Info: > [56317.522529] Node 0 DMA per-cpu: > [56317.522540] CPU 0: hi: 0, btch: 1 usd: 0 > [56317.522544] Node 0 DMA32 per-cpu: > [56317.522549] CPU 0: hi: 186, btch: 31 usd: 90 > [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0 > [56317.522556] active_file:95 inactive_file:793 isolated_file:0 > [56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0 > [56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523 > [56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0 > [56317.522556] free_cma:0 > [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB > active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB > pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [56317.522590] lowmem_reserve[]: 0 1973 1973 1973 > [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB > active_anon:91128kB inactive_anon:4132kB active_file:380kB > inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB > mapped:12560kB shmem:33308kB slab_reclaimable:405280kB > slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 > all_unreclaimable? yes > [56317.522621] lowmem_reserve[]: 0 0 0 0 > [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) > 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = > 7936kB > [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB > 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB > [56317.522660] 9223 total pagecache pages > [56317.522663] 0 pages in swap cache > [56317.522668] Swap cache stats: add 0, delete 0, find 0/0 > [56317.522671] Free swap = 0kB > [56317.522673] Total swap = 0kB > [56317.529054] 524287 pages RAM > [56317.529064] 12090 pages reserved > [56317.529068] 294744 pages shared > [56317.529071] 431526 pages non-shared > [56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [56317.529090] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [56317.529098] [ 815] 0 815 13018 110 20 0 > 0 rsyslogd > [56317.529105] [ 853] 104 853 12487 102 27 0 > 0 zabbix_agentd > [56317.529112] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [56317.529119] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [56317.529126] [ 911] 104 911 12487 267 27 0 > 0 zabbix_agentd > [56317.529133] [ 912] 104 912 12487 101 27 0 > 0 zabbix_agentd > [56317.529139] [ 913] 104 913 12487 101 27 0 > 0 zabbix_agentd > [56317.529146] [ 914] 104 914 12487 101 27 0 > 0 zabbix_agentd > [56317.529152] [ 915] 104 915 13005 109 29 0 > 0 zabbix_agentd > [56317.529159] [ 1016] 0 1016 86937 2043 155 0 > 0 apache2 > [56317.529166] [ 1026] 0 1026 4692 40 14 0 > 0 atd > [56317.529177] [ 1054] 0 1054 5747 69 16 0 > 0 cron > [56317.529184] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [56317.529190] [ 1104] 106 1104 9630 153 23 0 > 0 ntpd > [56317.529197] [ 1117] 8 1117 2947 45 12 0 > 0 nullmailer-send > [56317.529203] [ 1130] 0 1130 18989 395 35 0 > 0 searchd > [56317.529211] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [56317.529220] [ 1152] 105 1152 10265 1088 24 0 > 0 unbound > [56317.529228] [ 1172] 0 1172 1496 25 9 0 > 0 getty > [56317.529237] [ 1173] 0 1173 1496 24 8 0 > 0 getty > [56317.529245] [31557] 33 31557 88162 5277 161 0 > 0 apache2 > [56317.529252] [ 1031] 33 1031 87879 4726 161 0 > 0 apache2 > [56317.529261] [ 1579] 33 1579 87493 5027 158 0 > 0 apache2 > [56317.529267] [ 2901] 33 2901 87622 4707 158 0 > 0 apache2 > [56317.529274] [ 3447] 33 3447 35706 2055 71 0 > 0 apache2 > [56317.529281] [ 4559] 33 4559 87714 4945 160 0 > 0 apache2 > [56317.529287] [ 4560] 33 4560 35648 1981 71 0 > 0 apache2 > [56317.529296] [ 4561] 33 4561 87859 3840 159 0 > 0 apache2 > [56317.529304] [ 6205] 33 6205 87422 3053 154 0 > 0 apache2 > [56317.529311] [ 6776] 0 6776 10460 89 25 0 > 0 cron > [56317.529317] [ 6780] 0 6780 1002 23 8 0 > 0 sh > [56317.529324] [ 6784] 0 6784 7322 109 20 0 > 0 wget > [56317.529331] [ 6847] 0 6847 10460 89 25 0 > 0 cron > [56317.529338] [ 6856] 0 6856 1002 23 8 0 > 0 sh > [56317.529345] [ 6862] 0 6862 7322 108 20 0 > 0 wget > [56317.529351] [ 6924] 0 6924 10460 89 25 0 > 0 cron > [56317.529361] [ 6928] 0 6928 1002 24 8 0 > 0 sh > [56317.529367] [ 6932] 0 6932 7322 107 20 0 > 0 wget > [56317.529374] [ 6948] 33 6948 35279 1532 70 0 > 0 apache2 > [56317.529381] [ 6991] 33 6991 86937 1966 137 0 > 0 apache2 > [56317.529388] [ 6992] 33 6992 86937 1966 137 0 > 0 apache2 > [56317.529394] [ 6993] 0 6993 10460 88 25 0 > 0 cron > [56317.529403] [ 6994] 0 6994 10460 88 25 0 > 0 cron > [56317.529410] [ 6995] 0 6995 10460 88 25 0 > 0 cron > [56317.529416] [ 6999] 1337 6999 1002 23 8 0 > 0 sh > [56317.529423] [ 7000] 0 7000 1002 24 8 0 > 0 sh > [56317.529429] [ 7001] 0 7001 1002 23 8 0 > 0 sh > [56317.529436] [ 7002] 1337 7002 1481 21 9 0 > 0 sleep > [56317.529445] [ 7003] 0 7003 7322 110 20 0 > 0 wget > [56317.529452] [ 7004] 0 7004 7322 110 20 0 > 0 wget > [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or > sacrifice child > [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, > anon-rss:11004kB, file-rss:10104kB > [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, > oom_score_adj=0 > [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2 > [56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 > ffff88006e3ffbb0 > [56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a > ffff880001a8c8e0 > [56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da > 000000000000000b > [56680.628654] Call Trace: > [56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f > [56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d > [56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e > [56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 > [56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed > [56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [56680.628768] Mem-Info: > [56680.628771] Node 0 DMA per-cpu: > [56680.628776] CPU 0: hi: 0, btch: 1 usd: 0 > [56680.628782] Node 0 DMA32 per-cpu: > [56680.628788] CPU 0: hi: 186, btch: 31 usd: 115 > [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0 > [56680.628796] active_file:156 inactive_file:1004 isolated_file:0 > [56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0 > [56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653 > [56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0 > [56680.628796] free_cma:0 > [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB > active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB > pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [56680.628848] lowmem_reserve[]: 0 1973 1973 1973 > [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB > active_anon:78780kB inactive_anon:4132kB active_file:624kB > inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB > mapped:12820kB shmem:33308kB slab_reclaimable:410684kB > slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 > all_unreclaimable? yes > [56680.628883] lowmem_reserve[]: 0 0 0 0 > [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB > (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) > = 7928kB > [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) > 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB > [56680.628945] 9494 total pagecache pages > [56680.628949] 0 pages in swap cache > [56680.628957] Swap cache stats: add 0, delete 0, find 0/0 > [56680.628962] Free swap = 0kB > [56680.628965] Total swap = 0kB > [56680.634908] 524287 pages RAM > [56680.634916] 12090 pages reserved > [56680.634919] 291027 pages shared > [56680.634921] 431125 pages non-shared > [56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [56680.634937] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [56680.634943] [ 815] 0 815 13018 138 20 0 > 0 rsyslogd > [56680.634948] [ 853] 104 853 12487 102 27 0 > 0 zabbix_agentd > [56680.634953] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [56680.634957] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [56680.634961] [ 911] 104 911 12487 267 27 0 > 0 zabbix_agentd > [56680.634966] [ 912] 104 912 12487 101 27 0 > 0 zabbix_agentd > [56680.634971] [ 913] 104 913 12487 101 27 0 > 0 zabbix_agentd > [56680.634975] [ 914] 104 914 12487 101 27 0 > 0 zabbix_agentd > [56680.634980] [ 915] 104 915 13005 109 29 0 > 0 zabbix_agentd > [56680.634985] [ 1016] 0 1016 86937 2014 155 0 > 0 apache2 > [56680.634990] [ 1026] 0 1026 4692 40 14 0 > 0 atd > [56680.634995] [ 1054] 0 1054 5747 69 16 0 > 0 cron > [56680.634999] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [56680.635004] [ 1104] 106 1104 9630 147 23 0 > 0 ntpd > [56680.635008] [ 1117] 8 1117 2947 45 12 0 > 0 nullmailer-send > [56680.635013] [ 1130] 0 1130 18989 400 35 0 > 0 searchd > [56680.635018] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [56680.635022] [ 1152] 105 1152 10265 1099 24 0 > 0 unbound > [56680.635027] [ 1172] 0 1172 1496 25 9 0 > 0 getty > [56680.635144] [ 1173] 0 1173 1496 24 8 0 > 0 getty > [56680.635150] [ 1031] 33 1031 87879 4814 161 0 > 0 apache2 > [56680.635155] [ 1579] 33 1579 87941 5548 159 0 > 0 apache2 > [56680.635160] [ 2901] 33 2901 87622 4692 158 0 > 0 apache2 > [56680.635164] [ 4559] 33 4559 87714 4947 160 0 > 0 apache2 > [56680.635169] [ 4561] 33 4561 87859 3852 159 0 > 0 apache2 > [56680.635174] [ 6205] 33 6205 87294 3684 155 0 > 0 apache2 > [56680.635179] [ 6991] 33 6991 87434 3395 152 0 > 0 apache2 > [56680.635183] [ 6992] 33 6992 87593 3185 148 0 > 0 apache2 > [56680.635188] [ 7289] 0 7289 10460 89 25 0 > 0 cron > [56680.635192] [ 7297] 0 7297 1002 23 7 0 > 0 sh > [56680.635197] [ 7303] 0 7303 7322 110 20 0 > 0 wget > [56680.635202] [ 7363] 0 7363 10460 89 25 0 > 0 cron > [56680.635206] [ 7374] 0 7374 1002 25 7 0 > 0 sh > [56680.635213] [ 7378] 0 7378 7322 108 19 0 > 0 wget > [56680.635218] [ 7437] 0 7437 10460 89 25 0 > 0 cron > [56680.635222] [ 7438] 0 7438 10460 89 25 0 > 0 cron > [56680.635227] [ 7439] 0 7439 10460 89 25 0 > 0 cron > [56680.635231] [ 7443] 0 7443 1002 23 8 0 > 0 sh > [56680.635235] [ 7444] 0 7444 1002 23 8 0 > 0 sh > [56680.635239] [ 7445] 1337 7445 1002 24 7 0 > 0 sh > [56680.635244] [ 7446] 1337 7446 1481 22 8 0 > 0 sleep > [56680.635248] [ 7447] 0 7447 7322 109 19 0 > 0 wget > [56680.635252] [ 7448] 0 7448 7322 107 19 0 > 0 wget > [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or > sacrifice child > [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, > anon-rss:12092kB, file-rss:10100kB > [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, > oom_score_adj=0 > [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2 > [56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 > ffff88006d133bb0 > [56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a > ffff8800634686a0 > [56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da > 0000000000000009 > [56734.779011] Call Trace: > [56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f > [56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d > [56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e > [56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 > [56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed > [56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291 > [56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d > [56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27 > [56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [56734.779100] Mem-Info: > [56734.779102] Node 0 DMA per-cpu: > [56734.779106] CPU 0: hi: 0, btch: 1 usd: 0 > [56734.779108] Node 0 DMA32 per-cpu: > [56734.779112] CPU 0: hi: 186, btch: 31 usd: 62 > [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0 > [56734.779117] active_file:190 inactive_file:1109 isolated_file:0 > [56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0 > [56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857 > [56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0 > [56734.779117] free_cma:0 > [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB > active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB > pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [56734.779256] lowmem_reserve[]: 0 1973 1973 1973 > [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB > active_anon:75512kB inactive_anon:4132kB active_file:760kB > inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB > mapped:12744kB shmem:33308kB slab_reclaimable:412052kB > slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 > all_unreclaimable? yes > [56734.779293] lowmem_reserve[]: 0 0 0 0 > [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) > 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = > 7928kB > [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB > (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = > 5652kB > [56734.779351] 9645 total pagecache pages > [56734.779359] 0 pages in swap cache > [56734.779363] Swap cache stats: add 0, delete 0, find 0/0 > [56734.779367] Free swap = 0kB > [56734.779370] Total swap = 0kB > [56734.785510] 524287 pages RAM > [56734.785519] 12090 pages reserved > [56734.785523] 286620 pages shared > [56734.785526] 431108 pages non-shared > [56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [56734.785544] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [56734.785553] [ 815] 0 815 13018 138 20 0 > 0 rsyslogd > [56734.785560] [ 853] 104 853 12487 102 27 0 > 0 zabbix_agentd > [56734.785567] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [56734.785573] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [56734.785579] [ 911] 104 911 12487 208 27 0 > 0 zabbix_agentd > [56734.785586] [ 912] 104 912 12487 101 27 0 > 0 zabbix_agentd > [56734.785593] [ 913] 104 913 12487 101 27 0 > 0 zabbix_agentd > [56734.785600] [ 914] 104 914 12487 101 27 0 > 0 zabbix_agentd > [56734.785607] [ 915] 104 915 13005 109 29 0 > 0 zabbix_agentd > [56734.785614] [ 1016] 0 1016 86937 2024 155 0 > 0 apache2 > [56734.785620] [ 1026] 0 1026 4692 40 14 0 > 0 atd > [56734.785627] [ 1054] 0 1054 5747 70 16 0 > 0 cron > [56734.785634] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [56734.785641] [ 1104] 106 1104 9630 147 23 0 > 0 ntpd > [56734.785648] [ 1117] 8 1117 2947 45 12 0 > 0 nullmailer-send > [56734.785654] [ 1130] 0 1130 18989 395 35 0 > 0 searchd > [56734.785661] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [56734.785665] [ 1152] 105 1152 10265 1099 24 0 > 0 unbound > [56734.785670] [ 1172] 0 1172 1496 25 9 0 > 0 getty > [56734.785712] [ 1173] 0 1173 1496 24 8 0 > 0 getty > [56734.785718] [ 1031] 33 1031 87879 4814 161 0 > 0 apache2 > [56734.785722] [ 2901] 33 2901 87622 4692 158 0 > 0 apache2 > [56734.785727] [ 4559] 33 4559 87714 4954 160 0 > 0 apache2 > [56734.785731] [ 4561] 33 4561 87859 3815 159 0 > 0 apache2 > [56734.785737] [ 6205] 33 6205 87294 3674 155 0 > 0 apache2 > [56734.785741] [ 6991] 33 6991 87434 3398 152 0 > 0 apache2 > [56734.785745] [ 6992] 33 6992 87593 3228 150 0 > 0 apache2 > [56734.785750] [ 7363] 0 7363 10460 89 25 0 > 0 cron > [56734.785754] [ 7374] 0 7374 1002 25 7 0 > 0 sh > [56734.785760] [ 7378] 0 7378 7322 108 19 0 > 0 wget > [56734.785765] [ 7505] 0 7505 10460 89 25 0 > 0 cron > [56734.785769] [ 7506] 0 7506 10460 89 25 0 > 0 cron > [56734.785773] [ 7507] 0 7507 10460 89 25 0 > 0 cron > [56734.785777] [ 7508] 0 7508 10460 89 25 0 > 0 cron > [56734.785782] [ 7509] 0 7509 10460 89 25 0 > 0 cron > [56734.785786] [ 7510] 0 7510 10460 89 25 0 > 0 cron > [56734.785790] [ 7517] 0 7517 1002 24 8 0 > 0 sh > [56734.785794] [ 7518] 0 7518 1002 24 8 0 > 0 sh > [56734.785800] [ 7519] 0 7519 1002 24 8 0 > 0 sh > [56734.785804] [ 7520] 0 7520 1002 23 8 0 > 0 sh > [56734.785808] [ 7521] 1337 7521 1002 24 8 0 > 0 sh > [56734.785813] [ 7522] 0 7522 1002 23 7 0 > 0 sh > [56734.785817] [ 7523] 1337 7523 1481 21 8 0 > 0 sleep > [56734.785822] [ 7524] 0 7524 7322 148 20 0 > 0 wget > [56734.785826] [ 7525] 0 7525 7322 111 20 0 > 0 wget > [56734.785830] [ 7526] 0 7526 7322 110 19 0 > 0 wget > [56734.785834] [ 7527] 0 7527 7322 110 20 0 > 0 wget > [56734.785839] [ 7528] 0 7528 1002 24 7 0 > 0 maxlifetime > [56734.785846] [ 7530] 0 7530 16356 291 37 0 > 0 php5 > [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or > sacrifice child > [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, > anon-rss:11596kB, file-rss:8220kB > [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, > oom_score_adj=0 > [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2 > [56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 > 00000000000007c5 > [56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a > ffff880001a127c0 > [56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da > 0000000000000009 > [56793.525976] Call Trace: > [56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6 > [56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f > [56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d > [56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e > [56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 > [56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed > [56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291 > [56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d > [56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27 > [56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [56793.526160] Mem-Info: > [56793.526163] Node 0 DMA per-cpu: > [56793.526171] CPU 0: hi: 0, btch: 1 usd: 0 > [56793.526175] Node 0 DMA32 per-cpu: > [56793.526180] CPU 0: hi: 186, btch: 31 usd: 150 > [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5 > [56793.526188] active_file:78 inactive_file:778 isolated_file:0 > [56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0 > [56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096 > [56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0 > [56793.526188] free_cma:0 > [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB > active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB > pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:3 all_unreclaimable? yes > [56793.526238] lowmem_reserve[]: 0 1973 1973 1973 > [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB > active_anon:71364kB inactive_anon:4132kB active_file:312kB > inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB > mapped:11896kB shmem:33308kB slab_reclaimable:415876kB > slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 > all_unreclaimable? yes > [56793.526277] lowmem_reserve[]: 0 0 0 0 > [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) > 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = > 7928kB > [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB > 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB > [56793.526343] 9197 total pagecache pages > [56793.526347] 0 pages in swap cache > [56793.526352] Swap cache stats: add 0, delete 0, find 0/0 > [56793.526356] Free swap = 0kB > [56793.526359] Total swap = 0kB > [56793.532348] 524287 pages RAM > [56793.532355] 12090 pages reserved > [56793.532358] 284892 pages shared > [56793.532360] 430988 pages non-shared > [56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [56793.532376] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [56793.532381] [ 815] 0 815 13018 138 20 0 > 0 rsyslogd > [56793.532386] [ 853] 104 853 12487 102 27 0 > 0 zabbix_agentd > [56793.532391] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [56793.532396] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [56793.532400] [ 911] 104 911 12487 207 27 0 > 0 zabbix_agentd > [56793.532405] [ 912] 104 912 12487 101 27 0 > 0 zabbix_agentd > [56793.532409] [ 913] 104 913 12487 101 27 0 > 0 zabbix_agentd > [56793.532414] [ 914] 104 914 12487 101 27 0 > 0 zabbix_agentd > [56793.532418] [ 915] 104 915 13005 109 29 0 > 0 zabbix_agentd > [56793.532424] [ 1016] 0 1016 86937 2028 155 0 > 0 apache2 > [56793.532428] [ 1026] 0 1026 4692 40 14 0 > 0 atd > [56793.532433] [ 1054] 0 1054 5747 70 16 0 > 0 cron > [56793.532437] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [56793.532442] [ 1104] 106 1104 9630 147 23 0 > 0 ntpd > [56793.532447] [ 1117] 8 1117 2947 45 12 0 > 0 nullmailer-send > [56793.532451] [ 1130] 0 1130 18989 374 35 0 > 0 searchd > [56793.532456] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [56793.532461] [ 1152] 105 1152 10265 1099 24 0 > 0 unbound > [56793.532467] [ 1172] 0 1172 1496 25 9 0 > 0 getty > [56793.532484] [ 1173] 0 1173 1496 24 8 0 > 0 getty > [56793.532491] [ 1031] 33 1031 87879 4791 161 0 > 0 apache2 > [56793.532497] [ 2901] 33 2901 87622 4723 158 0 > 0 apache2 > [56793.532504] [ 4561] 33 4561 87859 3866 159 0 > 0 apache2 > [56793.532510] [ 6205] 33 6205 87294 3663 155 0 > 0 apache2 > [56793.532515] [ 6991] 33 6991 87434 3398 152 0 > 0 apache2 > [56793.532522] [ 6992] 33 6992 87593 3228 150 0 > 0 apache2 > [56793.532526] [ 7363] 0 7363 10460 89 25 0 > 0 cron > [56793.532531] [ 7374] 0 7374 1002 25 7 0 > 0 sh > [56793.532537] [ 7378] 0 7378 7322 108 19 0 > 0 wget > [56793.532547] [ 7555] 33 7555 86937 1967 137 0 > 0 apache2 > [56793.532553] [ 7593] 0 7593 10460 88 25 0 > 0 cron > [56793.532559] [ 7594] 0 7594 10460 88 25 0 > 0 cron > [56793.532564] [ 7595] 0 7595 10460 88 25 0 > 0 cron > [56793.532568] [ 7596] 0 7596 10460 88 25 0 > 0 cron > [56793.532573] [ 7597] 0 7597 10460 88 25 0 > 0 cron > [56793.532580] [ 7598] 0 7598 10460 88 25 0 > 0 cron > [56793.532584] [ 7599] 0 7599 10460 88 25 0 > 0 cron > [56793.532588] [ 7607] 0 7607 1002 25 8 0 > 0 sh > [56793.532593] [ 7608] 1337 7608 1002 24 8 0 > 0 sh > [56793.532597] [ 7609] 0 7609 1002 26 7 0 > 0 sh > [56793.532601] [ 7610] 0 7610 1002 24 7 0 > 0 sh > [56793.532606] [ 7611] 0 7611 1002 25 7 0 > 0 sh > [56793.532611] [ 7612] 0 7612 1002 23 8 0 > 0 sh > [56793.532615] [ 7613] 0 7613 1002 25 7 0 > 0 sh > [56793.532620] [ 7614] 0 7614 7322 156 20 0 > 0 wget > [56793.532624] [ 7615] 1337 7615 1481 19 8 0 > 0 sleep > [56793.532628] [ 7616] 0 7616 7322 110 19 0 > 0 wget > [56793.532632] [ 7617] 0 7617 7322 110 20 0 > 0 wget > [56793.532637] [ 7618] 0 7618 7322 109 19 0 > 0 wget > [56793.532641] [ 7619] 0 7619 7322 109 20 0 > 0 wget > [56793.532646] [ 7620] 0 7620 7322 109 20 0 > 0 wget > [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or > sacrifice child > [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, > anon-rss:9624kB, file-rss:9540kB > > ... > > [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, > oom_score_adj=0 > [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted > 2.6.50-dae-xen #2 > [57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 > ffff880073e21bb0 > [57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a > ffff8800788aea00 > [57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da > 0000000000000004 > [57131.763885] Call Trace: > [57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 > [57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 > [57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 > [57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca > [57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d > [57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 > [57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f > [57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d > [57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e > [57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 > [57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed > [57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 > [57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 > [57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182 > [57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7 > [57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 > [57131.764034] Mem-Info: > [57131.764037] Node 0 DMA per-cpu: > [57131.764041] CPU 0: hi: 0, btch: 1 usd: 0 > [57131.764045] Node 0 DMA32 per-cpu: > [57131.764049] CPU 0: hi: 186, btch: 31 usd: 62 > [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0 > [57131.764076] active_file:288 inactive_file:1121 isolated_file:0 > [57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0 > [57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420 > [57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0 > [57131.764076] free_cma:0 > [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB > active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB > managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB > pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > [57131.764115] lowmem_reserve[]: 0 1973 1973 1973 > [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB > active_anon:46972kB inactive_anon:4104kB active_file:1152kB > inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB > mapped:1432kB shmem:33312kB slab_reclaimable:434592kB > slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 > all_unreclaimable? yes > [57131.764151] lowmem_reserve[]: 0 0 0 0 > [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB > (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB > (R) = 7928kB > [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) > 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB > 0*4096kB = 5548kB > [57131.764242] 9744 total pagecache pages > [57131.764253] 0 pages in swap cache > [57131.764256] Swap cache stats: add 0, delete 0, find 0/0 > [57131.764263] Free swap = 0kB > [57131.764267] Total swap = 0kB > [57131.770252] 524287 pages RAM > [57131.770260] 12090 pages reserved > [57131.770262] 265440 pages shared > [57131.770265] 433233 pages non-shared > [57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents > oom_score_adj name > [57131.770280] [ 330] 0 330 4213 114 12 0 > -1000 udevd > [57131.770285] [ 815] 0 815 13048 150 20 0 > 0 rsyslogd > [57131.770290] [ 853] 104 853 12486 99 27 0 > 0 zabbix_agentd > [57131.770295] [ 909] 0 909 4212 113 12 0 > -1000 udevd > [57131.770299] [ 910] 0 910 4212 113 12 0 > -1000 udevd > [57131.770305] [ 1016] 0 1016 86937 2031 155 0 > 0 apache2 > [57131.770310] [ 1054] 0 1054 5747 69 16 0 > 0 cron > [57131.770314] [ 1076] 103 1076 5830 54 15 0 > 0 dbus-daemon > [57131.770319] [ 1104] 106 1104 9630 147 23 0 > 0 ntpd > [57131.770323] [ 1130] 0 1130 18989 374 35 0 > 0 searchd > [57131.770328] [ 1140] 0 1140 12307 138 28 0 > -1000 sshd > [57131.770334] [ 7363] 0 7363 10460 89 25 0 > 0 cron > [57131.770339] [ 7374] 0 7374 1002 25 7 0 > 0 sh > [57131.770343] [ 7378] 0 7378 7322 108 19 0 > 0 wget > [57131.770348] [ 8003] 0 8003 10460 88 25 0 > 0 cron > [57131.770352] [ 8006] 0 8006 10460 88 25 0 > 0 cron > [57131.770357] [ 8009] 0 8009 10460 88 25 0 > 0 cron > [57131.770361] [ 8020] 1337 8020 1002 23 7 0 > 0 sh > [57131.770366] [ 8023] 0 8023 1002 23 6 0 > 0 sh > [57131.770370] [ 8024] 0 8024 1002 24 7 0 > 0 sh > [57131.770399] [ 8030] 0 8030 7322 108 19 0 > 0 wget > [57131.770406] [ 8035] 0 8035 7322 109 20 0 > 0 wget > [57131.770410] [ 8051] 0 8051 1002 41 8 0 > 0 rc > [57131.770414] [ 8054] 0 8054 3085 60 11 0 > 0 startpar > [57131.770419] [ 8055] 1337 8055 1002 23 8 0 > 0 zabbix-launch-t > [57131.770424] [ 8057] 0 8057 1002 40 7 0 > 0 apache2 > [57131.770430] [ 8067] 1337 8067 2841 76 9 0 > 0 apache > [57131.770436] [ 8084] 0 8084 1002 24 8 0 > 0 apache2ctl > [57131.770443] [ 8092] 0 8092 26561 288 48 0 > 0 apache2 > [57131.770450] [ 8115] 0 8115 1002 22 8 0 > 0 zabbix-agent > [57131.770457] [ 8116] 0 8116 980 32 8 0 > 0 start-stop-daem > [57131.770461] [ 8158] 1337 8158 10130 168 22 0 > 0 zabbix_sender > [57131.770466] [ 8161] 33 8161 86937 1967 137 0 > 0 apache2 > [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or > sacrifice child > [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, > anon-rss:7868kB, file-rss:0kB > > > > _______________________________________________ > Xen-users mailing list > Xen-users@xxxxxxxxxxxxx > http://lists.xen.org/xen-users _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |