[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-devel] Kernel 3.11 / 3.12 OOM killer and Xen ballooning
Bob Liu wrote:
On 12/12/2013 12:30 AM, James Dingwall wrote:
Bob Liu wrote:
On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote:
On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote:
Konrad Rzeszutek Wilk wrote:
On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote:
Hi,
Since 3.11 I have noticed that the OOM killer quite frequently
triggers in my Xen guest domains which use ballooning to
increase/decrease their memory allocation according to their
requirements. One example domain I have has a maximum memory
setting of ~1.5Gb but it usually idles at ~300Mb, it is also
configured with 2Gb swap which is almost 100% free.
# free
total used free shared buffers
cached
Mem: 272080 248108 23972 0 1448 63064
-/+ buffers/cache: 183596 88484
Swap: 2097148 8 2097140
There is plenty of available free memory in the hypervisor to
balloon to the maximum size:
# xl info | grep free_mem
free_memory : 14923
An example trace (they are always the same) from the oom killer in
3.12 is added below. So far I have not been able to reproduce this
at will so it is difficult to start bisecting it to see if a
particular change introduced this. However it does seem that the
behaviour is wrong because a) ballooning could give the guest more
memory, b) there is lots of swap available which could be used as a
fallback.
Keep in mind that swap with tmem is actually no more swap. Heh, that
sounds odd -but basically pages that are destined for swap end up
going in the tmem code which pipes them up to the hypervisor.
If other information could help or there are more tests that I could
run then please let me know.
I presume you have enabled 'tmem' both in the hypervisor and in
the guest right?
Yes, domU and dom0 both have the tmem module loaded and tmem
tmem_dedup=on tmem_compress=on is given on the xen command line.
Excellent. The odd thing is that your swap is not used that much, but
it should be (as that is part of what the self-balloon is suppose to
do).
Bob, you had a patch for the logic of how self-balloon is suppose
to account for the slab - would this be relevant to this problem?
Perhaps, I have attached the patch.
James, could you please apply it and try your application again? You
have to rebuild the guest kernel.
Oh, and also take a look at whether frontswap is in use, you can check
it by watching "cat /sys/kernel/debug/frontswap/*".
I have tested this patch with a workload where I have previously seen
Thank you so much.
failures and so far so good. I'll try to keep a guest with it stressed
to see if I do get any problems. I don't know if it is expected but I
did note that the system running with this patch + selfshrink has a
kswapd0 run time of ~30mins. A guest without it and selfshrink disabled
Could you run the test again with this patch but selfshrink disabled and
compare the run time of kswapd0?
Here are the results against two vms with/without the patch. They are
running on the same dom0 and have comparable xen configs and were
restarted at the same point.
With patch:
# uptime ; ps -ef | grep [k]swapd0
14:58:55 up 6:32, 1 user, load average: 0.00, 0.01, 0.05
root 310 2 0 08:26 ? 00:00:01 [kswapd0]
### BUILD GLIBC
# ps -ef | grep [k]swapd0
root 310 2 0 08:26 ? 00:00:16 [kswapd0]
### BUILD KDELIBS
# ps -ef | grep [k]swapd0
root 310 2 1 08:26 ? 00:09:15 [kswapd0]
# for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
Without patch:
# uptime ; ps -ef | grep [k]swapd0
14:59:12 up 6:32, 1 user, load average: 0.00, 0.01, 0.05
root 309 2 0 08:26 ? 00:00:01 [kswapd0]
### BUILD GLIBC
# ps -ef | grep [k]swapd0
root 309 2 0 08:26 ? 00:00:09 [kswapd0]
### BUILD KDELIBS
# ps -ef | grep [k]swapd0
root 309 2 0 08:26 ? 00:01:18 [kswapd0]
# for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done
/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N
having run a similar workload has ~5mins. With the patch I also noted
the following kernel messages which I haven't seen before:
[ 8733.646820] init_memory_mapping: [mem 0x120000000-0x127ffffff]
[ 8733.646825] [mem 0x120000000-0x127ffffff] page 4k
[10506.639875] init_memory_mapping: [mem 0x128000000-0x137ffffff]
[10506.639881] [mem 0x128000000-0x137ffffff] page 4k
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
