[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Kernel 3.11 / 3.12 OOM killer and Xen ballooning



Bob Liu wrote:
On 01/16/2014 12:35 AM, James Dingwall wrote:
Bob Liu wrote:
On 01/15/2014 04:49 PM, James Dingwall wrote:
Bob Liu wrote:
On 01/07/2014 05:21 PM, James Dingwall wrote:
Bob Liu wrote:
Could you confirm that this problem doesn't exist if loading tmem
with
selfshrinking=0 during compile gcc? It seems that you are compiling
difference packages during your testing.
This will help to figure out whether selfshrinking is the root cause.
Got an oom with selfshrinking=0, again during a gcc compile.
Unfortunately I don't have a single test case which demonstrates the
problem but as I mentioned before it will generally show up under
compiles of large packages such as glibc, kdelibs, gcc etc.

So the root cause is not because enabled selfshrinking.
Then what I can think of is that the xen-selfballoon driver was too
aggressive, too many pages were ballooned out which causeed heavy
memory
pressure to guest OS.
And kswapd started to reclaim page until most of pages were
unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
triggered.
In theory the balloon driver should give back ballooned out pages to
guest OS, but I'm afraid this procedure is not fast enough.

My suggestion is reserve a min memory for your guest OS so that the
xen-selfballoon won't be so aggressive.
You can do it through parameters selfballoon_reserved_mb or
selfballoon_min_usable_mb.
I am still getting OOM errors with both of these set to 32 so I'll try
another bump to 64.  I think that if I do find values which prevent it
though then it is more of a work around than a fix because it still
suggests that swap is not being used when ballooning is no longer
Yes, it's like a work around. But I don't think there is a better way.

  From the recent OOM log your reported:
[ 8212.940769] Free swap  = 1925576kB
[ 8212.940770] Total swap = 2097148kB

[504638.442136] Free swap  = 1868108kB
[504638.442137] Total swap = 2097148kB

171572KB and 229040KB data are swapped out to swap disk, I think there
are already significantly values for guest OS has only ~300M usable
memory.
guest OS can't find out pages suitable for swap any more after so many
pages are swapped out, although at that time the swap device still have
enough space.

The OOM may not be triggered if the balloon driver can give back memory
to guest OS fast enough but I think it's unrealistic.
So the best way is reserve more memory to guest OS.

capable of satisfying the request.  I've also got an Ubuntu Saucy (3.11
kernel) guest running on the dom0 with tmem activated so I'm going to
see if I can find a comparable workload to see if I get the same issue
with a different kernel configuration.

I've done a bit more testing and seem to have found an extra condition
which is affecting the OOM behaviour in my guests.  All my Gentoo guests
have swap space backed by a dm-crypt volume and if I remove this layer
then things seem to be behaving much more reliably.  In my Ubuntu guests
I have plain swap space and so far I haven't been able to trigger an OOM
condition.  Is it possible that it is the dm-crypt layer failing to get
working memory when swapping something in/out and causing the error?

One possible reason is the dm layer and related dm target driver occupy
a significant mount of memory and there is no way for xenselfballoon to
know this. So selfballoon driver ballooned out more memory than the
system really requires.

I have made a patch by reserving extra 10% of original total memory, by
this way I think we can make the system much more reliably in all cases.
Could you please have a test? You don't need to set
selfballoon_reserved_mb by yourself any more.
I'm running a 3.12.7 kernel with all 3 patches and original swap configuration, so far so good. I shall keep testing and let you know how things go in a few days.

Thanks,
James

/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking Y

/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_hysteresis 20
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_inertia 6
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_selfshrinking 1 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_downhysteresis 8
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballooning 1
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_interval 5 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb 0 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb 74 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_uphysteresis 1

xen_selfballoon_deaggressive.patch


diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
index 21e18c1..8f33254 100644
--- a/drivers/xen/xen-selfballoon.c
+++ b/drivers/xen/xen-selfballoon.c
@@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
  #endif /* CONFIG_FRONTSWAP */
#define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT))
+#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
/*
   * Use current balloon size, the goal (vm_committed_as), and hysteresis
@@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
  int xen_selfballoon_init(bool use_selfballooning, bool 
use_frontswap_selfshrink)
  {
        bool enable = false;
+       unsigned long reserve_pages;
if (!xen_domain())
                return -ENODEV;
@@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool 
use_frontswap_selfshrink)
        if (!enable)
                return -ENODEV;
+ /*
+        * Give selfballoon_reserved_mb a default value(10% of total ram pages)
+        * to make selfballoon not so aggressive.
+        *
+        * There are two reasons:
+        * 1) The goal_page doesn't contain some pages used by kernel space,
+        *    like slab cache and pages used by device drivers.
+        *
+        * 2) The balloon driver may not give back memory to guest OS fast
+        *    enough when the workload suddenly aquries a lot of memory.
+        *
+        * In both cases, the guest OS will suffer from memory pressure and
+        * OOM killer may be triggered.
+        * By reserving extra 10% of total ram pages, we can keep the system
+        * much more reliably and response faster in some cases.
+        */
+       if (!selfballoon_reserved_mb) {
+               reserve_pages = totalram_pages / 10;
+               selfballoon_reserved_mb = PAGES2MB(reserve_pages);
+       }
        schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ);
return 0;


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.