Re: [Xen-devel] Kernel 3.11 / 3.12 OOM killer and Xen ballooning

[James Dingwall <james.dingwall@xxxxxxxxxxx>]

Bob Liu wrote:
> On 01/16/2014 12:35 AM, James Dingwall wrote:
> > Bob Liu wrote:
> > > On 01/15/2014 04:49 PM, James Dingwall wrote:
> > > > Bob Liu wrote:
> > > > > On 01/07/2014 05:21 PM, James Dingwall wrote:
> > > > > > Bob Liu wrote:
> > > > > > > Could you confirm that this problem doesn't exist if loading tmem
> > > > > > > with
> > > > > > > selfshrinking=0 during compile gcc? It seems that you are compiling
> > > > > > > difference packages during your testing.
> > > > > > > This will help to figure out whether selfshrinking is the root cause.
> > > > > > Got an oom with selfshrinking=0, again during a gcc compile.
> > > > > > Unfortunately I don't have a single test case which demonstrates the
> > > > > > problem but as I mentioned before it will generally show up under
> > > > > > compiles of large packages such as glibc, kdelibs, gcc etc.
> > > > > So the root cause is not because enabled selfshrinking.
> > > > > Then what I can think of is that the xen-selfballoon driver was too
> > > > > aggressive, too many pages were ballooned out which causeed heavy
> > > > > memory
> > > > > pressure to guest OS.
> > > > > And kswapd started to reclaim page until most of pages were
> > > > > unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
> > > > > triggered.
> > > > > In theory the balloon driver should give back ballooned out pages to
> > > > > guest OS, but I'm afraid this procedure is not fast enough.
> > > > >
> > > > > My suggestion is reserve a min memory for your guest OS so that the
> > > > > xen-selfballoon won't be so aggressive.
> > > > > You can do it through parameters selfballoon_reserved_mb or
> > > > > selfballoon_min_usable_mb.
> > > > I am still getting OOM errors with both of these set to 32 so I'll try
> > > > another bump to 64.  I think that if I do find values which prevent it
> > > > though then it is more of a work around than a fix because it still
> > > > suggests that swap is not being used when ballooning is no longer
> > > Yes, it's like a work around. But I don't think there is a better way.
> > >
> > >   From the recent OOM log your reported:
> > > [ 8212.940769] Free swap  = 1925576kB
> > > [ 8212.940770] Total swap = 2097148kB
> > >
> > > [504638.442136] Free swap  = 1868108kB
> > > [504638.442137] Total swap = 2097148kB
> > >
> > > 171572KB and 229040KB data are swapped out to swap disk, I think there
> > > are already significantly values for guest OS has only ~300M usable
> > > memory.
> > > guest OS can't find out pages suitable for swap any more after so many
> > > pages are swapped out, although at that time the swap device still have
> > > enough space.
> > >
> > > The OOM may not be triggered if the balloon driver can give back memory
> > > to guest OS fast enough but I think it's unrealistic.
> > > So the best way is reserve more memory to guest OS.
> > >
> > > > capable of satisfying the request.  I've also got an Ubuntu Saucy (3.11
> > > > kernel) guest running on the dom0 with tmem activated so I'm going to
> > > > see if I can find a comparable workload to see if I get the same issue
> > > > with a different kernel configuration.
> > I've done a bit more testing and seem to have found an extra condition
> > which is affecting the OOM behaviour in my guests.  All my Gentoo guests
> > have swap space backed by a dm-crypt volume and if I remove this layer
> > then things seem to be behaving much more reliably.  In my Ubuntu guests
> > I have plain swap space and so far I haven't been able to trigger an OOM
> > condition.  Is it possible that it is the dm-crypt layer failing to get
> > working memory when swapping something in/out and causing the error?
> One possible reason is the dm layer and related dm target driver occupy
> a significant mount of memory and there is no way for xenselfballoon to
> know this. So selfballoon driver ballooned out more memory than the
> system really requires.
>
> I have made a patch by reserving extra 10% of original total memory, by
> this way I think we can make the system much more reliably in all cases.
> Could you please have a test? You don't need to set
> selfballoon_reserved_mb by yourself any more.
I'm running a 3.12.7 kernel with all 3 patches and original swap 
configuration, so far so good.  I shall keep testing and let you know 
how things go in a few days.
Thanks,
James

/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking Y

/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_hysteresis 
20
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_inertia 6
/sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_selfshrinking 
1
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_downhysteresis 
8
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballooning 1
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_interval 
5
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb 
0
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb 
74
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_uphysteresis 
1
> xen_selfballoon_deaggressive.patch
>
>
> diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c
> index 21e18c1..8f33254 100644
> --- a/drivers/xen/xen-selfballoon.c
> +++ b/drivers/xen/xen-selfballoon.c
> @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void)
>   #endif /* CONFIG_FRONTSWAP */
>   
>   #define MB2PAGES(mb)	((mb) << (20 - PAGE_SHIFT))
> +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT))
>   
>   /*
>    * Use current balloon size, the goal (vm_committed_as), and hysteresis
> @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning);
>   int xen_selfballoon_init(bool use_selfballooning, bool 
> use_frontswap_selfshrink)
>   {
>         bool enable = false;
> +       unsigned long reserve_pages;
>   
>   	if (!xen_domain())
>                 return -ENODEV;
> @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool 
> use_frontswap_selfshrink)
>         if (!enable)
>                 return -ENODEV;
>   
> +	/*
> +        * Give selfballoon_reserved_mb a default value(10% of total ram pages)
> +        * to make selfballoon not so aggressive.
> +        *
> +        * There are two reasons:
> +        * 1) The goal_page doesn't contain some pages used by kernel space,
> +        *    like slab cache and pages used by device drivers.
> +        *
> +        * 2) The balloon driver may not give back memory to guest OS fast
> +        *    enough when the workload suddenly aquries a lot of memory.
> +        *
> +        * In both cases, the guest OS will suffer from memory pressure and
> +        * OOM killer may be triggered.
> +        * By reserving extra 10% of total ram pages, we can keep the system
> +        * much more reliably and response faster in some cases.
> +        */
> +       if (!selfballoon_reserved_mb) {
> +               reserve_pages = totalram_pages / 10;
> +               selfballoon_reserved_mb = PAGES2MB(reserve_pages);
> +       }
>         schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ);
>   
>   	return 0;
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel