Re: [Xen-devel] freemem-slack and large memory environments

[Mike Latimer <mlatimer@xxxxxxxx>]

Hi Wei,

On Friday, February 13, 2015 11:13:50 AM Wei Liu wrote:
> On Tue, Feb 10, 2015 at 02:34:27PM -0700, Mike Latimer wrote:
> > On Monday, February 09, 2015 06:27:54 PM Mike Latimer wrote:
> > > It seems that there are two approaches to resolve this:
> > >  - Introduce a hard limit on freemem-slack to avoid unnecessarily large
> > > reservations
> > > 
> > >  - Increase the retry count during domain creation to ensure enough time
> > >  is set aside for any cycles spent freeing memory for freemem-slack (on
> > > the test machine, doubling the retry count to 6 is the minimum required)
> > > 
> > > Which is the best approach (or did I miss something)?
> > 
> > Sorry - forgot to CC relevant maintainers.
> 
> Oops, I replied to your other email before looking at this one. Sorry.
> 26GB out of 2TB is overkill IMHO. And the 15% limit dates back to 2010
> which a) is just empirical, b) doesn't take into account large system.

Agreed.  :)

> I think we might be able to do both, introducing a hard limit as well as
> tweaking the retry count (which function are you referring to?).

Well, as I mentioned in the other thread, the situation is a bit complicated. 
As I've continued debugging here, there are actually a few issues related to 
freemem-slack and ballooning.

- tools/libxl/libxl.c:libxl__fill_dom0_memory_info sets free_mem_slack_kb to an 
empirical maximum of 15% of memory. This is obviously far too large in large 
memory environments.

Even if freemem-slack is lowered...

- tools/libxl/xl_cmdimpl.c:freemem sets dom0's memory target lower during each 
iteration of the freemem loop (despite the fact that the first iteration 
already lowered dom0's target by the amount required for domU). This causes 
dom0 to balloon down far too much, but has the interesting side effect that it 
balloons dom0 down far enough to get past the freemem-slack barrier. :)

- tools/libxl/xl_cmdimpl.c:freemem does not know about or compensate for the 
memory reservation requirement imposed by freemem-slack. As adding this 
awareness to the tools side would have ramifications for all tools, it makes 
more sense to compensate for this requirement in 
tools/libxl/libxl.c:libxl_set_memory_target - so any ballooning of dom0 will 
make sure freemem-slack space is preserved.

Even if xl sets dom0's memory target once, and libxl compensates for the 
amount of memory required by freemem-slack...

- tools/libxl/xl_cmdimpl.c:freemem does not account for the amount of time 
required to balloon dom0 past the freemem-slack barrier. Lowering the freemem-
slack amount will help, but it may still be worth increasing the retry count 
to something higher than 3.

The following patch addresses all but the first issue above. (I'm not sure what 
the best option here is (i.e. another empirical hard limit?).) I'm open to 
other ideas, but this patch has worked in testing. (Starting a 512G domain on 
a 2T host used to take 1.5 hours and ballooned dom0 down to ~20G. With this 
patch, the domain started in 30 minutes and dom0 ballooned down to the 
expected 1.5T (2T-(512G+26G freemem-slack)).

---
 tools/libxl/libxl.c      | 29 +++++++++++++++++++++++++++++
 tools/libxl/xl_cmdimpl.c | 30 +++++++++++++++++++-----------
 2 files changed, 48 insertions(+), 11 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index cd6f42c..f336c9a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -4710,10 +4710,12 @@ int libxl_set_memory_target(libxl_ctx *ctx, uint32_t 
domid,
     uint32_t memorykb = 0, videoram = 0;
     uint32_t current_target_memkb = 0, new_target_memkb = 0;
     uint32_t current_max_memkb = 0;
+    uint32_t freemem_slack = 0;
     char *memmax, *endptr, *videoram_s = NULL, *target = NULL;
     char *dompath = libxl__xs_get_dompath(gc, domid);
     xc_domaininfo_t info;
     libxl_dominfo ptr;
+    libxl_physinfo physinfo;
     char *uuid;
     xs_transaction_t t;

@@ -4783,6 +4785,33 @@ retry_transaction:
         goto out;
     }

+    /* When setting the target on dom0, compare free memory to freemem_slack
+       required reservation. If there is insufficient free memory to satisfy
+       freemem_slack reservation, dom0's target should be decreased by the
+       required freemem_slack space. */
+    if (!domid) {
+        rc = libxl_get_physinfo(ctx, &physinfo);
+        if (rc != 0) {
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                    "libxl_get_physinfo failed "
+                    "rc=%d\n", rc);
+            abort_transaction = 1;
+            goto out;
+        }
+        rc = libxl__get_free_memory_slack(gc, &freemem_slack);
+        if (rc != 0) {
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                    "libxl__get_free_memory_slack failed "
+                    "rc=%d\n", rc);
+            abort_transaction = 1;
+            goto out;
+        }
+        if ((physinfo.free_pages + physinfo.scrub_pages) * 4 < freemem_slack) 
{
+            new_target_memkb -= (freemem_slack - ((physinfo.free_pages
+                                              + physinfo.scrub_pages) * 4));
+        }
+    }
+
     if (!domid && new_target_memkb < LIBXL_MIN_DOM0_MEM) {
         LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
                 "new target %d for dom0 is below the minimum threshold\n",
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 440db78..630b427 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2199,7 +2199,7 @@ static int preserve_domain(uint32_t *r_domid, 
libxl_event *event,
 static int freemem(uint32_t domid, libxl_domain_build_info *b_info)
 {
     int rc, retries;
-    const int MAX_RETRIES = 3;
+    const int MAX_RETRIES = 10;
     uint32_t need_memkb, free_memkb, free_memkb_prev = 0;

     if (!autoballoon)
@@ -2209,19 +2209,20 @@ static int freemem(uint32_t domid, 
libxl_domain_build_info *b_info)
     if (rc < 0)
         return rc;

-    retries = MAX_RETRIES;
-    do {
-        rc = libxl_get_free_memory(ctx, &free_memkb);
-        if (rc < 0)
-            return rc;
+    rc = libxl_get_free_memory(ctx, &free_memkb);
+    if (rc < 0)
+        return rc;

-        if (free_memkb >= need_memkb)
-            return 0;
+    if (free_memkb >= need_memkb)
+        return 0;

-        rc = libxl_set_memory_target(ctx, 0, free_memkb - need_memkb, 1, 0);
-        if (rc < 0)
-            return rc;
+    /* Set memory target for dom0 before entering wait loop */
+    rc = libxl_set_memory_target(ctx, 0, free_memkb - need_memkb, 1, 0);
+    if (rc < 0)
+        return rc;

+    retries = MAX_RETRIES;
+    do {
         rc = libxl_wait_for_free_memory(ctx, domid, need_memkb, 10);
         if (!rc)
             return 0;
@@ -2234,6 +2235,13 @@ static int freemem(uint32_t domid, 
libxl_domain_build_info *b_info)
         if (rc < 0)
             return rc;

+        rc = libxl_get_free_memory(ctx, &free_memkb);
+        if (rc < 0)
+            return rc;
+
+        if (free_memkb >= need_memkb)
+            return 0;
+
         /*
          * If the amount of free mem has increased on this iteration (i.e.
          * some progress has been made) then reset the retry counter.
--

Is there a better way to handle these issues?

Thanks,
Mike


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel