[Xen-devel] [PATCH RFC 00/10] Xen balloon page compaction support

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Hi all

This is a prototype to make Xen balloon driver work with balloon page
compaction. The goal is to reduce guest physical address space fragmentation
introduced by balloon pages. Having guest physical address space as contiguous
as possible is generally useful (because guest can have higher order pages), and
it should also help improve HVM performance (provided that guest kernel knows
how to use huge pages -- Linux has hugetlbfs and transparent huge page).

The approach is simple. Core balloon driver is made one of the page sources of
Xen balloon driver. Those pages allocated from core balloon driver are subject
to balloon page compaction.

It can be found at [0] how memory compaction is supposed to work.  Balloon page
compaction makes use of page migration infrastructure in kernel to migrate
balloon pages from start of a zone to end of a zone, hence creating contiguous
physical address space for guest. Callback to migrate balloon page to another
location is provided by Xen balloon driver. In essence the callback just
inflates one page and deflates the other.

Looking at the stats of my experiment, to completely defragment guest address
space can be very time consuming because 1) compaction thread can only work
with a small amount of pages in one pass and 2) compaction thread moves
ballooned pages towards end of zone while buddy page allocator can allocate
ballooned pages in the middle of a zone, which increases compaction thread's
workload.

Here are some statistics.

VM has 1024 MB ram. Balloon it up and down in 64 MB trunk while compiling
kernel. After a while VM memory is set to 768 MB and compilation is stopped.

Before compaction starts, /proc/buddyinfo looks like this.
  Node 0, zone      DMA    406    276     79     19      5      3      4      1 
     0      1      0   
  Node 0, zone    DMA32   8162  16373  10204   3944    874    137     23     10 
     2      0      4   

Use following command to repeatedly trigger memory compaction.
  while true; do echo 3 > /proc/sys/vm/drop_caches; echo 1 > 
/proc/sys/vm/compact_memory; sleep 5; done

After compaction runs for about 20 minutes, /proc/buddyinfo looks like this.
  Node 0, zone      DMA     25     26     16      7      5      5      5      4 
     2      2      0 
  Node 0, zone    DMA32   2078   2480   1696    796    376    293    221    199 
   189     43     16 

Numbers of high order pages increase. The number of 2 MB pages increases from 8
(4 * 2 + 0) to 75 (16 * 2 + 43).

If VM is ballooned down when there's nothing running inside, buddyinfo looks
like this.
  Node 0, zone      DMA      5     15      8      2      3      3      4      2 
     1      3      0 
  Node 0, zone    DMA32    603    380    152     85     48     32     11      8 
     2      1    137 

The number of 2 MB pages is 275 (137 * 2 + 1).

In conclusion, balloon page compaction cannot guarantee the end result is the
same as a quietly ballooned down system. But it does provide some improvement.

The nice thing about this approach is that it doesn't rely on any hypervisor
side feature to work. Also if the defragmentation algorithm improves we get
free improvement as well.

Any thought on this patch series?

Wei.

[0] https://lwn.net/Articles/368869/

Wei Liu (10):
  balloon_compaction: don't BUG() when it is not necessary
  xen/balloon: fix code comment for free_xenballooned_pages
  xen/balloon: consolidate data structures
  xen/balloon: factor out function to update balloon stats
  xen/balloon: rework increase_reservation
  xen/balloon: make use of generic balloon driver
  xen/balloon: factor out some helper functions
  xen/balloon: implement migratepage
  balloon: BALLOON_COMPACTION now depends on XEN_BALLOON
  XXX: balloon bitmap and sysrq key to dump bitmap

 drivers/virtio/virtio_balloon.c    |    2 +-
 drivers/xen/balloon.c              |  630 ++++++++++++++++++++++++++----------
 drivers/xen/xen-balloon.c          |   24 +-
 include/linux/balloon_compaction.h |    3 +-
 include/xen/balloon.h              |   19 +-
 mm/Kconfig                         |    2 +-
 mm/balloon_compaction.c            |    9 +-
 7 files changed, 501 insertions(+), 188 deletions(-)

-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel