[Xen-devel] [RFC] Transcendent Memory ("tmem"): a new approach to physical memory management

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

At last year's Xen North America Summit in Boston, I gave a talk
about memory overcommitment in Xen.  I showed that the basic
mechanisms for moving memory between domains were already present
in Xen and that, with a few scripts, it was possible to roughly
load-balance memory between domains.  During this effort, I
discovered that "ballooning" had a lot of weaknesses, even
though it is the foundation for "time-sharing" physical
memory in every major virtualization system existing today.
These weaknesses have led in many cases to unacceptable performance
issues when VMs are densely packed; as a result, memory is becoming
the bottleneck in many deployments of virtualization.

Transcendent Memory -- or "tmem" for short -- is phase II of that
work and it essentially augments ballooning and "fixes" many of
its weaknesses.  It requires paravirtualization, but the changes
(to Linux) are fairly small and minimally-invasive.  The changes
to Xen are larger, but also fairly non-invasive.  (No shell scripts
this time! :-)  The concept and code is modular and may easily
port to Windows, as well as KVM.  It may even be useful in
containers and in a native physical operating system. And,
yes, it is machine-independent so should be easily portable
to ia64 too!

Basically, instead of moving the ownership of all physical memory
between one domain and another, tmem instead collects system-wide
underutilized memory into a "pool" in the hypervisor and provides
indirect access to that memory so that it can serve the needs
of domains without necessarily being directly addressible by the
domains it serves.  It is implemented with a small set of
(hyper)calls that enable pages to be copied between a domain
and Xen, controlled by a carefully-crafted set of semantics that
make it easy in most cases for memory to be easily reclaimed
by Xen as memory needs vary (as they often do -- rapidly and
unpredictably).  As a result, physical memory is utilized more
efficiently, reducing unnecessary paging and the likelihood
of thrashing and thus increasing performance and/or allowing
greater VM density.

If you are interested in this topic, please see:

http://oss.oracle.com/projects/tmem
(note, site is sometimes slow)

for more information.  This site will be updated frequently,
with patches, documentation, and FAQs.  The site also
supports mailing lists, though I'd prefer to have all
Xen-related discussions start on xen-devel.

Linux patches based on 2.6.18-xen, 2.6.27-xen, and 2.6.28
are available.  The Xen patch is currently-based on 3.3.0+
and I am in the process of updating it and cleaning it up, so
will post it in the near future, but can provide it to anyone
who is very interested in seeing/trying it now.  I could
use some help on the "control plane" python software,
in performance evaluation, and in "porting".

Comments and questions welcome.  I also plan to submit an
abstract for the upcoming Xen summit and, if accepted, give
a talk about tmem there.

Thanks,
Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel