Re: [Xen-devel] Memory Sharing on HVM guests

[Andres Lagar-Cavilla <andreslc@xxxxxxxxxxxxxx>]

> On Thu, 2013-08-08 at 9:29 +0100, Ian wrote:
>> On Wed, 2013-08-07 at 10:35 -0700, waitxie wrote:
>>> Hi, Jui-Hao,
>> 
>> Did you intend to CC this person?
>> 
>>> I want to enable memory sharing in xen. I am a newbie in Xen. I searched in
>>> internet. There is few information about memory sharing. Your google
>>> document can not be access any more.
>>> 
>>> https://docs.google.com/document/pub?id=16cK1JtXhhyNkGHtV_0XTkFnM2nkVxSP144EjozioC6w#h.y4a96-augwr7
>>> 
>>> Can you give me some instruction about enabling memory sharing in xen?
>>> Thanks.
> 
> Maybe, but since it was in reply to a 2.5 year old email, I'm not sure
> anything would be forthcoming.  However, I am also interested in
> memory sharing, mainly among Linux domains running off a common COW
> core image, so allow me to bring in a few specific questions:
> 
> 1) Is memory sharing currently working?  I assume it is, given today's
> emails from Tim Deegan and Nai Xia, but it seems there may have been
> times it was not working.
Correct and correct, but since 4.2 we have a working subsystem.
> 
> 2) It sounds like for domains running off a common base image, and in
> particular Linux, the blkback2-initiated memory sharing is very
> effective, and introduces little overhead.  Is the blkback2 memory
> sharing currently working (it more definitely appears that this was
> disabled at some point)?  How is it invoked (for example, configure
> options and/or command line parameters)?
That part hasn't been tested for a while. What certainly works is the 
hypervisor substrate. You need to rebuild & relink blktap* to use tools/memshr, 
and then figure out whether that works and how to leverage it.

tools/memshr is one of potentially many policies you can write in user-space to 
identify what to share.
> 
> 3) Various literature points out zero-page sharing is effective for
> Windows, mainly due to scrubbing of pages when they are freed (page
> scrubbing by Xen on domain startup also seems to help).  Is there any
> code available for identifying zero pages and getting the needed
> hypercalls invoked?  This might be a naive page scanner running on
> dom0, or more a more OS-aware scanner capable of crawling through the
> kernel to pull out free page info.  Has anyone modified the Linux
> kernel to zero out freed pages to better exploit zero-page sharing?

YMMV there. You are sharing pages that will be unshared with high certainty. 
There is no code doing what you propose, but it's not terribly difficult to 
write one. The "naive page scanner" as you put it, would scan, find a zero 
page, nominate for sharing, scan again to prevent TOCTTOU race between end of 
scan and nomination, and then attempt to share. Note that the sharing hyper 
call will automatically fail in the case of TOCTTOU between nominate and share 
actions.

No one has modified Linux to do that to the best of my knowledge.

> 
> 4) Memory ballooning seems fairly new for HVMs.  Does it conflict with
> memory sharing?  For example, it might only be safe to do blkback2
> sharing, but better to allow only ballooning to manage free pages to
> avoid confusion over page ownership.  Also, when a page is freed due
> to sharing, is it then available for ballooning?  I suppose if all of
> an HVM's free pages are being quickly scrubbed and shared, ballooning
> could be superfluous for that domain.

Depends on your definition of fairly new :)

We took great care to ensure ballooning and sharing coexist peacefully. I think 
it's useless to balloon shared pages because you are not winning a single byte 
back into the system. But it can be done.

"when a page is freed due to sharing" it goes back to the host-wide pool. It is 
*not* available for ballooning. The guest has no idea it is a shared page. The 
host is free to use the freed page for whatever purposes.
> 
> 5) Is there any code available that runs inside a domain for
> identifying zero pages (and possibly identifying and hashing pages
> that might be less dynamic) and passing that information to dom0?  For
> example, a kernel module or other code with appropriate kernel hooks.

Nope that I know of.
> 
> 6) Does OS caching need to be disabled, or more likely dialed down to
> a small maximum, to ensure free memory gets shared and relinquished?

OS guest side caching is irrelevant. OS dom0 caching is disabled in most cases 
when using blkback or blktap. I am not sure what you are talking about here 
though: you are linking caching to freeing memory to sharing. Caching is a good 
thing and that's precisely what tools/memshr, the page cache on the OS guest 
side, aims to share.

> 
> 7) What is the behavior when a page needs to be unshared on an
> overcommitted system, and no pages are available?  For example, does
> the unlucky domain get hung or destroyed?  Swapping has been
> mentioned, with accompanying issues of identifying appropriate pages
> to be swapped.  Is there any code implementing swapping?

You need to come up with an appropriate fallback policy. The hypervisor will 
use the men event subsystem to alert you that the page cannot be unshared due 
to ENOMEM, and will pause the vcpu until you kick it back allowing it to try 
again. Look for the shared memory ring in <xen>/xen/arch/x86/mm/mem_sharing and 
<xen>/tools/libxc/xc_mem_sharing.

> 
> 8) My understanding is that only HVM-, and not PV-based, domUs can
> share pages.  However, even if such a feature is not presently
> implemented, could dom0 share pages with an HVM domain?

You cannot share pages in PV because PV controls the physical to machine 
mapping of pages. Sharing needs to change this arbitrarily, asynchronously, and 
at any point in time, to be able to share and unshare. It will never work for 
pure PV.
> 
> 9) Aside from the occasional academic projects, is memory sharing
> being actively used anywhere?

Gridcentric Inc for sure, all the time.

Andres
> 
> 10) Any caveats besides the general risk of an overcommited set of
> domains requiring more memory than is available?
> 
> 
> Sorry; that was a few more questions than I started off with.  They
> are in order of importance, if that helps...
> 
> Thank you,
> Eric


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel