RE: [Xen-devel] [RFC] Transcendent Memory ("tmem"): a new approach to physical memory management

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

Hi Alan --

Sorry for the terse reply yesterday.  I was told that my
reply was unclear, and I also see I neglected to reply
to some of your questions, so please let me try again.

> I assume you've looked at how S/390 handles this problem

Could you point me to the two one liners?  If they are
simply informing the hypervisor, that is certainly a step
in the right direction.  IBM's CMM2 is of course much more
complex (and I'm told was abandoned for that reason).
Tmem falls in between and is probably a good compromise;
the change is small (though certainly greater than two one
liners) but provides lots of useful information to the
hypervisor.  One additional advantage of tmem is that
the Linux kernel actively participates in the "admission
policy" so this information need not be inferred outside
of the kernel by the hypervisor.

> I would look at the patches but the URL you give contains 
> nothing but an empty repository
> I'd be interested to see how the kernel patches look

Sorry, the patch should be there now.  The site is still
under construction :-}

Feedback very much appreciated.

(NOTE: The following refers to advanced features of tmem
still under development, not part of the core features
already submitted.)

> and also how you implement migration of some of the users of a shared
> pool object - do you implement a full clustered memory manager

Shared ephemeral pools (e.g. precache) don't need to migrate.
Since the tmem client (e.g. ocfs2 in the Linux kernel) is
responsible for ensuring consistency, there should be no dirty
pages that need to be migrated and clean pages can be left
behind (if there are other sharing "nodes") or automatically
flushed (when the last sharing node migrates or is shut down).

> and what do the
> performance figures look like across networks ? How do you
> find a pool across the network ?

I'm not trying to implement distributed shared memory.
There is no "across the network" except what the cluster
fs handles already.  The clusterfs must ensure that the
precache (shared between co-resident nodes) is consistent,
probably by flushing any inodes/objects for which another
node (physically co-resident or not) requests an exclusive
lock. 

> Its interesting as you can do a lot of other interesting 
> things with this
> kind of interface.

Indeed I hope so.  I'd like to discuss with you (offline?)
if this interface might have some value for a future
SSD interface or might help for hot-swap memory.
I also think it might be used like compcache, but with
the advantage that clean page cache pages can also be
compressed.

> Larry McVoy's bitcluster SMP proposal was 

I'll try to look that up.

Thanks again for your reply!

Dan

> -----Original Message-----
> From: Alan Cox [mailto:alan@xxxxxxxxxxxxxxxxxxx]
> Sent: Thursday, January 08, 2009 2:38 PM
> To: Dan Magenheimer
> Cc: Xen-Devel (E-mail)
> Subject: Re: [Xen-devel] [RFC] Transcendent Memory ("tmem"): a new
> approach to physical memory management
> 
> 
> > Comments and questions welcome.  I also plan to submit an
> > abstract for the upcoming Xen summit and, if accepted, give
> > a talk about tmem there.
> 
> I assume you've looked at how S/390 handles this problem - 
> the guests can
> mark pages as stable or unused and the rest is up to the 
> hypervisor ? No
> complex pool interfaces and the resulting interface slots 
> into the Linux
> kernel as a pair of 1 liners in existing arch_foo hooks in the mm. The
> S/390 keeps the question of shared/private memory objects 
> separate from
> the question of whether they are currently used - a point on which I
> think their model and interface is probably better.
> 
> I would look at the patches but the URL you give contains 
> nothing but an
> empty repository. I'd be interested to see how the kernel patches look
> and also how you implement migration of some of the users of a shared
> pool object - do you implement a full clustered memory 
> manager and what do the
> performance figures look like across networks ? How do you find a pool
> across the network ?
> 
> Its interesting as you can do a lot of other interesting 
> things with this
> kind of interface. Larry McVoy's bitcluster SMP proposal was 
> built on a
> similar idea using refcounted page loans to drive a 
> multiprocessor NUMA
> box as a cluster with page sharing.
> 
> Alan
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel