RE: [Xen-devel] Re: NUMA and SMP

["Petersson, Mats" <Mats.Petersson@xxxxxxx>]

> -----Original Message-----
> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> Anthony Liguori
> Sent: 15 January 2007 17:22
> To: David Pilger
> Cc: xen-devel; Ryan Harper
> Subject: [Xen-devel] Re: NUMA and SMP
> 
> David Pilger wrote:
> > Hi all,
> > 
> > 1. Does desktop computers, such as intel dual core really 
> benefit from 
> > NUMA?
> 
> No.  NUMA standards Non-Uniform Memory Architecture.  It's 
> basically a 
> system where you have nodes (which are essentially independent 
> computers) that are connected via a high speed bus.  Each 
> node has it's 
> own memory but through the magic of NUMA, every node can access the 
> other nodes memory as if it's own.  Most NUMA systems (if not 
> all) are 
> very high end servers.

Good description, but you have to agreet that AMD has a NUMA-style
architecture in the Opteron class systems. However, this is sometimes
also called "SUMO" (Sufficiently Uniform Memory Organization), which
means that non-NUMA-aware software will operate correctly on the system,
although not optimally (because the software will allocate memory
without regard to it's locality, and thus potentially incurr penalties
that aren't necessary). It's "sufficiently uniform" because the penalty
(compared with "true NUMA") for "bad" memory allocation is in the same
order as a normal memory fetch (but of course, that means about 2X to 3X
a local memory fetch). 

On other NUMA systems, the penalty for accessing out-of-node memory can
be 10-100x the local memory access time, which is obviously a much more
noticable effect.
> 
> > 2. Does it have a real effect on the performance of Xen?
> 
> On a NUMA system, absolutely.  If you have a domain running on a 
> particular node, you want to make sure that it's using memory 
> that's in 
> it's node if at all possible.  Accessing memory on a local node is 
> considerably faster than access memory on other nodes.  Prior 
> to Ryan's 
> NUMA work, Xen would just blindly allocate memory to a domain without 
> taking into account memory locality.

Absolutely, there's a noticable benefit. 
> 
> > 3. Can't we let the guest OS manage NUMA instead of Xen? what is the
> > difference? and why is it implemented in Xen?
> 
> If a guest OS spans multiple nodes, then you would want it to be NUMA 
> aware.  However, you always want Xen to, at least, be NUMA 
> aware so that 
> it allocates memory appropriately.

Ideally, we'd want the NUMA information exported to the guest, but at
least if Xen knows that memory allocated for a particular guest is local
to the same (group of) processor(s), there's a benefit.

You can't "just" leave it to the guest OS tho', because the guest has no
control over which bits of memory it's actually gets, Xen doles that
out, and if the OS is NUMA aware but gets memory from Node1 and procesor
on Node0, then it's not much the OS can do to make things better, right?

--
Mats
> 
> Regards,
> 
> Anthony Liguori
> 
> > Thanks,
> > David.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel