Re: [Xen-devel] NUMA-aware VM placement in Xen

[Dario Faggioli <raistlin@xxxxxxxx>]

On Fri, 2012-02-24 at 10:16 +0000, George Dunlap wrote: 
> > Basically, given alloc_domheap_pages uses d->node_affinity for deciding
> > from which node(s) to actually take memory from, I was planning to
> > either use the same mask or build a new one with similar purposes, the
> > problem being _where_ to populate it with the proper nodes.
> > I'm now looking at xc_domain_setmaxmem-->do_domctl(XEN_DOMCTL_max_mem),
> > although I think it's too early, and I'd end up guessing wrt a lot of
> > aspects... But considering xm/xend was doing the same even earlier (at
> > least I think)...
> So the first question is, where should the decision about NUMA placement 
> be made, and the second is how that level should implement it.
> 
Yes, indeed.

> Doing it at the libxc level I think is not right.  
>
Ok, same here. Just to be sure I understood what you're saying, if you
refer to xc_domain_setmaxmem, as I'll end up doing it in do_domctl, it'd
be in Xen, but anyway it still won't look like the way I wanted it to be
(see below). :-(

> It seems to me we 
> have two options:
> * Have libxl do the NUMA placement on behalf of the toolstack.  In that 
> case, the libxl_domain_create_new function should look at the available 
> memory, the NUMA layout, &c, and then set d->node_affinity before 
> calling xc_hvm_build.
>
This can be done. If I got it correctly it is more or less what xm/xend
already does.

> * Have the toolstack do it.  In this case, you'd be modifying xl to set 
> d->node_affinity before calling libxl's domain creation function.
> 
I'm not sure I'm getting this right... It seems very similar to the one
above.

> Do those options work?  Let me know if I've misunderstood anything.
> 
I think they can be implemented. "work", it depends on how we define
"work". :-D

That's why I was struggling for putting this in the hypervisor and not
in the toolstack because I really think it should live there if
possible. For example it would be nice for the decision to be protected
by the proper locking. I mean, what's the point in checking the amount
of free memory in a node somewhere in (lib)xl, if when the actual
allocation will happen (in Xen) that might be a completely different
value (due to concurrent domain creation, destruction, etc.)?

> Any thoughts one way or the other from anyone?
> 
Any ideas on how to put that thing _in_ Xen?

> I'd be tempted to have it be optional -- you can set "numa=auto" and the 
> domain creation function will do the simple thing; or you can set 
> "numa=manual" and have the toolstack / config file set the nodes 
> manually.  That would translate pretty well to config files as well -- 
> more "set the knobs" administrators could set the numa layout in the 
> config file manually if they wanted.
> 
I agree and that was already my plan: configurable and per-domain.

I think the config file, supporting cpupools and vcpu-pinning, already
offer almost all the facilities for manually deploying a VM reflecting a
specific NUMA-layout. What I was thinking adding was the "numa=auto" or
whatever switch, so that if one does not (want to) specify cpupools or
pinning, VM still gets NUMA-sensible placement.

But anyway, no problem adding other knobs if considered worthwhile, the
problem is the other part! :-P

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel