Re: [Xen-devel] [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On mer, 2013-11-06 at 11:44 +0000, George Dunlap wrote:
> On 06/11/13 10:00, Dario Faggioli wrote:
> > I see, and that sounds sensible to me... It's mostly a matter a matter
> > of deciding whether o not we want something like that, and, if yes,
> > whether we want it based on hard of soft.
> >
> > Personally, I think I agree with you on having it based on hard
> > affinities by default.
> >
> > Let's see if George get to say something before I get to that part of
> > the (re)implementation. :-)
> 
> I would probably have it based on soft affinities, since that's where we 
> expect to have the domain's vcpus actually running most if the time; 
>
True. However, doing that would rule out cpupool and vcpu-pinning. That
is to say, if you create a domain in a pool or with its vcpu pinned (by
specifying "pool=" or "cpus=" in the config file), it'll get its memory
striped over all the nodes. In fact, in both these cases, there really
is no soft affinity involved. This is bad, because people may be already
used to create a domain with "cpus=", and have the memory allocated from
the relevant nodes.

OTOH, there is no similar thing (i.e., a user interface/API) for soft
affinities, and the way I was using it at the xl and libxl level for the
sake of NUMA performance, I can easily implement there on top of soft
affinities.

So, in summary, I'd have liked to have it based on soft affinity too,
but I think that would break more things than having it based on hard
ones.

> but 
> it's really a bike-shed issue, and something we can change / adjust in 
> the future.
> 
That is also true, indeed.

> (Although I suppose ideal behavior would be for the allocator to have 
> three levels of preference instead of just two: allocate from soft 
> affinity first; if that's not available, allocate from hard affinity; 
> and finally allocate wherever you can find ram.  But that's probably 
> more work than it's worth at this point.)
> 
I like this too, but that's definitely something longer term than this
or next week.

> So what's the plan now Dario?  You're going to re-spin the patch series 
> to just do hard and soft affinities at the HV level, plumbing the 
> results through the toolstack?
> 
Yes, that was what I had in mind, and what I already started working on
(see the other mail I sent before lunch about (re)naming). I should be
able to craft and send something by ether today or tomorrow.

> I think for now I might advise putting off doing a NUMA interface at the 
> libxl level, and do a full vNUMA interface in another series (perhaps 
> for 4.5, depending on the timing).
> 
Well, I agree that all this is of very little use without vNUMA, but at
the same time, it's not necessarily only useful for it. Also, whatever
it is vcpu-node-affinity or soft-affinity, if it is not wired up
properly up to the higher layers, there's very few point of having the
HV part only... So my idea was to redo and resend everything, including
libxl and xl bits.

Of course, that doesn't mean we must necessarily have this for 4.4
(although I think it would still be feasible), just that we either
check-in or wait for both the implementation and the interface. Again,
how's the updated release schedule?

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel