[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity

On 06/11/13 15:14, Jan Beulich wrote:
On 06.11.13 at 15:56, George Dunlap <george.dunlap@xxxxxxxxxxxxx> wrote:
On 06/11/13 14:26, Dario Faggioli wrote:
On mer, 2013-11-06 at 11:44 +0000, George Dunlap wrote:
On 06/11/13 10:00, Dario Faggioli wrote:
I see, and that sounds sensible to me... It's mostly a matter a matter
of deciding whether o not we want something like that, and, if yes,
whether we want it based on hard of soft.

Personally, I think I agree with you on having it based on hard
affinities by default.

Let's see if George get to say something before I get to that part of
the (re)implementation. :-)
I would probably have it based on soft affinities, since that's where we
expect to have the domain's vcpus actually running most if the time;

True. However, doing that would rule out cpupool and vcpu-pinning.
I guess I was assuming that a vcpu's soft affinity would always be
considered a subset of its hard affinity and the cpus in its cpupool.
For CPU pools I agree, but didn't we mean hard affinity to control
memory allocation, and soft affinity scheduling decisions (in which
case there's no strict ordering between the two)? If not, I guess I'd
need a brief but clear definition what "soft" and "hard" are supposed
to represent...

We have two "affinities" at the moment:
* per-vcpu cpu affinity. This is this is a scheduling construct, and is used to restrict vcpus to run on specific pcpus. This is what we would call hard affinity -- "hard" because vcpus are *not allowed* to run on cpus not in this mask. * Domain NUMA affinity. As of 4.3 this has two functions: both a memory allocation function, and a scheduling function. For the scheduling function, it acts as a "soft affinity", but it's domain-wide, rather than being per-vcpu. It's "soft" because the scheduler will *try* to run it on pcpus in that mask, but if it cannot, it *will allow* them to run on pcpus not in that mask.

At the moment you can set this manually, or if it's not set, then Xen will set it automatically based on the vcpu hard affinities (and I think the cpupool that it's in).

What we're proposing is to remove the domain-wide soft affinity and replace it with a per-vcpu soft affinity. That way each vcpu has some pcpus that it prefers (in the soft affinity *and* the hard affinity), some that it doesn't prefer but is OK with (hard affinity but not soft affinity), and some that it cannot run on at all (not in the hard affinity).

The question Dario has is this: given that we now have per-vcpu hard and soft scheduling affinity, how should we automatically construct the per-domain memory allocation affinity, if at all? Should we construct it from the "hard" scheduling affinities, or from the "soft" scheduling affinities?

I said that I thought we should use the soft affinity; but I really meant the "effective soft affinity" -- i.e., the union of soft, hard, and cpupools.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.