[Xen-devel] [PATCH 0/2] Credit2: fix per-socket runqueue setup

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

Hi everyone,

Code is already there in the credit2 scheduler, for setting up one runqueue per
each socket found in the system, despite the fact that comments in
sched_credit2.c still report this (well, something like this )to be a TODO
item.

However, such code is not functional. In fact, on the following host:

cpu_topology           :
cpu:    core    socket     node
  0:       0        1        0
  1:       0        1        0
  2:       1        1        0
  3:       1        1        0
  4:       2        1        0
  5:       2        1        0
  6:       8        1        0
  7:       8        1        0
  8:       9        1        0
  9:       9        1        0
 10:      10        1        0
 11:      10        1        0
 12:       0        0        1
 13:       0        0        1
 14:       1        0        1
 15:       1        0        1
 16:       2        0        1
 17:       2        0        1
 18:       8        0        1
 19:       8        0        1
 20:       9        0        1
 21:       9        0        1
 22:      10        0        1
 23:      10        0        1
numa_info              :
node:    memsize    memfree    distances
   0:     18432      17488      10,20
   1:     19200      17932      20,10

What I see during boot and scheduler initialization is as follows (for all the
24 pCPUs):

(XEN) init_pcpu: Strange, cpu 1 already initialized!
(XEN) init_pcpu: Strange, cpu 2 already initialized!
...
(XEN) init_pcpu: Strange, cpu 23 already initialized!

Also, once the system booted, here's what happened:
root@tg03:~# xl dmesg |grep -i runqueue
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
...
(XEN) Adding cpu 23 to runqueue 0

I.e., only one runqueue was created, despite the host having 2 sockets, and all
the pCPUs have been assigned to it.

This series fixes that by doing the following three things:
- anticipate the identification of the topology of the boot CPU to happen
  _before_ the scheduler's initialization. This is necessary to make sure that,
  when credit2 will try to figure out on which socket CPU 0 it, it will find the
  correct info (patch 1);
- properly initialize the cpuinfo_x86 data structure, and more specifically the
  phys_proc_id field, that hosts the socket, to something invalid (-1). In fact,
  having that field in all elements set to 0 would induce credit2 to think that
  the pCPU have already been initialized, and that all are on socket 0 (patch 
1); 
- use the info made available by early boot CPU identification in credit2
  (patch 2).

After this series is applied, no more "Strange, cpu X already initialized!" can
be found in the logs. Also, here's how the scheduler is being setup:

root@tg03:~# xl dmesg |grep -i runqueue
(XEN) Adding cpu 0 to runqueue 1
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 1
(XEN) Adding cpu 2 to runqueue 1
(XEN) Adding cpu 3 to runqueue 1
(XEN) Adding cpu 4 to runqueue 1
(XEN) Adding cpu 5 to runqueue 1
(XEN) Adding cpu 6 to runqueue 1
(XEN) Adding cpu 7 to runqueue 1
(XEN) Adding cpu 8 to runqueue 1
(XEN) Adding cpu 9 to runqueue 1
(XEN) Adding cpu 10 to runqueue 1
(XEN) Adding cpu 11 to runqueue 1
(XEN) Adding cpu 12 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 13 to runqueue 0
(XEN) Adding cpu 14 to runqueue 0
(XEN) Adding cpu 15 to runqueue 0
(XEN) Adding cpu 16 to runqueue 0
(XEN) Adding cpu 17 to runqueue 0
(XEN) Adding cpu 18 to runqueue 0
(XEN) Adding cpu 19 to runqueue 0
(XEN) Adding cpu 20 to runqueue 0
(XEN) Adding cpu 21 to runqueue 0
(XEN) Adding cpu 22 to runqueue 0
(XEN) Adding cpu 23 to runqueue 0

Which makes a lot more sense. :-)

Regards,
Dario

---
Dario Faggioli (2):
      x86: during boot, anticipate identifying the boot cpu
      sched: credit2: use boot CPU info for CPU #0


 xen/arch/x86/setup.c            |    8 ++++++--
 xen/arch/x86/smpboot.c          |    3 ++-
 xen/common/sched_credit2.c      |   12 +++++-------
 xen/include/asm-x86/processor.h |    6 ++++--
 4 files changed, 17 insertions(+), 12 deletions(-)

--
s happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel