[Xen-devel] [PATCH 0/9] xen: sched: improve (a lot! :-D) Credit2 runqueue handling

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

Hi everyone,

Code is already there, in the Credit2 scheduler, for setting up one runqueue
per each socket found in the system. However, that is not functional and only
one runqueue is being created, with all the pCPUs in it.

Some more details are available here:

  http://lists.xen.org/archives/html/xen-devel/2014-08/msg02168.html

As well as in our bug #36 "credit2 only uses one runqueue instead of one runq
per socket" (http://bugs.xenproject.org/xen/bug/36).

Also, during a previous OPW round, Uma Sharma did some investigation, together
with me and George, on how we could introduce hyperthreading support in
Credit2. It turned out that arranging runqueues to be 'per-core' (rather than
per-socket) was the most obvious first step. She performed some experiments,
and what came out was that aggregating pCPUs per-core is actually providing
better performance, even better than Credit1, in some cases. There still are
scenarios where more investigation and refinements are necessary, but the
per-core approach looks more promising.

Some of the measurements she performed are available here:

  http://lists.xen.org/archives/html/xen-devel/2015-03/msg01499.html

I made some more myself, for my presentation at the last XenDevSummit, which is
available here:

  
http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf

Since there was some preparatory work to be done (i.e., what's in the first
half of this series), Uma couldn't really upstream her patches at the time, and
we agreed that I would have followed up, and re-submit her patches after the
preliminary adjustments were done. Therefore, this series encompasses Uma's
work, sometimes in the exact same form in which she originally submitted it,
sometimes tweaked to fit on top of the updated Credit2 codebase. 


So, in summary, this series does the following:

 1) first of all, it splits allocation and initialization phases of scheduling
    related per-pCPU data. Thanks to this, all allocations can be performed
    during the CPU_UP_PREPARE phase of pCPU bringup, while proper
    initialization can happen later, during CPU_STARTING. We really want
    --especially for Credit2-- to do the initialization in CPU_STARTING, as
    before that, we won't have the necessary topology information available.

    However, we can't move the alloc_pdata hook all together (and keep doing
    both allocation and initialization in there), as we can't really allocate
    memory while in CPU_STARTING (due to IRQs being disabled). Therefore, the
    split looks like the best solution, and it also seems to me to make a lot
    of sense to me, architecturally. In fact, right now, we have schedulers
    that does not need to allocate anything, but have to use alloc_pdata for
    initialization! :-/

    This was discussed already in some detail, a few time ago, in this
    subthread:

      http://lists.xen.org/archives/html/xen-devel/2015-03/msg02309.html

 2) At this point, we can easily fixup Credit2 runqueue creation phase, as we
    now have all the necessary topology information available when we need
    them. We therefore do create one runqueue per socket, and this time, it
    works. :-)

    While there, we make it possible for the user to decide how pCPUs should
    be aggregated in runqueues, with a Xen boot time parameter (socket and
    core is what is supported for now).

 3) Finally, as the per-core runqueue approach looks promising, we set
    per-core runqueue to be the default.


The series is structured as follows:
 * patch 1 and 2 are: one Credit2 debug dump output fix (patch 1), and some
   Credit2 boot parameter variables scope and placement improvements (patch 2);
 * patch 3 fixes a locking inconsistency in the scheduling interface;
 * patch 4, 5 and 6, arrange for the introduction of the new initialization
   hook (init_pcpu). Whe implementing it, for the schedulers that need that,
   code is borrowed from the allocation hook (i.e., step 1 above);
 * patch 7 and 8 fix Credit2 runqueue creation, and introduce a boot time
   parameter for influencing the runqueue aggregation strategy (i.e., step
   2 above);
 * patch 9 makes per-core the default runqueue aggregation strategy. This
   is done in a separate patch to avoid mixing technical and (default) policy
   changes.


This series is available as git branch, on top of staging, here:

  git://xenbits.xen.org/people/dariof/xen.git  rel/sched/credit2/fix-runqueues
  
http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit2/fix-runqueues

There is also a branch containing this series, rebased on top of the following
(already sent) three patches:

  xen: sched: get rid of cpupool_scheduler_cpumask()
  xen: sched: adjustments to some performance counters
  xen: credit1: fix tickling when it happens from a remote pCPU

just in case those go in before this series (which I'd call likely), and one
wants to pull the series on top of that:

  git://xenbits.xen.org/people/dariof/xen.git  
rel/sched/credit2/fix-runqueues_rebase
  
http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit2/fix-runqueues_rebase

Regards,
Dario

---
Dario Faggioli (8):
      xen: sched: fix an 'off by one \t' in credit2 debug dump
      xen: sched: make locking for {insert,remove}_vcpu consistent
      xen: sched: add .init_pdata hook to the scheduler interface
      xen: sched: make implementing .alloc_pdata optional
      xen: sched: implement .init_pdata in all schedulers
      xen: sched: fix per-socket runqueue creation in credit2
      xen: sched: allow for choosing credit2 runqueues configuration at boot
      xen: sched: per-core runqueues as default in credit2

Uma Sharma (1):
      xen: sched: improve scope and placement of credit2 boot parameters

 docs/misc/xen-command-line.markdown |   11 ++
 xen/common/sched_arinc653.c         |   31 ------
 xen/common/sched_credit.c           |   47 +++++++--
 xen/common/sched_credit2.c          |  186 +++++++++++++++++++----------------
 xen/common/sched_rt.c               |   19 +++-
 xen/common/schedule.c               |   28 +++--
 xen/include/xen/sched-if.h          |    1 
 7 files changed, 182 insertions(+), 141 deletions(-)

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel