[Xen-devel] [PATCH 0/3] xen: sched: Credit1 shouldn't boost vcpus being migrated.

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

Hi,

This series aims at stopping the Credit1 scheduler from granting BOOST priority
to vcpus that are migrated to a new pcpu.

In fact, BOOST is for vcpus that are waking up after, e.g., having received an
event, to, well, "boost" their wakeup latency and get them quickly to handle
the event itself. However, since vcpu_migrate() basically forces the vcpu into
a sleep+wakeup cycle, vcpus being migrated to a new pcpu, were also being
granted BOOST priority, and that is not correct.

Performance does not seem to be affected much, neither in good or bad way, by
this series. In fact, I've run a few benchmarks:
 - I've considered both a Xen build, and the Iperf network benchmark, running
   either inside 1 or 2 (HVM) VMs;
 - VMs had 12 vcpus, on a 16 pcpus host. Therefore, when only 1 VM is used, the
   host has some "spare" pcpus, while when 2 VMs where used, the host was
   overloaded;
 - each configuration was repeated 10 times, and I'm showing average and
   standard deviation;
 - I've considered the following configurations:
   1) `make xen' run inside 1 VM
   2) `make xen' run, concurrently, inside 2 VMs
   3) Iperf run inside 1 VM
   4) Iperf run, concurretnly, inside 2 VMs
   5) `make xen' and Iperf run, concurrently, each one inside one of the VMs

*** *** ***
1) XEN BUILD INSIDE 1 HVM GUEST (lower is better)
-------------------------------
baseline :  20.673+/-0.263
patched  :  20.812+/-0.797

2) XEN BUILD INSIDE 2 HVM GUESTS, CONCURRENTLY (lower is better)
----------------------------------------------
            vm1             vm2             overall
baseline :  35.641+/-1.236  35.104+/-0.392  35.372+/-0.934
patched  :  35.251+/-0.527  35.341+/-0.811  35.296+/-0.667

3) IPERF FROM 1 HVM GUEST TO HOST (higher is better)
---------------------------------
baseline :  25.25+/-0.63
patched  :  25.56+/-1.90

4) IPERF FROM 2 HVM GUESTS TO HOST, CONCURRENTLY (lower is better)
------------------------------------------------
            vm1            vm2           overall
baseline :  13.01+/-1.27   14.25+/-0.97  13.63+/-1.27
patched  :  13.29+/-1.87   14.11+/-1.48  13.70+/-1.69

5) MAKEXEN & IPERF IN ONE HVM GUEST EACH, CONCURRENTLY
------------------------------------------------------
           makexen          iperf
baseline : 30.483+/-0.843   14.92+/-1.59
patched  : 30.885+/-0.882   16.42+/-1.74
*** *** ***

So, basically:
 * 1 and 2 shows that we negligibly regress on a Xen build done in underload
   conditions, as well as we negligibly improve on a Xen build (well multiple
   Xen builds) done in overload.
 * 3 and 4, show negligible to small improvement in Iperf performance, with the
   series applied.
 * 5 shows that, in an heterogeneous workload, including both cpu load and I/O,
   the CPU load is slightly penalized by the series, while the I/O load benefits
   from it. This can probably be explained by the fact that, without this 
series,
   the CPU hungry vcpus were being boosted from time to time, even if they
   shouldn't have been, and this was impacting the vcpus doing I/O.

If anyone can and/or want to run more benchmarks (and/or include this patch in
benchmarks that you routinely run... Marcus, Malcolm, Andrew, maybe?), I'd be
interested in seeing the results.


In any case, I think non-boosting on migration is, conceptually, the right
thing to do, and that's what is done in patch 3.

In order to be able to tell, inside Credit1 code, whether a vcpu is waking up
after being blocked/asleep, or because of a migration, the wake scheduler hook
is augmented (in patch 2) with a flag field. The caller of the hook can
specify, by means of a set of flags, what has been the actual cause of the
wakeup. Right now, after all the series is applied, only two flags exists:
'regular wakeups', and 'migration wakeups'. The mechanism is extensible, and we
can add more i future, it there will come the need for this.

While there, the series introduces (patch 1) also some tracing and a new
performance counter about Credit1 boosting logic, for making it easier to
identify issues like this in future.

Thanks and Regards,
Dario
---
Dario Faggioli (3):
      xen: credit1: trace vCPU boost/unboost
      xen: sched: add wakeup flags to the scheduler interface
      xen: credit1: avoid boosting vCPUs being "just" migrated

 xen/common/sched_arinc653.c  |    2 +-
 xen/common/sched_credit.c    |   21 ++++++++++++++++-----
 xen/common/sched_credit2.c   |    2 +-
 xen/common/sched_rt.c        |    2 +-
 xen/common/schedule.c        |   13 +++++++++----
 xen/include/xen/perfc_defn.h |    1 +
 xen/include/xen/sched-if.h   |    3 ++-
 xen/include/xen/sched.h      |   10 ++++++++++
 8 files changed, 41 insertions(+), 13 deletions(-)
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa (Italy)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel