[Xen-users] Xen Clocksource, the VDSO)... not how, but why... and some micro-benchmarks.

[Hans van Kranenburg <Hans.van.Kranenburg@xxxxxxxxxx>]

Hi,

The hypervisor command line and the domX commandline (using linux here
as example) allow settings related to clocksource.

https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#clocksource-x86

  clocksource (x86) = pit | hpet | acpi | tsc

Or, for linux:

https://raw.githubusercontent.com/torvalds/linux/master/Documentation/admin-guide/kernel-parameters.txt

  clocksource [X86-64] hpet,tsc

The question I want to raise in this post is not really what can be set,
or how, since that's clear. The question is: why should I change
defaults to something else (something better?).

By default, dom0 and domU use clocksource 'xen' (which can be seen at
/sys/devices/system/clocksource/clocksource0/current_clocksource).

A well known 'issue' with the 'xen' clocksource is that it does not have
VDSO support to accellerate a certain group of syscalls. A rather well
known blog about this is...


https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/

So, I'm looking at the possibility to set...

  clocksource=tsc tsc=stable:socket

...for the hypervisor, since I think my hardware can do this and...

  clocksource=tsc

...for the domU kernel command line.

I still haven't found the exact reason why we should add clocksource=tsc
tsc=stable:socket to the hypervisor command line. It's not needed to
make all tsc vdso trickery in the domU work. Moreover, it does not even
seem needed at all to set clocksource=tsc in xen to be able to use it in
the domU?

The only place where I can find tsc=stable:socket being mentioned is
https://lore.kernel.org/patchwork/cover/849340/ which should be a
changeset to enable vdso calls for the xen clocksource (in linux 4.15).
But that never got merged.

==== Some benchmarks ====

Anyway, I did some micro-benchmarks today to see what different
combinations of settings do.

The victim hardware is a HP dl360 gen8 with Intel(R) Xeon(R) CPU E5-2650
v2 @ 2.60GHz cpus. cpuid says: Intel Xeon E5-1600/E5-2600 v2 (Ivy
Bridge-EP C1/M1/S1), 22nm. It's running Xen 4.11 (from commit
87f51bf366), and I have a test domU running Debian Stretch with 4.19.20
kernel. Dom0 is also Debian Stretch with 4.19.20 kernel.

What I'm testing is a totally not-real-life scenario of just calling
gettimeofday 5 million times:

-$ cat test2.c
#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

int
main(int argc, char *argv[])
{
        struct timeval tv;
        int i = 0;
        for (; i<5000000; i++) {
                gettimeofday(&tv,NULL);
        }

        return 0;
}

The hypervisor command line has "clocksource=tsc tsc=stable:socket" in
all cases.

The results of running this with different combinations of PV, PVH, xen
and tsc clocksource, and tsc clocksource after live migration are
attached in xen-tsc-timings.txt. The results are... rather interesting.

==== TSC and no migrate ====

I have read the information at:


https://xenbits.xen.org/docs/unstable/man/xen-tscmode.7.html#TSC-INVARIANT-BIT-and-NO_MIGRATE

The information about only being able to use tsc when no migrate is set
seems no longer true, since I can set clocksource=tsc and then live
migrate. When doing so, the following things change in cpuid output:

--- cpuid       2019-03-07 15:57:39.045024075 +0100
+++ cpuid-after-migrate 2019-03-07 15:59:27.456474458 +0100
@@ -272,14 +272,14 @@
       MSR base address                   = 0x40000000
       MMU_PT_UPDATE_PRESERVE_AD supported = false
    hypervisor time features (0x40000003/00):
-      vtsc                = false
+      vtsc                = true
       host tsc is safe    = true
       boot cpu has RDTSCP = true
       tsc mode            = 0x0 (0)
       tsc frequency (kHz) = 2593772
-      incarnation         = 0x1 (1)
-      cpu frequency (kHZ) = 747622702
-   0x40000004 0x00: eax=0x0000001f ebx=0x00000001 ecx=0x00000005
edx=0x00000000
+      incarnation         = 0x2 (2)
+      cpu frequency (kHZ) = 0
+   0x40000004 0x00: eax=0x0000001f ebx=0x00000000 ecx=0x0000000d
edx=0x00000000
    0x40000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000
edx=0x00000000
    extended feature flags (0x80000001/edx):
       SYSCALL and SYSRET instructions        = true
@@ -341,7 +341,7 @@
       software thermal control (STC) = false
       100 MHz multiplier control     = false
       hardware P-State control       = false
-      TscInvariant                   = false
+      TscInvariant                   = true
    Physical Address and Linear Address Size (0x80000008/eax):
       maximum physical address bits         = 0x2e (46)
       maximum linear (virtual) address bits = 0x30 (48)

So it seems I'm running a virtualized tsc then.

==== H'okay, so... ====

So the remaining questions are:

* Why should I set clocksource=tsc on the hypervisor line at all? I can
see it makes the clocksource in dom0 change to tsc.
* What's this tsc=stable:socket about? What difference does it make for
Xen? Do I want this if my hardware can do it?
* What other things am I missing?
* Should we write a HOWTO wiki page about this, as addition to the
reference documentation?
* Any other feedback?

Thanks,

Hans

Attachment: xen-tsc-timings.txt
Description: xen-tsc-timings.txt

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users