Re: [Xen-users] Xen Clocksource, the VDSO)... not how, but why... and some micro-benchmarks.

[Hans van Kranenburg <Hans.van.Kranenburg@xxxxxxxxxx>]

On 3/14/19 9:32 AM, Juergen Gross wrote:
> On 11/03/2019 01:05, Hans van Kranenburg wrote:
>> On 3/7/19 5:39 PM, Hans van Kranenburg wrote:
>>> [...]
>>>
>>> I still haven't found the exact reason why we should add clocksource=tsc
>>> tsc=stable:socket to the hypervisor command line. It's not needed to
>>> make all tsc vdso trickery in the domU work. Moreover, it does not even
>>> seem needed at all to set clocksource=tsc in xen to be able to use it in
>>> the domU?
>>>
>>> The only place where I can find tsc=stable:socket being mentioned is
>>> https://lore.kernel.org/patchwork/cover/849340/ which should be a
>>> changeset to enable vdso calls for the xen clocksource (in linux 4.15).
>>> But that never got merged.
>>
>> Well, I went on a journey to find out what tsc=stable:socket is actually
>> doing, and that journey ended rather soon. :D
>>
>> The tsc=stable:socket was introduced in...
>>
>> commit bc900cbc8f37b93cc6c9f6370beb14e6430b334d
>> Author: Joao Martins <joao.m.martins@xxxxxxxxxx>
>> Date:   Fri Sep 23 18:26:19 2016 +0200
>>
>>     x86/time: extend "tsc" param with "stable:socket"
>>
>> ...and nothing at all seems to be using TSC_RELIABLE_SOCKET or the
>> tsc_flags introduced there in the code further on.
>>
>> +/* TSC is reliable across sockets */
>> +#define TSC_RELIABLE_SOCKET (1 << 0)
>>
>> So, that command line part is effectively a noop. Maybe it was added
>> because that linux patchset for xen clocksource wanted to use it? But,
>> those patches also don't explain what would be special about this extra
>> option and what it would be used for.
>>
> 
> What about its usage in init_tsc()? It will clearly have an effect on
> systems with multiple sockets.

Right, how did I miss that. Maybe some accidental filter on a file or
directory while showing the commit contents. :|

So, in that case, let's continue the investigation...

BIOS hyperthreading disabled
Xen smt=off clocksource=tsc tsc=stable:socket loglvl=all
----
(XEN) Brought up 12 CPUs
(XEN) TSC: CPU Hotplug intended <- logged on WARNING level
(XEN) TSC: Not setting it as clocksource <- logged on DEBUG level
----
nr_cpus                : 12
max_cpu_id             : 31
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1

When looking at the code:

    if ( nr_cpu_ids != num_present_cpus() )
    {
        printk(XENLOG_WARNING "TSC: CPU Hotplug intended\n");
        ret = 0;
    }

Hm... nr_cpu_ids? In my case nr_cpu_ids seems to have value 32, because
the hardware reports there can be 32 cpu cores in total.

That number 32 is interesting, since I guess it means that if I would
replace the 2x 6-core cpus in this pizzabox with 2x 8-core cpus that I
buy somewhere and enable hyperthreading, I can reach the number 32.

(XEN) SMP: Allowing 32 CPUs (20 hotplug CPUs)

However, I can't hotswap these physical cpus when the machine is
running, and I also can't change the BIOS hyperthreading setting while
it's running, so I don't really get what kind of CPU Hotplug I would be
intending here.

Dum dum dum... This means I can't ever get TSC.

Also... turning off hyperthreading is encouraged for security reasons
nowadays, so even if I would buy some 8 core cpus, I would have to
enable hyperthreading in BIOS and Xen to get TSC.

Other combinations of settings:

BIOS hyperthreading enabled
Xen smt=off clocksource=tsc tsc=stable:socket loglvl=all
----
(XEN) Brought up 12 CPUs
(XEN) Parked 12 CPUs
(XEN) TSC: CPU Hotplug intended
(XEN) TSC: Not setting it as clocksource
----
nr_cpus                : 12
max_cpu_id             : 31
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1

and clocksource=tsc tsc=stable:socket loglvl=all

BIOS hyperthreading enabled
Xen smt=on
(XEN) Brought up 24 CPUs
(XEN) TSC: CPU Hotplug intended
(XEN) TSC: Not setting it as clocksource
nr_cpus                : 24
max_cpu_id             : 31
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2

    if ( nr_sockets > 1 && !(tsc_flags & TSC_RELIABLE_SOCKET) )
    {
        printk(XENLOG_WARNING "TSC: Not invariant across sockets\n");
        ret = 0;
    }

It's actually not easy to find which hardware can be used with
tsc=stable:socket.

This test box has Xeon 5600 series cpus. I can find something related in
the datasheet:

https://www.intel.com/content/www/us/en/processors/xeon/xeon-5600-vol-1-datasheet.html
On page 63 "Note: In order to ensure Timestamp Counter (TSC)
synchronization across sockets in multi-socket systems, the RESET#
deassertion edge should [...]"

But the hardware around the cpus is a HP DL360 G7, which has to comply
to these rules in the datasheet. I haven't been able to find any info
about that yet.

---- >8 ----

    if ( !ret )
        printk(XENLOG_DEBUG "TSC: Not setting it as clocksource\n");

    return ret;

I didn't see the "TSC: Not setting it as clocksource" at first, because
it's logged at DEBUG level. It would make sense to have that on WARNING
level, since as a user I'm explicitly telling it to use TSC and it's
refusing and doing something else, so I'd like to know.

---- >8 ----

I also couldn't really find out yet which clocksource it's actually
using now, there is no info readily available in e.g. xl info output.

In xl dmesg, I earlier see...
  (XEN) Platform timer is 14.318MHz HPET
...so maybe that's it?

---- >8 ----

The dom0 linux kernel seems more happy about it. Without even specifying
clocksource=, it chooses tsc.

-# dmesg |grep -i tsc
[    0.034139] tsc: Fast TSC calibration using PIT
[    0.034141] tsc: Detected 2666.872 MHz processor
[    0.034142] tsc: Detected 2666.784 MHz TSC
[    1.042078] clocksource: tsc-early: mask: 0xffffffffffffffff
max_cycles: 0x2670acaa8d1, max_idle_ns: 440795300001 ns
[    1.165111] clocksource: Switched to clocksource tsc-early
[    2.642276] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
0x2670acaa8d1, max_idle_ns: 440795300001 ns
[    2.642462] clocksource: Switched to clocksource tsc

---- >8 ----

Now, this is all very interesting of course, but taking one step back
again: Should I care? Does the result of choices in the hypervisor have
effect on what I can do in dom0 and domU?

-- 
Hans van Kranenburg
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users