[Xen-devel] [PATCH RFC 0/6] x86/time: PVCLOCK_TSC_STABLE_BIT support

[Joao Martins <joao.m.martins@xxxxxxxxxx>]

Hey!

I've been working on pvclock vdso support on Linux guests, and came
across Xen lacking support for PVCLOCK_TSC_STABLE_BIT flag which is
required for vdso as of the latest pvclock ABI shared with KVM.

In addition, I've found some problems which aren't necessarily visible
to kernel as the pvclock algorithm in linux keeps the highest pvti
time read among all cpus. But as is, a process using vdso gettimeofday
observes a significant amount of warps (i.e. time going backwards) and
it could be due to 1) time calibration skew in xen rendezvous
algorithm, 2) xen clocksource not in sync with TSC and 3) in
situations when guests unaware of VCPUS moving to a different PCPU.
The warps are seen more frequently on PV guests (potentially because
vcpu time infos are only updated when guest is in kernel mode, and
perhaps lack of tsc_offset?), and in rare ocasions on HVM guests. And
it is worth noting that with guests VCPUs pinned, only PV guests see
these warps. But on HVM guests specifically: such warps only occur
when one of guest VCPUs is pinned to CPU0.

PVCLOCK_TSC_STABLE_BIT is the flag telling the guest that the
vcpu_time_info (pvti) are monotonic as seen by any CPU,
and supporting it could help fixing the issue mentioned above. This
series aims to propose a solution to that and it's divided as
following:

        * Patch 1: Adds the missing flag field to vcpu_time_info.
        * Patch 2: Adds a new clocksource based on TSC
        * Patch 3, 4: Cleanups for patch 5
        * Patch 5: Implements the PVCLOCK_TSC_STABLE_BIT.
        * Patch 6: Same as 5 before but for other platform timers

I have some doubts on the correctness of Patch 6 but was the only
solution I found for supporting PVCLOCK_TSC_STABLE_BIT when using
other clocksources (i.e. != tsc). The test was running time-warp-test,
that constantly calls clock_gettime/gettimeofday on every CPU. It
measures a delta with the previous returned value and mark a warp if
it's negative. I measured it during periods of 1h and 6h and check how
many warps and their values (alongside the amount of time skew).
Measurements are included in individual patches.

Note that most of the testing has been done with Linux 4.4 in which
these warps/skew could be easily observed as vdso would use each vCPU
pvti. Though Linux 4.5 will change this behaviour and use only the
vCPU0 pvti though still requiring PVCLOCK_TSC_STABLE_BIT usage.

I've been using it for a while in the past weeks with no issues so far
though still requires testing on bad TSC machines. But I would like to
get folks comments/suggestions if this is the correct approach in
implementing this flag.

Thanks!
Joao

Joao Martins (6):
  public/xen.h: add flags field to vcpu_time_info
  x86/time: implement tsc as clocksource
  x86/time: streamline platform time init on plt_init()
  x86/time: refactor read_platform_stime()
  x86/time: implement PVCLOCK_TSC_STABLE_BIT
  x86/time: convert counter to tsc for non-tsc clocksource

 xen/arch/x86/time.c      | 141 +++++++++++++++++++++++++++++++++++++++++------
 xen/include/public/xen.h |   6 +-
 2 files changed, 128 insertions(+), 19 deletions(-)

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel