Re: [Xen-devel] [PATCH] x86/time: update TSC stamp on restore from deep C-state

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, Jan 14, 2020 at 07:36:21PM +0000, Igor Druzhinin wrote:
> If ITSC is not available on CPU (e.g if running nested as PV shim)
> then X86_FEATURE_NONSTOP_TSC is not advertised in certain cases, i.e.
> all AMD and some old Intel processors. In which case TSC would need to
> be restored on CPU from platform time by Xen upon exiting deep C-states.
> 
> As platform time might be behind the last TSC stamp recorded for the
> current CPU, invariant of TSC stamp being always behind local TSC counter
> is violated. This has an effect of get_s_time() going negative resulting
> in eventual system hang or crash.
> 
> Fix this issue by updating local TSC stamp along with TSC counter write.

Thanks! I haven't seen such issue because I've been running the shim
with nomigrate in order to prevent the vTSC overhead.

> 
> Signed-off-by: Igor Druzhinin <igor.druzhinin@xxxxxxxxxx>
> ---
> This caused reliable hangs of shim domains with multiple vCPUs on all AMD
> systems. The problem got also reproduced on bare-metal by artifically
> masking ITSC feature bit. The proposed fix has been verified for both
> cases.
> ---
>  xen/arch/x86/time.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
> index e79cb4d..f6b26f8 100644
> --- a/xen/arch/x86/time.c
> +++ b/xen/arch/x86/time.c
> @@ -955,10 +955,16 @@ u64 stime2tsc(s_time_t stime)
>  
>  void cstate_restore_tsc(void)
>  {
> +    struct cpu_time *t = &this_cpu(cpu_time);
> +
>      if ( boot_cpu_has(X86_FEATURE_NONSTOP_TSC) )
>          return;
>  
> -    write_tsc(stime2tsc(read_platform_stime(NULL)));
> +    t->stamp.master_stime = read_platform_stime(NULL);
> +    t->stamp.local_tsc = stime2tsc(t->stamp.master_stime);
> +    t->stamp.local_stime = t->stamp.master_stime;
> +
> +    write_tsc(t->stamp.local_tsc);

In order to avoid the TSC write (and the likely associated vmexit),
could you instead do:

t->stamp.local_stime = t->stamp.master_stime = read_platform_stime(NULL);
t->stamp.local_tsc = rdtsc_ordered();

I think it should achieve the same as it syncs the local TSC stamp and
times, would avoid the TSC write and slightly simplifies the logic.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel