RE: [Xen-devel] [PATCH] TSC scaling for live migration between platforms with different TSC frequecies

["Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>]

John Levon wrote:
> On Fri, Jun 19, 2009 at 10:25:36AM +0800, Zhang, Xiantao wrote:
> 
>> Actually, we had done some optimizations and make most of rdtsc not
>> trap to hypervisor and always keep tsc monotonically increasing in
>> each virtual processor, but it is hard to solve the tsc drift issue
>> between all vcpus.  In our solution, tsc drift may exceeds 100000
>> cycles. You know, TSC drift maybe also exist in real platforms, but
>> maybe not quite large like 1000000 cycles. Do you know what's the
> 
> I'm not sure what you mean by "drift of 1000000 cycles". In what time
> period? Or are you talking about skew (a constant difference between
> TSC values read between CPUs) ? Solaris handles arbitrary skew (I
> think) but that's only accounted for at boot time. A fudge factor
> (tsc_max_delta) accounts for any minor post-calibration deltas.

I mean inter-cpu TSC drift at any time. In our solution, we always keep guest's 
TSC across all vcpus synced to its expected TSC in 1 ms or 0.5 ms.  That is to 
say, all vcpus' TSC monotonically increases, but may generate little drift(10^5 
~10^6 cycles) due to unsync vm exits of vcpus. 


> On HVM or VMWare we don't even try, since we can't possibly know the
> real CPUs skew: the assumption is the VM platform has already done
> this for us. And at least Xen attempts to sync up the physical CPUs.
> Significant drift (where different CPUs are ticking at different
> rates) is bad news, and can easily lead to non-monotonicity. I don't
> know what "significant" means though, unfortunately.

We can guanrantee each vcpu's TSC is increasing monotonically, but there maybe 
some diff between vcpus.  I am not sure 10^5 cycles is significant, but it 
should exceed a stable hardware's drift in general. 


> Finally, a change across all CPUs in the tsc tick rate (so no drift,
> but a sudden change after, say, a migration) is also bad news.
> Solaris used to recalibrate the scale rate once a second, but that
> was removed.

Bad news. 

> All this is HVM/metal code of course.
> 
> regards
> john


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel