[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH 0/2] Time-related fixes for migration

On 03/31/2014 10:41 AM, Tian, Kevin wrote:
* The second patch keeps TSCs synchronized across VPCUs after save/restore.
Currently TSC values diverge after migration because during both save and
we calculate them separately for each VCPU and base each calculation on
newly-read host's TSC.

The problem can be easily demonstrated with this program for a 2-VCPU guest
(I am assuming here invariant TSC so, for example,
tsc_mode="always_emulate" (*)):

main(int argc, char* argv[])

   unsigned long long h = 0LL;
   int proc = 0;
   cpu_set_t set;

   for(;;) {
     unsigned long long n = __native_read_tsc();
     if(h && n < h)
         printf("prev 0x%llx cur 0x%llx\n", h, n);
     proc = (proc + 1) & 1;
     CPU_SET(proc, &set);
     if (sched_setaffinity(0, sizeof(cpu_set_t), &set)) {

     h = n;

what's the backward drift range from above program? dozens of cycles?
hundreds of cycles?

For "raw" difference (i.e. TSC registers themselves) it's usually tens of thousands, sometimes more (and sometimes *much* more).

For example, here are outputs of 'xl debug-key v' before and after a migrate for a 2p guest:

root@haswell> xl dmesg |grep Offset
(XEN) TSC Offset = ffffff54e63f1cab
(XEN) TSC Offset = ffffff54e63f1cab
(XEN) TSC Offset = ffffff6cb0a59ea9
(XEN) TSC Offset = ffffff6cb0a566ae

For guest's view, taking into account, for example, the fact that sched_affinity() takes quite some time, it's a few hundreds of cycles.

And obviously as you inclrease number of VCPUs (and I think guest memory as well) the largest difference grows further.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.