[Xen-devel] Clock jumping during live migrations

[Nathan March <nathan@xxxxxx>]

Hi everyone,

Having an issue where the clock on a VM will jump ahead by the timezone 
offset but only certain directions between certain machines.
For example, if I run a "sleep 1; date" loop on the VM and migrate it 
from xen4 to xen7:
Wed Oct 26 21:14:46 PDT 2011
Thu Oct 27 04:19:03 PDT 2011

Trying multiple times with the same dom0's it seems to be consistently 7 
hours (PST offset maybe?) 4 minutes  and 17 seconds:
Thu Oct 27 04:23:27 PDT 2011
Thu Oct 27 11:27:45 PDT 2011

Thu Oct 27 11:43:01 PDT 2011
Thu Oct 27 18:47:18 PDT 2011

If I keep migrating back and forth it keeps going ahead each time, it 
never snaps back. Those were both migrating from xen4 -> xen7, migrating 
the VM back to xen4 does not result in a clock change of any noticeable 
amount.
I've tried using hwclock to make sure the RTC's are in sync with the 
system clock but it didn't seem to make a difference (although they were 
wrong initially, unless xen requires a reboot to pick up a change here).
If I try from xen8 to xen7 it does it, but by a much smaller amount... 
again, going the reverse direction causes no change.
Thu Oct 27 18:50:08 PDT 2011
Thu Oct 27 18:51:57 PDT 2011

and again the time between multiple runs seems to be a consistent 1:50:

Fri Oct 28 16:16:27 PDT 2011
Fri Oct 28 16:18:17 PDT 2011

All the dom0's are identical as far as I'm aware except that some have 
hyperthreading on and some have it off. I've seen this issue between two 
hyperthreading enabled hosts so doesn't seem to be specific to moving 
between them.
I've tested this with the VM using both the xen and the tsc clocksource 
but it made no difference.
Other info for xen4 / xen7:

xen4 ~ # uname -a
Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen4 ~ # uptime
 14:48:33 up 1 day,  6:03,  2 users,  load average: 0.00, 0.01, 0.05
xen4 ~ # xm info
host                   : xen4
release                : 3.0.3
version                : #3 SMP Thu Sep 1 23:39:43 PDT 2011
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 87822
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Thu Sep  1 23:34:06 PDT 2011
xend_config_format     : 4
xen4 ~ # cat /etc/timezone
PST8PDT
xen4 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

xen7 ~ # uname -a
Linux xen7 3.0.3 #2 SMP Tue Aug 23 12:58:39 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen7 ~ # uptime
 14:48:34 up 42 days, 23:38,  3 users,  load average: 0.04, 0.08, 0.13
xen7 ~ # xm info
host                   : xen7
release                : 3.0.3
version                : #2 SMP Tue Aug 23 12:58:39 PDT 2011
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 24329
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Mon Aug 22 11:28:50 PDT 2011
xend_config_format     : 4
xen7 ~ # cat /etc/timezone
PST8PDT
xen7 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

and the test VM i've been using:

nathanxen1 ~ # uname -a
Linux nathanxen1 2.6.32.27-grsec-gt-r2 #1 SMP Wed Aug 10 17:13:56 PDT 
2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/available_clocksource
xen tsc
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Any suggestions?

Unfortunately I don't know when this bug appeared, it used to work fine 
so not sure what I upgraded to give me an idea of when it was introduced.
Thanks,
Nathan

--
Nathan March<nathan@xxxxxx>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel