Re: [Xen-devel] timekeepeing in XEN - request fo comments

[Bartosz Lis <bartoszl@xxxxxxxxxxxxx>]

Jeremy,

Thanks very much for your answers. I think, now, I understand what causes my 
problems.

Dnia wtorek, 10 listopada 2009 o 01:59:13 Jeremy Fitzhardinge napisaÅ(a):
> On 11/09/09 04:21, Bartosz Lis wrote:
> > I have some issues with time synchronization among dom0 and domUs. In my
> > case helped reconfiguring kernel with HZ=1000 for dom0 and HZ=100 for
> > domUs and running ntpd also on domUs. I think, running ntpd on domU is
> > not a good idea,
> 
> Why not?
> 

I run ntpd in dom0, so it is synchronized to internet time. This 
synchronization (done by a daemon) costs in terms of processing power, memory 
usage and network traffic. This cost generated by dom0's ntpd is small and 
acceptable. I think it would be much cheaper to synchronize domUs' clocks to 
dom0 clock using hypercalls rather than running ntpd daemon on each of them.

> > but it works acceptably good as a workaround for my configuration: XEN
> > 3.4.1 + kernel 3.6.31.5 with pv_ops (taken from git last week). When I
> > don't run ntpd on domU clock slowly drifts. I have choosen xen
> > clocksource in both dom0 and domU.
> 
> Drifts with respect to what?

Drifts with respect to dom0's clock, which is synchronized to internet time by 
ntpd.

> 
> If you run all domains without ntp then they should remain in lockstep,
> because they are all being run off the same timebase.  But the xen
> clocksource does have a drift correction factor which can be updated via
> ntpd/adjtimex, so if each kernel updates its factor differently then
> they will appear to drift with respect to each other.

I had run ntpd in domUs for several hours, then I switched it off, and 
observed what happens then. Clocks started drifting from dom0 (which was still 
synchronized to global time by ntpd) and from each other. Yes, it's possible 
that during the period that each domU had ntpd running their drift correction 
factors were set in a different way. Domains had different loads, so ntpd on 
more loaded domain could have had worse accuracy.

> 
> > I'm not the only one to have time synchronization issues. Please write
> > some document specifying how time synchronization between dom0 and domU
> > should be achieved. I have a number of special questions, but probably
> > there are more questions that should be aswered:
> >
> > 1. What is the mechanics behind the clocksource named xen? How does it
> > depend on phisical time sources available on different platforms?
> 
> You can think of it as an augmented tsc; the tsc is used to convey
> fine-grained time information, and a set of additional parameters allow
> the guest to correct for things like tsc skew between cpus.  (And,
> ideally, differing tsc rates, but it is hard to get things perfectly
> monotonic in that case.)
> 
> > 2. How gets domU's clock initialized when domU boots? (I observe that
> > just after domU has started its clock differs from dom0. The difference
> > is about 10s.)
> 
> It reads the Xen wallclock at boot to initialize the time of day.  Xen's
> wallclock time may not be correct/match dom0 if dom0 has failed to
> update it (which is currently the case in pvops dom0 kernels).

Please, tell me if I'm right (I'm 99% sure, but, please, assure me):
* date - displays and sets system clock (dom0 or domU).
* hwclock - displays and sets cmos clock if run in dom0 (in domU doesn't work 
from obvious reasons).

Is there any shell command (or xm subcommand) to display xen wallclock?

> 
> > 3. What are the consequences of not selecting xen clocksource (by
> > selecting one of hpet, acpi_pm, etc.) for dom0 and for domU?
> 
> Xen itself uses and claims ownership of hardware resources like hpet and
> acpi_pm, so they aren't available to guest kernels.  DomU kernels simply
> don't have any hardware access; dom0 is specifically not allowed to
> touch hpet (and I think acpi_pm).
> 
> You could force a kernel to use tsc a clocksource, but that will only
> work correctly if the system has inherently synchronized tscs, and there
> would be discontinuities over suspend/resume/migrate.
> 
> > 4. What are the best practices to achieve clock synchronization?
> 
> ntp is it at present.
> 
> > 5. What has changed recently in timekeeping? What is going to change? Are
> > there any improvements/bugfixes added to redhat or suse kernels? Are
> > these bugfixes planned to be added upstream?
> 
> As far as I know the pvops kernel is state of the art.  It currently
> fails to set the Xen wallclock time when asked (not implemented), but
> aside from that it should be OK.
> 
> The basic principle is that Xen doesn't know all that much about time of
> day, and doesn't really care.  It offers a token get/set time of day
> interface to access the corresponding hardware, but very little beyond
> that.  Time is really considered to be a local matter for domains to
> work out.  If they want to synchronize, then ntp is a perfectly
> reasonable mechanism for doing that.
> 
> Would you like to see any particular changes?  What are your requirements?

No, I think, that's enough from the user's point of view.

Kind regards,

-- 
Bartosz Lis @ Inst. of Information Technology, Technical Univ. of Lodz, Poland
   bartoszl @ ics.p.lodz.pl

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel