[Xen-devel] Re: Bizarre pv kernel ultra-high frequency rdtsc?!?

[Keir Fraser <keir.fraser@xxxxxxxxxxxxx>]

Maybe trace what the guest thinks system time is, compared with Xen itself.
If they get out of whack, you can get nastiness with guest trying to set
timers it thinks are in the future but Xen thinks are in the past, and end
up in a loop.

I still think this is due to the big +ve 64-bit vTSC values looking like -ve
values in pv_soft_rdtsc() and accidentally triggering the
TSC-hasn't-gone-forwards check. As soon as system time goes awry between
guest and Xen, I think you are in trouble. There could be more 'false -ves'
like that elsewhere -- so just add BIG_OFFSET rather than subtract it. In
practice your TSC values are never going to have the 64th bit set (would
take 30 years for TSC to tick that high assuming a 10GHz tick rate), so
adding BIG_OFFSET is a more reasonable synthetic test.

 -- Keir

On 21/11/2009 17:31, "Dan Magenheimer" <dan.magenheimer@xxxxxxxxxx> wrote:

> One big clue: Looking at /proc/interrupts inside
> the PV guest, the number of timer0 interrupts is
> about 300K/second.
> 
> Not remembering well how timer interrupts are handled
> in a PV guest... could this high frequency be happening
> because the Linux-side PV code is setting a timer
> or because the Xen-side interrupt delivery code is
> getting confused?
> 
>> -----Original Message-----
>> From: Dan Magenheimer
>> Sent: Friday, November 20, 2009 4:45 PM
>> To: Keir Fraser; Jeremy Fitzhardinge
>> Cc: Xen-Devel (E-mail)
>> Subject: Bizarre pv kernel ultra-high frequency rdtsc?!?
>> 
>> 
>> Hi Jeremy/Keir (and any other PV time experts out there) --
>> 
>> Working on tsc_mode stuff I've run into a roadblock where
>> there is some time-related interaction between xen and a
>> PV kernel that I don't understand.  I'm hoping you
>> might provide a clue.  There's also a reasonable chance
>> that this might be uncovering a significant bug that's
>> been around awhile, but never noticed as other than
>> a barely noticeable vague slowdown because rdtsc was
>> unemulated (and "fast").
>> 
>> The problem:
>> 
>> In order to preserve TSC across save/restore/migrate, I
>> have implemented a "tsc offset" (and also a "tsc scale"
>> but that isn't used yet).
>> 
>> The result is that the PV kernel starts doing rdtsc's at
>> a VERY high frequency (1 MILLION / sec).  I suspect this
>> may be a variation of what Jeremy reported at one point
>> when emulated rdtsc was first in-tree, but seemed to go away.
>> 
>> By adding some debug code (and confirmed with xenctx)
>> I can see that the millions of rdtsc's are half in
>> get_nsec_offset() and half in do_gettimeofday() (presumably
>> inlined from get_usec_offset()).  This is a 32-bit 2.6.18-based
>> PV kernel, not upstream.  Poring through the 2.6.18 PV time
>> code, I can find several places where an essentially infinite
>> loop might happen if the version fields are wacko, but
>> none where the timestamp contents make any difference
>> in control flow, so don't see how modifying these
>> values (by adding the offset) could cause a behavioral
>> change in Linux, but obviously a big change is happening!
>> 
>> I can reproduce the problem with a very simple patch
>> on xen-unstable that adds a fake fixed offset in the
>> three places I add the "tsc offset", see attached.
>> By changing BIG_OFFSET to 0, in this patch, the
>> frequency of rdtsc's becomes normal again.
>> 
>> Suspecting some interaction with wallclock time, I
>> tried shutting off ntpd and with/without independent
>> wallclock in the PV guest.  No difference.
>> 
>> I also added debug code to see if the Xen-side code
>> was churning through version numbers... it is not.
>> 
>> Any ideas?  (And, sorry, I know you're on a trans-
>> hemisphere trip right now.)
>> 
>> Thanks,
>> Dan
>> 
>> diff -r bec27eb6f72c xen/arch/x86/time.c
>> --- a/xen/arch/x86/time.c Sat Nov 14 10:32:59 2009 +0000
>> +++ b/xen/arch/x86/time.c Fri Nov 20 16:58:18 2009 -0500
>> @@ -813,6 +813,8 @@ s_time_t get_s_time(void)
>>  #define version_update_begin(v) (((v)+1)|1)
>>  #define version_update_end(v)   ((v)+1)
>>  
>> +#define BIG_OFFSET 10000000000ULL
>> +
>>  static void __update_vcpu_system_time(struct vcpu *v, int force)
>>  {
>>      struct cpu_time       *t;
>> @@ -827,7 +829,7 @@ static void __update_vcpu_system_time(st
>>  
>>      /* Don't bother unless timestamps have changed or we are
>> forced. */
>>      if ( !force && (u->tsc_timestamp == (v->domain->arch.vtsc
>> -                                         ? t->stime_local_stamp
>> +                                         ?
>> t->stime_local_stamp - BIG_OFFSET
>>                                           : t->local_tsc_stamp)) )
>>          return;
>>  
>> @@ -835,8 +837,8 @@ static void __update_vcpu_system_time(st
>>  
>>      if ( v->domain->arch.vtsc )
>>      {
>> -        _u.tsc_timestamp     = t->stime_local_stamp;
>> -        _u.system_time       = t->stime_local_stamp;
>> +        _u.tsc_timestamp     = t->stime_local_stamp - BIG_OFFSET;
>> +        _u.system_time       = t->stime_local_stamp - BIG_OFFSET;
>>          _u.tsc_to_system_mul = 0x80000000u;
>>          _u.tsc_shift         = 1;
>>      }
>> @@ -1598,6 +1600,8 @@ void pv_soft_rdtsc(struct vcpu *v, struc
>>  
>>      spin_unlock(&v->domain->arch.vtsc_lock);
>>  
>> +    now -= BIG_OFFSET;
>> +
>>      regs->eax = (uint32_t)now;
>>      regs->edx = (uint32_t)(now >> 32);
>>  }
>> 
>> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel