RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

Hi Dongxiao --

There are two approaches to adding rdtscp support:

1) Faithful full implementation of rdtscp instruction
2) Support pvrtdtscp algorithm

For (1), you would enable the rdtscp bit in cpuid.  Then
on hardware that supports rdtscp, you would do context
switching of TSC_AUX.  On hardware that doesn't support
rdtscp, you would intercept the illegal instruction trap
and emulate the instruction.  (TSC_AUX emulation
could be handled "lazily", no need to do context
switch for that.)

BUT if you look at how TSC_AUX is used by a native
OS**, the OS sets TSC_AUX to each physical CPU number
so an application can easily determine if successive
rdtscp instructions were not executed on the same
processor.  (This was important on older processors
that did not have invariant TSC.)  Unfortunately,
on Xen, this mechanism is worthless and misleading
because the OS believes it is setting TSC_AUX to
a physical CPU number but it is actually setting
it to a virtual CPU number, and the physical CPU
number may change at any time due to scheduling
or migration.  So an app using rdtscp will get a
wrong answer.

As a result, I do NOT recommend (1) and do recommend
that Xen should continue to return zero for the rdtscp
bit in cpuid.

For (2), setting TSC_AUX in __update_vcpu_system_time()
is fine (I think).  On hardware that supports, for HVM
you would need to ensure that the rdtscp instruction
works natively (even though the rdtscp bit in cpuid
is not turned on for the guest).  On hardware that
does not support rdtscp, you would intercept the illegal
instruction trap and call the existing code in
pv_soft_rdtsc().

Does that make sense?

Thanks,
Dan

** I've looked at RHEL5.  Windows actually always
returns 0 for TSC_AUX.

> -----Original Message-----
> From: Xu, Dongxiao [mailto:dongxiao.xu@xxxxxxxxx]
> Sent: Thursday, December 10, 2009 4:22 AM
> To: Dan Magenheimer; Nakajima, Jun; 
> xen-devel@xxxxxxxxxxxxxxxxxxx; Keir
> Fraser
> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
> 
> 
> Hi, Dan, 
>       I am now trying to add the rdtscp support for Xen HVM guest. 
>       I have some questions about your pvrdtscp patch. See below. 
> 
> Dan Magenheimer wrote:
> > Hi Jun --
> > 
> >> But it's possible that multiple domains use the pvrdtscp
> >> algorithm, and the incarnation number is domain specific.
> > 
> > OK, I see.  The code for writing TSC_AUX is in
> > __update_vcpu_system_time() not in context switch.
> 
> Will you modify the place where Hypervisor writes TSC_AUX MSR? 
> In the current pvrdtscp logic, I think this MSR should be 
> written while
> vcpu context switch. Also, this will make HVM support much easier 
> because that MSR would not be modified by Hypervisor time to time.
> 
> > 
> >> We also have the issue when adding RDTSCP support for
> >> HVM guests.
> > 
> > Only if you expose the rdtscp bit via cpuid.  This could
> > certainly be done but, as I said, is probably pointless.
> > (The pvrdtscp algorithm uses the instruction whether or
> > not the rdtscp bit is set in cpuid, since Xen emulates
> > it -- for PV domains only now -- if the physical machine
> > doesn't support the instruction.
> 
> We are planning to add HVM support for RDTSCP, and the 
> behavior for this instruction
> will follow the native way. 
> This caused a problem that RDTSCP instruction in application 
> has different experience
> upon PV and HVM domains. Do you have any comment about this? Thanks!
> 
> Thanks!
> Dongxiao
> 
> > 
> > Dan
> > 
> >> -----Original Message-----
> >> From: Nakajima, Jun [mailto:jun.nakajima@xxxxxxxxx]
> >> Sent: Wednesday, December 09, 2009 10:08 AM
> >> To: Dan Magenheimer; xen-devel@xxxxxxxxxxxxxxxxxxx
> >> Subject: RE: Saving/Restoring IA32_TSC_AUX MSR
> >> 
> >> 
> >> Dan Magenheimer wrote on Wed, 9 Dec 2009 at 08:59:59:
> >> 
> >>> Hi Jun --
> >>> 
> >> 
> >> Dan,
> >> 
> >>> Xen doesn't expose the TSC rdtscp bit so assumes that
> >>> no guests depend on it.  So no save/restore of TSC_AUX
> >>> is necessary.  Xen could provide support for the TSC
> >> 
> >> But it's possible that multiple domains use the pvrdtscp
> >> algorithm, and the incarnation number is domain specific. We
> >> also have the issue when adding RDTSCP support for HVM guests.
> >> 
> >>> rdtscp bit and allow a guest OS to manage TSC_AUX, but
> >>> the existing use of TSC_AUX by Linux would fail to
> >>> provide the desired result across migration, so there's
> >>> little point.  Also the pvrdtscp algorithm now assumes
> >>> that Xen itself is responsible for updating TSC_AUX
> >>> whenever a migration (across physical machines) occurs.
> >>> 
> >>> The #define for write_rdtscp_aux is from Linux source,
> >>> so I didn't change the code and define the constant.
> >>> 
> >>> Dan
> >>> 
> >>>> -----Original Message-----
> >>>> From: Nakajima, Jun [mailto:jun.nakajima@xxxxxxxxx]
> >>>> Sent: Wednesday, December 09, 2009 9:42 AM
> >>>> To: xen-devel@xxxxxxxxxxxxxxxxxxx
> >>>> Cc: Dan Magenheimer
> >>>> Subject: Saving/Restoring IA32_TSC_AUX MSR
> >>>> 
> >>>> 
> >>>> I see the code like (in arch/x86/time.c), and wondering how
> >>>> IA32_TSC_AUX MSR is saved/restored at domain switch time.
> >>>> 
> >>>>     if ( (d->arch.tsc_mode ==  TSC_MODE_PVRDTSCP) &&
> >>>>          boot_cpu_has(X86_FEATURE_RDTSCP) )
> >>>>         write_rdtscp_aux(d->arch.incarnation);
> >>>> 
> >>>> BTW,
> >>>> 
> >>>> include/asm-x86/msr.h
> >>>> #define write_rdtscp_aux(val) wrmsr(0xc0000103, (val), 0)
> >>>> 
> >>>> We should write like wrmsr(MSR_TSC_AUX, (val), 0) by adding
> >>>> +#define MSR_TSC_AUX             0xc0000103 /* Auxiliary TSC */
> >>>> in include/asm-x86/msr-index.h
> >>>> 
> >>>> Thanks,
> >>>> Jun
> >>>> ---
> >>>> Intel Open Source Technology Center
> >>>> 
> >>>> 
> >> 
> >> Jun
> >> ___
> >> Intel Open Source Technology Center
> >> 
> >> 
> >> 
> >> 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel