RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR

["Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>]

Dan Magenheimer wrote:
> Well, although it might be nice to be able to use
> rdtscp and TSC_AUX to determine pcpu/vcpu/pnode/vnode
> information, I think Jeremy and Jan convinced me in
> another thread a couple of months ago that in userland:
> 
> x = vgetcpu()
> do_other_stuff();
> y = vgetcpu()
> 
> if x==1 and y==2, there's no way to determine that
> do_other_stuff() was executed on cpu 1 vs cpu 2,
> or (though unlikely) even on cpu 3.  And if
> x==y==4, there's  no guarantee that do_other_stuff()
> is executed on cpu 4.
> 
> If this is true the only safe use of TSC_AUX is for
> its originally designed intent: To determine if two
> successive rdtscp instructions were or were not
> executed on the same processor.  Since this cannot
> be guaranteed in a VM, that's a reasonable argument
> that TSC_AUX shouldn't be exposed at all (meaning the
> rdtscp bit in cpuid should be turned off by Xen).

Why do you think this is the design intent of this instruction ?  

For guest NUMA support,  it should be a must to pin each vcpu of one VM to some 
logical proceossors which belong to one specific node(disable vcpu migration 
between nodes), I think, otherwise, virutal numa may suffer from performance 
loss.  For example, in a numa system which has two nodes and each node has 4G 
memory and 8 logical processors. And in this Xen-configured system,  if we 
carete a VM with 2 G memory with4  vcpu support,  Xen system may allocate 1 G 
memory from physical node 0 and another 1 G memory from physical node 1.  And 
in this case, if we virtualize numa for this VM, vcpu0 and vcpu1 can be 
assinged to virtual node0 , vcpu2 and vcpu3 can be configured for virtual 
node1, certainly, we also can safely pin vcpu0 and vpcu1 to the physical 
node0's 8 locial processors and accordingly pin vcpu2 and vcpu3 to the physical 
node1's 8 physical processors.  Since virtual TSC_AUX is virtualized for each 
vcpu, and the value is saved/restored for the vcpu when its migration occurs, 
so if one application always runs on a virtual processors, it should get a 
fixed value when it calls vgetcpu, envn if this vcpu often migrates among 
logical processors of one node.   

Back to this topic, in all,  we can't mix the virtual  TSC_AUX of guest with 
the host's TSC_AUX.  If switch to HVM's vcpu context,  load this vcpu's virtual 
 TSC_AUX_MSR to physical TSC_AUX_MSR, and when it is sheduled out,  host's 
TSC_AUX_MSR(which maybe used for pv guests) is loaded.  


> True, as long as the information is ONLY used
> heuristically to obtain pcpu/vcpu/pnode/vnode info,
> and no guarantee of correctness is implied or expected,
> it might be useful some of the time.
> 
> But frankly, if "performance sucks" when the heuristic
> fails due to the fact that the app is running on
> a VM instead of native OS, I'd see that as a problem
> and suggest the proper way to fix that is to define
> more App-to-Xen ABIs so that the app can get the
> real information, not a heuristic.  Which also argues
> for Xen leaving the rdtscp bit in cpuid turned off
> 
> Dan
> 
>> -----Original Message-----
>> From: Nakajima, Jun [mailto:jun.nakajima@xxxxxxxxx]
>> Sent: Friday, December 11, 2009 12:30 PM
>> To: Jeremy Fitzhardinge; Dan Magenheimer
>> Cc: Keir Fraser; Zhang, Xiantao; Xu, Dongxiao;
>> xen-devel@xxxxxxxxxxxxxxxxxxx; Dugger, Donald D
>> Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
>> 
>> 
>> Jeremy Fitzhardinge wrote on Fri, 11 Dec 2009 at 10:50:29:
>> 
>>> On 12/11/09 10:35, Dan Magenheimer wrote:
>>>>> However, the vcpu number is definitely useful to usermode apps,
>>>>> so they can get some idea how they're moved between (v)cpus.  I
>>>>> don't think it will matter to them that it isn't pcpu.
>>>>> 
>>>> My point is that an app running on native Linux can
>>>> safely assume that, if TSC_AUX==3 at time T1 and
>>>> TSC_AUX is still 3 at time T2,it is running
>>>> on the same processor and the same node at both T1
>>>> and T2.  In a virtual environment it cannot even
>>>> assume it is running on the same machine.
>>>> Further if the app sees that TSC_AUX==2 at time T3
>>>> and TSC_AUX==3 at time T4, on native Linux it
>>>> can safely assume that it is running on a different
>>>> processor.  While rarer, in a virtual environment,
>>>> this may also be a false assumption.
>>>> 
>>>> That's why I say the information is misleading.
>>>> 
>>>  Sure, but that info is, at best, of heuristic value, and won't
>>> cause any correctness problems if it is wrong.  The performance may
>>> suck, but that's part of the larger problem of running NUMA-aware
>>> code in a virtual environment. 
>>> 
>> 
>> And to utilize various NUMA optimizations in the kernel/apps
>> in the guest, we need "the virtual numa info bears some vague
>> resemblance to the real topology" (from Jeremy's email) with
>> the vcpus bound to the CPU/node.
>> 
>> I understand that enabling RDTSCP in HVM will disable the
>> pvrdtscp algorithm if used by the kernel. One way is to mask
>> off the feature in CPUID (by default). Then kernel won't use it.
>> 
>> Jun
>> ___
>> Intel Open Source Technology Center


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel