Re: [Xen-devel] [PATCH] VMX: wbinvd when vmentry under UC

["Liu, Jinsong" <jinsong.liu@xxxxxxxxx>]

Liu, Jinsong wrote:
> Auld, Will wrote:
>>> -----Original Message-----
>>> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
>>> Sent: Monday, November 25, 2013 8:47 AM
>>> To: Andrew Cooper; Liu, Jinsong
>>> Cc: sherry.hurwitz@xxxxxxx; suravee.suthikulpanit@xxxxxxx; Dugger,
>>> Donald D; Dong, Eddie; Nakajima, Jun; Auld, Will; Zhang, Xiantao;
>>> xen- devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx;
>>> zhenzhong.duan@xxxxxxxxxx; keir@xxxxxxx; tim@xxxxxxx Subject: Re:
>>> [PATCH] VMX: wbinvd when vmentry under UC
>>> 
>>>>>> On 25.11.13 at 17:39, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>>> wrote:
>>>> On 25/11/13 16:14, Liu, Jinsong wrote:
>>>>> +/*
>>>>> + * wbinvd is a _very_ time consuming operation, so
>>>>> + * 1. wbinvd ... timer has a good possibility to expire while
>>>>> + *    irq disabled, it then would be delayed until
>>>>> + * 2. ... vmentry back to guest (and irq enalbed), timer
>>>>> interrupt + *    then occurs and drops guest at once;
>>>>> + * 3. drop to hypervisor ... then vmentry and wbinvd again; + *
>>>>> + * This loop will run again and again, until lucky enough wbinvd
>>>>> + * happens not to expire timer and then loop break, usually it
>>>>> would + * occur 10K~60K times, blocking guest 10s~60s. + *
>>>>> + * reprogram timer to avoid dead_like_loop.
>>>>> + */
>>>>> +static inline void uc_wbinvd_and_timer_adjust(void) { +
>>>>> reprogram_timer(0); +    wbinvd();
>>>>> +    reprogram_timer(NOW() + MILLISECS(1));
>>>> 
>>>> Do you have any number of the time delta across the wbinvd() ?
> 
> Generally it depends on how big and how dirty the cache is.
> In my environment (L1/L2/L3 cache: 64/256/20480K, 2.3G processor), it
> varies from 1~3ms: (XEN) wbinvd overhead: 0x6be9ad tsc, 3082209 ns
> ...
> (XEN) wbinvd overhead: 0x26bc68 tsc, 1106378 ns
> 
>>>> 
>>>> As it currently stands, I do not think it is reasonable to
>>>> reprogram the timer like this.
>>> 
>>> Indeed I was wondering too, but didn't get to look in detail at what
>>> consequences would arise from doing this.
>>> 
>>> Jan
>> 
>> Basically, increase the timer setting so that it is unlikely to fire
>> during wbinvd() but still be there as a safeguard. Then reset as you
>> are currently doing after wbinvd().
>> 
>> Will
> 
> Yes. reprogram_timer here just delay timer a little slot, say, 1~2ms.
> I think it's OK, i.e. at any point of wbinvd() operation at
> hypervisor, or any irq disabled area, timer interrupt in fact also
> has good chance to be delayed some time -- however at TIMER_SOFTIRQ,
> all expired thing would be executed, and re-calculated and set next
> time point via reprogram_timer -- that's OK.     

Comments/thoughts about this option?

> 
> Another option is to trust guest. We can remove wbinvd from vmentry
> -- per SDM when guest sets cr0.cd it should explicitly flush cache,
> otherwise itself cannot guarantee cache coherency.  

OK, as you said we don't consider option 2.

Thanks,
Jinsong


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel