Re: [Xen-devel] [PATCH] VMX: wbinvd when vmentry under UC

["Liu, Jinsong" <jinsong.liu@xxxxxxxxx>]

Auld, Will wrote:
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
>> Sent: Monday, November 25, 2013 8:47 AM
>> To: Andrew Cooper; Liu, Jinsong
>> Cc: sherry.hurwitz@xxxxxxx; suravee.suthikulpanit@xxxxxxx; Dugger,
>> Donald D; Dong, Eddie; Nakajima, Jun; Auld, Will; Zhang, Xiantao;
>> xen- devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx;
>> zhenzhong.duan@xxxxxxxxxx; keir@xxxxxxx; tim@xxxxxxx Subject: Re:
>> [PATCH] VMX: wbinvd when vmentry under UC 
>> 
>>>>> On 25.11.13 at 17:39, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>> wrote:
>>> On 25/11/13 16:14, Liu, Jinsong wrote:
>>>> +/*
>>>> + * wbinvd is a _very_ time consuming operation, so
>>>> + * 1. wbinvd ... timer has a good possibility to expire while
>>>> + *    irq disabled, it then would be delayed until
>>>> + * 2. ... vmentry back to guest (and irq enalbed), timer interrupt
>>>> + *    then occurs and drops guest at once;
>>>> + * 3. drop to hypervisor ... then vmentry and wbinvd again; + *
>>>> + * This loop will run again and again, until lucky enough wbinvd
>>>> + * happens not to expire timer and then loop break, usually it
>>>> would + * occur 10K~60K times, blocking guest 10s~60s.
>>>> + *
>>>> + * reprogram timer to avoid dead_like_loop.
>>>> + */
>>>> +static inline void uc_wbinvd_and_timer_adjust(void) { +   
>>>> reprogram_timer(0); +    wbinvd();
>>>> +    reprogram_timer(NOW() + MILLISECS(1));
>>> 
>>> Do you have any number of the time delta across the wbinvd() ?

Generally it depends on how big and how dirty the cache is.
In my environment (L1/L2/L3 cache: 64/256/20480K, 2.3G processor), it varies 
from 1~3ms:
(XEN) wbinvd overhead: 0x6be9ad tsc, 3082209 ns
...
(XEN) wbinvd overhead: 0x26bc68 tsc, 1106378 ns

>>> 
>>> As it currently stands, I do not think it is reasonable to reprogram
>>> the timer like this.
>> 
>> Indeed I was wondering too, but didn't get to look in detail at what
>> consequences would arise from doing this.
>> 
>> Jan
> 
> Basically, increase the timer setting so that it is unlikely to fire
> during wbinvd() but still be there as a safeguard. Then reset as you
> are currently doing after wbinvd().  
> 
> Will

Yes. reprogram_timer here just delay timer a little slot, say, 1~2ms. I think 
it's OK, i.e. at any point of wbinvd() operation at hypervisor, or any irq 
disabled area, timer interrupt in fact also has good chance to be delayed some 
time -- however at TIMER_SOFTIRQ, all expired thing would be executed, and 
re-calculated and set next time point via reprogram_timer -- that's OK.

Another option is to trust guest. We can remove wbinvd from vmentry -- per SDM 
when guest sets cr0.cd it should explicitly flush cache, otherwise itself 
cannot guarantee cache coherency.

Thoughts?

Thanks,
Jinsong
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel