Re: [Xen-devel] FW: VT-d async invalidation for Device-TLB.

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 18.06.15 at 13:31, <quan.xu@xxxxxxxxx> wrote:
>> On June 18, 2015 5:19 PM, <JBeulich@xxxxxxxx> wrote:
>> >>> On 18.06.15 at 10:09, <quan.xu@xxxxxxxxx> wrote:
>> 
>> >> On 16.06.15 09:59, <mailto:JBeulich@xxxxxxxx> wrote:
>> >> >>> On 16.06.15 at 09:59, <yang.z.zhang@xxxxxxxxx> wrote:
>> >> > Jan Beulich wrote on 2015-06-16:
>> >> >>>>> On 16.06.15 at 05:07, <yang.z.zhang@xxxxxxxxx> wrote:
>> >> >>> Jan Beulich wrote on 2015-06-15:
>> >> >>>>>>> On 13.06.15 at 16:44, <quan.xu@xxxxxxxxx> wrote:
>> >> >>>>>> On 12.06.15 at 14:47, <JBeulich@xxxxxxxx> wrote:
>> >> >>>>>>>>> On 12.06.15 at 04:40, <quan.xu@xxxxxxxxx> wrote:
>> >
>> >> >
>> >> > which one? 1.4us for sync case and 4.3us for async case?
>> >>
>> >> The difference between the two (i.e. why is the async variant three
>> >> times as long).
>> >>
>> >
>> > I have tested iotlb async/sync invalidation to get another data. Also
>> > I disabled 'P State' / 'C State' in bios.
>> > For async invalidation: about 2.67 us.
>> > For sync invalidation: about 1.28 us.
>> >
>> > I also tested VCPU_KICK_SOFTIRQ irq cost.
>> >  When hypervisor calls cpu_raise_softirq(..VCPU_KICK_SOFTIRQ) to raise
>> > an VCPU_KICK_SOFTIRQ irq, and vcpu_kick_softirq() is the
>> > VCPU_KICK_SOFTIRQ interrupt handler.
>> > I got the cost between cpu_raise_softirq(..VCPU_KICK_SOFTIRQ) and
>> > vcpu_kick_softirq(). It is about 1.21 us.
>> >
>> > I think the difference is interrupt cost between async invalidation
>> > and sync invalidation.
>> 
>> Which contradicts what I think Yang said in an earlier reply.
>> 
> Talked with Yang who is confused at what he said. :(
> Could you share more?

Upon me asking, he specifically indicated the numbers do _not_
include interrupt handling overhead.

>> > 2.67us is almost ideal for async invalidation cost. There are 4
>> > reasons to cost much more time:
>> >    1.  If enable 'P State' / 'C State' in bios.
>> >    2.  Hypervisor is running in No-root mode.
>> >    3.  The time doesn't include the cost of handling of interrupt. I
>> > just record it at the entry of interrupt handler.
>> >    4.  More pass-through VMs runs.
>> >
>> > So there are maybe some performance issues when we replace the current
>> > spinning for the non-ATS case.
>> > We can start from ATS case firstly, And apply it to non-ATS case later
>> > if the async approach performance is acceptable.
>> > Jan, Do you agree with this?
>> 
>> No, I'm still not convinced that leaving the non-ATS case alone initially is 
> the right
>> approach. But maybe I'm the only one?
>> 
> I hope for someone else to give some comments.
> 
> I tried to replace the current spinning for the non-ATS case, but Xen 
> crashed.
> Based on dmesg, it seems that VT-d is enabled before enabling IO-APIC IRQs. 
> 
> I can send out two serious of patch:
> 1st: VT-d async invalidation for ATS case.
> 2nd: VT-d async invalidation for non-ATS case.
> 
> 
> I think the 1st serious of patch is high priority, as it is not correct to 
> spin 1 second for ATS case. I can implement these source code and send out 
> ASAP.
> 2nd serious of patch is low priority, as it's optimization. Also I can 
> provide a serious of patch to fix it later.

That's fine as long as the second series won't arrive only months
later.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel