Re: [Xen-devel] VT-d async invalidation for Device-TLB.

["Xu, Quan" <quan.xu@xxxxxxxxx>]

> From: Zhang, Yang wrote on June 16, 2015 11:07 AM
>> Jan Beulich wrote on 2015-06-15:
> >>>> On 13.06.15 at 16:44, <quan.xu@xxxxxxxxx> wrote:
> >>> On 12.06.15 at 14:47, <JBeulich@xxxxxxxx> wrote:
> >>>>>> On 12.06.15 at 04:40, <quan.xu@xxxxxxxxx> wrote:
> >>>>>>>>> On 10.06.15 at 16:05, <JBeulich@xxxxxxxx> wrote:
> >>>>>>>> On 03.06.15 at 09:49, <quan.xu@xxxxxxxxx> wrote:
> >>>>>>     For Context Invalidation and IOTLB invalidation without
> >>>>>> Device-TLB invalidation, Invalidation Queue flushes synchronous
> >>>>>> invalidation as before(This is a tradeoff and the cost of
> >>>>>> interrupt is
> >>> overhead).
> >>>>>
> >>>>> DMAR_OPERATION_TIMEOUT being 1s, are you saying that you're not
> >>>>> intending to replace the current spinning for the non-ATS case?
> >>>>
> >>>> Yes, we are not intending to replace the current spinning for the
> >>>> non-ATS case.
> >>>
> >>> I'm not really happy about that.
> >>
> >> Jan, could you share more about what you expect? We can enable it
> >> step by step.
> >> Do you want to replace the current spinning for all of queued invalidation?
> >
> > Yes.
> >
> >>>>> Considering that expiring these loops results in panic()s, I would
> >>>>> expect these to become asynchronous _and_ contained to the
> >>>>> affected VM alongside the ATS induced changed behavior. You
> >>>>> talking of overhead - can you quantify that?
> >>>>>
> >>>>
> >>>> I tested it by a Myri-10G Dual-Protocol NIC, which is an ATS device.
> >>>> for an invalidation:
> >>>>  By sync way, it takes about 1.4 ms.
> >>>>  By async way, it takes about 4.3 ms.
> 
> It is a typo for the time: should be 1.4 us and 4.3 us.
> 

yes, it is 1.4 us and 4.3 us.
Thanks.

I also have tested iotlb sync and async invalidation.
For sync way: it takes about 3.2 us.
For async way: it takes about 5.3 us.




> >>>
> >>> What's the theory on why this is? After all, it shouldn't matter how
> >>> the completion of the invalidation gets signaled.
> >>>
> >>
> >> Let me introduce more about how I get these test data.
> >>
> >> For sync way, I get the time by NOW() macro, when I update Queue Tail
> >> Register  (qinval_update_qtail()).
> >> Then get time by NOW() macro in current spinning when it has wrote
> >> the value in the Status Data field to the address specified in the Status
> Address.
> >> The difference of these 2 value is the time of an sync invalidation.
> >>
> >>
> >> For async way, as the previous email, I have introduced the IF bit of
> >> Invalidation Wait Descriptor.
> >> (IF: Indicate the invalidation wait descriptor completion by
> >> generating an invalidation completion event per the programing of the
> >> Invalidation Completion Event Registers.) I have implemented an
> >> interrupt for invalidation completion event.
> >> Also I get the time by NOW() macro, when I update Queue Tail Register
> >> (by qinval_update_qtail()).
> >> Then get time by NOW() macro in invalidation completion event handler.
> >> The difference of these 2 value is the time of an async invalidation.
> >
> > Okay, thanks for the explanation. As this includes the time it takes
> > to deliver and (partly) handle the interrupt, the difference is of
> > course within what one would expect (and also of what would seem
> acceptable).
> 
> The time doesn't include the cost of handling of interrupt. We just record it 
> at
> the entry of interrupt handler. So the cost should bigger than 4.3 us if 
> taking the
> handing cost into consideration. And the costs will much bigger if there are 
> more
> pass-through VMs runs. We can start from ATS case firstly. And apply it to
> non-ATS case later if the async approach doesn't hurt the performance.
> 

Agreed. Now we can implement source code for ATS case first, then we can apply 
it to no-ATS case later.

> >
> > Jan
> 
> 
> Best regards,
> Yang
Quan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel