Re: [Xen-devel] [Patch RFC 00/13] VT-d Asynchronous Device-TLB Flush for ATS Device

[Tim Deegan <tim@xxxxxxx>]

Hi,

At 17:02 +0000 on 07 Oct (1444237344), Xu, Quan wrote:
> __scheme A__
> Q1: - when to take the references?
>     take the reference when the IOMMU entry is _created_;
>     in detail:
>      --iommu_map_page(), or
>      --ept_set_entry() [Once IOMMU shares EPT page table.]
> 
> That leaves one question:
>     -- how to do with hot-plug ATS device pass-through? As the EPT doesn't 
> aware IOMMU will share EPT page table when EPT page table was _created_.

Either you have to take all the references even before the hot-plug,
or you have to do an (interruptable) pass over the EPT table when
needs_iommu becaomes set.  Of those I prefer the first one.  EPT
entries already take references to foreign pages, and I think that
extending that to other kinds should be OK.

> Q2: how do you know when to drop them?
>    - log (or something) when the IOMMU entry is removed/overwritten; and
>    - drop the entry when the flush completes.
>    
>    -- We can add a new page_list_entry structure per page_info, and Add the 
> page with the new page_list_entry structure to per domain page list, when
>     the IOMMU entry is removed/overwritten; and drop the entry when the flush 
> completes. 

As Jan says, I think that might be too much overhead -- a new
page_list_entry would be a pretty big increase to page_info.
(And potentially might not be enough if a page ever needs to be on two
lists? Or can we make sure that never happens?)

Storing the list of to-be-flushed MFNs separately sounds better.  The
question is how to make sure we don't run out of memory to store that
list.  Maybe have a pool allocated that's big enough for sensible use,
and fail p2m updates with -EAGAIN when we run out?

> Q3: what to do about mappings of other domains' memory (i.e. grant and 
> foreign mappings).
>    Between two domains, now I have only one idea to fix this tricky issue -- 
> waitqueue.
>    I.e. grant.
>     For gnttab_transfer /gnttab_unmap , wait on a waitqueue before updating 
> grant flag, until the Device-TLB flush is completed.
>     For grant-mapped, it is safe as the modification of gnttab_unmap.

Waitqueues are probably more heavyweight than is needed here.  The
grant unmap operation can complete immediately, and we can queue the
reference drop (and maybe even some of the maptrack updates) to be
finished when the flush completes.  It could maybe use the same
queue/log as the normal p2m updates.

> __scheme B__
> Q1: - when to take the references?
> 
>     take the reference when the IOMMU entry is _ removed/overwritten_;
>     in detail:
>      --iommu_unmap_page(), or
>      --ept_set_entry() [Once IOMMU shares EPT page table.]

Hmm.  That might be practical, taking a typecount of whatever type the
page happens to be at the time.  I would prefer scheme A though, if it
can be made to work.  It fits better with the usual way refcounts are
used, and it's more likely to be a good long-term fix.  This is the
second time we've had to add refcounting for an edge case, and I
suspect it won't be the last.

Cheers,

Tim.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel