Re: [PATCH] xen/arm: Enhance IPMMU-VMSA driver robustness and debug output

[Julien Grall <julien@xxxxxxx>]

On 30/06/2025 07:37, Jahan Murudi wrote:
> Hi Julien,
>
> On 25/06/2025 16:53, Julien Grall wrote:
>
> > Hi Jahan,
> > > > > +    dsb(sy);
> > > > Any clue why Linux (mainline) does not do that?
> > One process remark, we typically comment inline rather than pasting a quote and 
> > replying at the top of the e-mail.
>   Thanks for the style note - I'll follow the inline commenting convention 
> moving forward.
>
> > > The implementation writel() which contains an implicit dsb(st) which likely 
> > > sufficient for Linux for its Stage-1 IOMMU usage where CPU and IOMMU 
> > > interactions are coherent.
> > > However, Xen uses the IPMMU as a Stage-2 IOMMU for non-coherent DMA operations (such 
> > > as PCIe passthrough), requiring the stronger dsb(sy) to ensure writes fully propagate 
> > > to the IPMMU >>hardware before continuing.
> > I don't follow. Are you saying the IPMMU driver in Linux doesn't non-coherent 
> > DMA operations?
> Let me clarify my understanding:  In native Linux, the IOMMU works at stage-1 (VA -> PA) and typically assumes coherency between CPU and IOMMU. The implicit dsb(st) in writel() is enough there. But in Xen, we use this as stage-2 (GPA -> HPA) for cases like PCI passthrough where devices might be non-coherent. 

I understand for the PCI passthrough, Xen will be using stage-2, so in 
theory the stage-1 could be used by the guest OS. But ultimately, this 
is the same PCI device behind. So if it is not coherent, it should be 
for both stages. Do you have any pointer to the documentation that would 
state otherwise?
> We might need stronger barrier dsb(sy) in xen because: 1) We can't 
assume the TLB walker is coherent for stage -2
Why would the TLB walker coherent for stage-2 but not stage-1? Any 
pointer to the documentation?
Note, I just noticed that IOMMU_FEAT_COHERENT_WALK is not set for the 
IPMMU. So the "dsb sy" is coherent. However, I find doubful an IOMMU 
would have a difference of coherency between two stages. So maybe we 
should set the flag either unconditionally or based on a register.
> and we must also prevent(minimise) any DMA operations during TLB 
invalidation( observed some IPMMU hardware limitations in the 
documentation) .
I don't understand what you wrote in parentheses. But isn't it what you 
wrote all true for stage-1?
Cheers,

--
Julien Grall