Re: Disable IOMMU in Dom0

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

I am fine with adding this functionality only to device tree initially.


It is certainly true that if a DMA-capable device is behind an IOMMU,
then we can skip swiotlb-xen for foreign pages address transactions.
Those addresses will be translated just fine thanks to the IOMMU.
Skipping swiotlb-xen could provide a non-negligible performance
improvement, which is good.

However, you should be aware that your proposal should not be needed for
correctness or to make devices with a 4GB DMA mask work.

Thanks to commit 04085ec1a "xen/arm: fix gnttab_need_iommu_mapping"
foreign pages are added to the Dom0 p2m when mapped to Dom0. swiotlb-xen
translates foreign pages gfn to mfn then uses the mfn to program the
device DMA. The DMA transaction will be accepted by the IOMMU thanks to
the 1:1 GFN<->MFN entry added to Dom0 at the time of mapping.

If the mfn is >4GB and the device requires an address <4GB, then the
dma_capable check at the beginning of xen_swiotlb_map_page fails and the
DMA transaction gets bounced on a swiotlb-xen buffer lower than 4GB.

Am I missing anything?


On Thu, 9 Sep 2021, Roman Skakun wrote:
> Hi Julien,
> Thanks for the clarification!
> 
> I aim towards to prepare implementation for upstream to disable SWIOTLB for 
> IOMMU-protected devices in Dom0.
> To provide this functionality need to add additional binding for each 
> protected device in device-tree.
> After this step, I will also prepare the patch to make ensure that ballooning 
> code prepares all allocations below 4GB.
> 
> We are going to prepare this functionality only for device-tree based system 
> configurations.
> We don't have resources to support ACPI configuration.
> 
> Would you be ok with upstreaming only device-tree configuration?
> 
> Cheers,
> Roman
> 
> ___________________________________________________________________________________________________________________________________________
> From: Julien Grall <julien@xxxxxxx>
> Sent: Wednesday, September 1, 2021 1:22 PM
> To: Roman Skakun <Roman_Skakun@xxxxxxxx>; Stefano Stabellini 
> <sstabellini@xxxxxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx <xen-devel@xxxxxxxxxxxxxxxxxxxx>; Bertrand 
> Marquis <bertrand.marquis@xxxxxxx>; Andrii Anisov
> <Andrii_Anisov@xxxxxxxx>; Roman Skakun <rm.skakun@xxxxxxxxx>; Oleksandr 
> Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>
> Subject: Re: Disable IOMMU in Dom0  
> Hi Roman
> 
> On 01/09/2021 10:59, Roman Skakun wrote:
> >> If you have a setup  where Dom0 is not 1:1 mapped (which is not currently
> >> possible with upstream  Xen but it is possible with cache coloring) and
> >> uses the IOMMU to make  device DMA work like regular DomUs, then passing
> >> XENFEAT_not_direct_mapped  to Linux would make it work. Without
> >> XENFEAT_not_direct_mapped,  Linux would try to use swiotlb-xen which has
> >> code that relies on  Linux being 1:1 mapped to work properly.
> >
> > I'm using 1:1 Dom0.
> > According to your patch series, xen-swiotlb fops will be applied for all
> > devices
> > because XENFEAT_direct_mapped is active, as shown here:
> >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4i
> bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ 
> [elixir[.]bootlin[.]com]
> ><https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4
> ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ 
> [elixir[.]bootlin[.]com]>
> >
> > I agreed, that xen-swiotlb should work correctly, but in my case, I
> > retrieved MFN here:
> >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ib
> LDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]
> ><https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4i
> bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ 
> [elixir[.]bootlin[.]com]>
> > which is greater than 32bit and xen-swiotlb tries to use bounce buffer
> > as expected.
> > It can lead to decrease a performance because I have a long buffer ~4MB.
> >
> > I thought, that we can disable swiotlb fops for devices which are
> > controlled by IOMMU.
> 
> Yes you can disable swiotlb for devices which are controlled by the
> IOMMU. But this will not make your problem disappear, it simply hides
> until it bites you in a more subttle way.
> 
>  From what you wrote, you have a 32-bit DMA capable. So you always need
> to have an address below 4GB. For foreign mapping, there is no guarantee
> the Guest Physical Address will actually be below 4GB.
> 
> Today, the ballooning code only ask Linux to steal *a* RAM page for
> mapping the foreign page. This may or may not be below 4GB depending on
> how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB).
> 
> But that's the current behavior. One of your work colleague is looking
> at avoid to steal RAM page to avoid exhausting the memory. So foreign
> mapping may end up to be a lot higher in memory.
> 
> IOW, you will need to be able to bounce the DMA buffer for your device.
> If you want to avoid bouncing, the proper way would be to rework the
> ballonning code so pages are allocated below 4GB.
> 
> Cheers,
> 
> --
> Julien Grall
> 
>