Re: Disable IOMMU in Dom0

[Roman Skakun <Roman_Skakun@xxxxxxxx>]

Hi, Stefano!

Thanks for responding!

> If you have a setup where Dom0 is not 1:1 mapped (which is not currently
> possible with upstream Xen but it is possible with cache coloring) and
> uses the IOMMU to make device DMA work like regular DomUs, then passing
> XENFEAT_not_direct_mapped to Linux would make it work. Without
> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
> code that relies on Linux being 1:1 mapped to work properly.

I'm using 1:1 Dom0.

According to your patch series, xen-swiotlb fops will be applied for all devices 

because XENFEAT_direct_mapped is active, as shown here:
https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c#L56

I agreed, that xen-swiotlb should work correctly, but in my case, I retrieved MFN here:

https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c#L366

which is greater than 32bit and xen-swiotlb tries to use bounce buffer as expected.
It can lead to decrease a performance because I have a long buffer ~4MB.

I thought, that we can disable swiotlb fops for devices which are controlled by IOMMU.

Cheers,

---

From: Stefano Stabellini <sstabellini@xxxxxxxxxx>
Sent: Wednesday, September 1, 2021 12:50 AM
To: Roman Skakun <Roman_Skakun@xxxxxxxx>
Cc: Stefano Stabellini <sstabellini@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx <xen-devel@xxxxxxxxxxxxxxxxxxxx>; Bertrand Marquis <bertrand.marquis@xxxxxxx>; Andrii Anisov <Andrii_Anisov@xxxxxxxx>; Roman Skakun <rm.skakun@xxxxxxxxx>
Subject: Re: Disable IOMMU in Dom0

 

On Tue, 31 Aug 2021, Roman Skakun wrote:
> Hi, Stefano!
>
> I have seen your negotiation of disabling xen-swiotlb for devices which are controlled by IOMMU in Dom0:
> https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/__;!!GF_29dbcQIUBPA!hl4_2pWlI3NQmAkwh4uvJFu7NhcOay9O9bIJRr7eOOMKD_I2z8MPbwnSXpNmqU6F$ [patchwork[.]kernel[.]org]
>
> As I was thinking to create a common implementation because I have the issue
> when device controlled by IOMMU using xen-swiotlb fops and bounce buffer as the result.
> https://urldefense.com/v3/__https://lore.kernel.org/xen-devel/060b5741-922c-115c-7e8c-97d8aa5f46f4@xxxxxxx/T/__;!!GF_29dbcQIUBPA!hl4_2pWlI3NQmAkwh4uvJFu7NhcOay9O9bIJRr7eOOMKD_I2z8MPbwnSXp1TdIa7$ [lore[.]kernel[.]org]
>
> Do you have any future plans to finish implementation for upstream?

Hi Roman,

The feature is already upstream in Linux as f5079a9a2, and the new
feature flags are XENFEAT_direct_mapped and XENFEAT_not_direct_mapped.

If you have a setup where Dom0 is not 1:1 mapped (which is not currently
possible with upstream Xen but it is possible with cache coloring) and
uses the IOMMU to make device DMA work like regular DomUs, then passing
XENFEAT_not_direct_mapped to Linux would make it work. Without
XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
code that relies on Linux being 1:1 mapped to work properly.

Is that the same problem that you have, or is dom0 1:1 mapped in your
case? If dom0 is 1:1 mapped then swiotlb-xen should work regardless of
whether the IOMMU is enabled or disabled.

I hope this helps.

Cheers,

Stefano