[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Disable swiotlb for Dom0

On 11/08/2021 09:49, Roman Skakun wrote:
Hi, Julien!

Hi Roman,

> I have observed your patch here:
> And I collided with the same issue, when Dom0 device trying to use
> swiotlb fops for devices which are controlled by IOMMU.

The issue Stefano reported was when the dom0 is not direct mapped.

I applied these patches:
https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578 <https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578> https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e <https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e>
to check this more pragmatically.

Also, I added the log in xen_swiotlb_detect() and can see that swiotlb still used (other devices within dom0 used too), when dom0 is direct mapped:

[    1.870363] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb [    1.878352] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb [    1.886309] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb

This means, that all devices are using swiotlb-xen DMA fops.
By the way, before applying this patches, dom0 always used swiotlb-xen fops for initial domain by design.

This is expected because your domain is direct mapped.

Any reason to not use  the stable branch for 5.10? I don't know whether
your issue will be  fixed there, but the stable branch usually contains a
lot of bug fixes (including  security one). So it is a good idea to use
it over the first release  of a kernel version.

Yes, sure, current BSP release based on 5.10 kernel:
https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1 <https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1> based on https://github.com/renesas-rcar/linux-bsp <https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2>
BTW, I specified the wrong kernel URL in the previous massage, sorry.

> Issue caused in xen_swiotlb_map_page():
> ```
> dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400,
page_to_phys: 64b1d0000,
> xen_phys_to_dma(phys): 64b1d0000
> ```

I can't seem to find this printk in Linux 5.10. Did you add it yourself?

Yes, it's my own log.

Ok. Would you be able to provide more information on where the dom0 memory is allocated and the list of host RAM?

This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
general, the use of the bounce buffer should be rare. So I would suggest
to find out why this is used.

Looking at the code, this suggests that one of the following check is false:

       * If the address happens to be in the device's DMA window,
* we can safely return the device addr and not worry about bounce
* buffering it.
if (dma_capable(dev, dev_addr, size, true) &&
!range_straddles_page_boundary(phys, size) &&
!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
swiotlb_force != SWIOTLB_FORCE)
goto done;

I checked this earlier and saw that dma_capable(dev, dev_addr, size, true)returns false as expected because we got dev_addr equals 64b1d0000 and according to this expression under dma_capable():

dma_addr_t end = dev_addr + size - 1;
return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
As result, DMA mask more than 32bit.
Let me start with that I agree we should disable swiotlb when we know
the device is protected. However, from what you describe, it sounds like
the same issue would appear if the IOMMU was disabled.

Yes, it looks like a potential issue. This means that swiotlb should be worked correctly, when it's needed, agreed. But this is also potential improvement, and I presented this idea to discuss and create some patches.

You might be able to remove the Xen swiotlb but I am not sure you will be able to remove the swiotlb completely if you have a device that only supports 32-bit DMA.

Therefore, I think we should first find out why Linux wants to bounce
the DMA buffer.

We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver wants to use only 32 bit boundary address, but that's consequence.

Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

I think, the main reason of using bounce buffer is MFN address, not DMA phys address.

I don't understand this sentence. Can you clarify it?


Julien Grall



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.