Re: Disable swiotlb for Dom0

[Julien Grall <julien@xxxxxxx>]

On 11/08/2021 09:49, Roman Skakun wrote:
> Hi, Julien!
Hi Roman,

> > > I have observed your patch here:
> > >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$
> >  
> > <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org]
> > >
> > > And I collided with the same issue, when Dom0 device trying to use
> > > swiotlb fops for devices which are controlled by IOMMU.
> >
> > The issue Stefano reported was when the dom0 is not direct mapped.
> > However...
> I applied these patches:
> https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578 
> <https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578>
> https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e 
> <https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e>
> to check this more pragmatically.
>
> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb 
> still used (other devices within dom0 used too), when dom0 is direct mapped:
> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> This means, that all devices are using swiotlb-xen DMA fops.
> By the way, before applying this patches, dom0 always used swiotlb-xen 
> fops for initial domain by design.
This is expected because your domain is direct mapped.

> > Any reason to not use  the stable branch for 5.10? I don't know whether
> > your issue will be  fixed there, but the stable branch usually contains a
> > lot of bug fixes (including  security one). So it is a good idea to use
> > it over the first release  of a kernel version.
> Yes, sure, current BSP release based on 5.10 kernel:
> https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1 
> <https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1>
> based on https://github.com/renesas-rcar/linux-bsp 
> <https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2>
> BTW, I specified the wrong kernel URL in the previous massage, sorry.
>
> > > Issue caused in xen_swiotlb_map_page():
> > > ```
> > > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, 
> page_to_phys: 64b1d0000,
> > > xen_phys_to_dma(phys): 64b1d0000
> > > ```
> >
> > I can't seem to find this printk in Linux 5.10. Did you add it yourself?
> Yes, it's my own log.
Ok. Would you be able to provide more information on where the dom0 
memory is allocated and the list of host RAM?
> > This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
> > general, the use of the bounce buffer should be rare. So I would suggest
> > to find out why this is used.
> >
> > Looking at the code, this suggests that one of the following check is false:
> >
> > /*
> >        * If the address happens to be in the device's DMA window,
> > * we can safely return the device addr and not worry about bounce
> > * buffering it.
> > */
> > if (dma_capable(dev, dev_addr, size, true) &&
> > !range_straddles_page_boundary(phys, size) &&
> > !xen_arch_need_swiotlb(dev, phys, dev_addr) &&
> > swiotlb_force != SWIOTLB_FORCE)
> > goto done;
> I checked this earlier and saw that dma_capable(dev, dev_addr, size, 
> true)returns false as expected because
> we got dev_addr equals 64b1d0000 and according to this expression under 
> dma_capable():
> ```
> dma_addr_t end = dev_addr + size - 1;
> return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
> ```
> As result, DMA mask more than 32bit.
> > Let me start with that I agree we should disable swiotlb when we know
> > the device is protected. However, from what you describe, it sounds like
> > the same issue would appear if the IOMMU was disabled.
> Yes, it looks like a potential issue. This means that swiotlb should be 
> worked correctly, when it's needed, agreed.
> But this is also potential improvement, and I presented this idea to 
> discuss and create some patches.
You might be able to remove the Xen swiotlb but I am not sure you will 
be able to remove the swiotlb completely if you have a device that only 
supports 32-bit DMA.
> > Therefore, I think we should first find out why Linux wants to bounce
> > the DMA buffer.
> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver 
> wants to use only 32 bit boundary address, but that's consequence.
Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

> I think, the main reason of using bounce buffer is MFN address, not DMA 
> phys address.
I don't understand this sentence. Can you clarify it?

Cheers,

--
Julien Grall