Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

Hi everyone,

This reply is intended to clarify the latest test results and bring the
clarifications and other relevant discussion to the xen-devel mailing list.

On 11/1/2023 5:14 AM, Julien Grall wrote:
> 
> 
> On 01/11/2023 08:45, Chuck Zmudzinski wrote:
>> On 11/1/2023 4:27 AM, Julien Grall wrote:
>>> Hi,
>>>
>>> @Stefano, as you pointed out, there is already a thread on xen-users for
>>> this discussion. Could we use this thread for any discussion? This would
>>> make easier to follow.
>>>
>>> Some high level comment below.
>> 
>> I agree to keep the discussion here and not at other places.
> 
> I was meant to suggest the other thread :). But either is fine with me. 
> I just want to avoid avoid multiple seperate threads for the discussion.
> 
>> 
>> I just want to add that the best results for Xen dom0 so far are
>> by implementing Marek's suggestion to disable these two settings
>> in the 6.1.59 kernel config, but leaving everything else the same,
>> including keeping the EXYNOS_IOMMU support enabled:

I got even better results with a small patch in arch/arm/mm to disable
the overwriting of dma_ops with xen_swiotlb_dma_ops for a device when
the dma_ops are already set to use the iommu_ops, otherwise, the
current behavior of overwriting or setting dma_ops for the first time
with the xen_swiotlb_dma_ops is done. This totally fixes the error,
and also allows the HDMI port to work with Linux dom0 on Xen!

> That's good news! I would be interested to hear how this works once you 
> start to have PV backend in dom0 (I expect that the IOMMU will get 
> confused with grant mapping).

I did lots of tests such as building a kernel in a domU with PV block
and network frontend drivers connected to dom0 on the backend while
also building the qemu device model in dom0 using a Linux kernel in dom0
with the aforementioned patch to not overwrite dma_ops if dma_ops is
already set to iommu_ops, and on this Chromebook IOMMU had no confusion
and the feared DMA errors and memory corruption did not materialize!

So I am preparing to submit a patch to lkml to fix the exynos mixer.
on Xen. I just finished testing a version of the patch implemented as
a new config option that is set when support for the device causing
the trouble, the exynos mixer, is present in Linux and EXYNOS_IOMMU
config option is also enabled. I think this is a conservative approach
to add a new config option that can be set for cases like this
Chromebook when the devices that need to use IOMMU are behaving nicely
and do not cause any trouble on Xen. The default will continue to be
that Linux will overwrite IOMMU dma_ops with xen_swiotlb_dma_ops on
Xen unless the new config option is set.

> 
> Also, do you plan to passthrough any of the devices protected by IOMMU?

No. On this Chromebook, the only two devices using IOMMMU in the system on
dom0 with the soon-to-be proposed patch are the exynos-fimd and the
exynos-mixer, which support video for dom0. All other devices in the system
are using the xen_swiotlb_dma_ops. These facts, I think, explain why the
feared DMA errors and IOMMU confusion with the PV drivers for other guests
on the system did not materialize in this case. 

> 
>> 
>> # CONFIG_DRM_EXYNOS_MIXER is not set
>> 
>> Disabling the mixer also makes this unavailable:
>> 
>> # CONFIG_HDMI
>> 
>> With this change, the GPU is working well enough to allow the display
>> manager and an X11 session to run normally on the built-in display of the
>> Chromebook. The Wifi also works well.

As mentioned earlier, these settings worked also, but with the disadvantage
of disabling support for the HDMI port on the Chromebook. My latest tests
indicate we can fix this on Xen without giving up support for the HDMI!

> 
> I saw your other answer about the Wifi not working when the IOMMU is not 
> used. I was about to reply there, but instead I will do it here.
> 
> TBH, I am quite surprised this is the case. The only difference with 
> baremetal would be the RAM regions. Do you know if the Wifi dongle only 
> accept certain physical address?

The Wifi device worked well enough to associate with a Wifi access point
without EXYNOS_IOMMU enabled, it just failed to get IP addresses from
DHCP. I don't know if the exynos Wifi device requires a certain physical
address for the function of acquiring IP addresses from DHCP to work. Marek
might be able to answer that question.

In any case, since the Chromebook works fine on Xen, including Wifi, when
the EXYNOS_IOMMU is used by Linux dom0 as long as Linux does not overwrite
the dma_ops with xen_swiotlb_dma_ops when they had previously been set to
iommu_ops in arch/arm/mm/dma-mapping.c, finding the answer to the problem of
Wifi not working when the IOMMU is not used is not essential because the
default for both exynos systems and multi_v7 arm systems in Linux is to use
the exynos IOMMU when it is available, both on bare metal and on Xen / dom0.

Cheers,

> 
> Cheers,
>