Re: IOMMU problem on Xen dom0 arm (Was: Re: Xen on arm Chromebook seems to cause no display on screen)

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 10/26/2023 5:24 PM, Julien Grall wrote:
> (+ Bertrand, Stefano)
> 
> Hi Chuck,
> 
> Thanks for the report.
> 
> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>> We also have not yet done a thorough analysis of the differences
>>> in the kernel boot logs when booting on the bare metal vs. booting
>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>> cause of this problem after a quick look at the logs.
>> 
>> After a more careful look at the logs, this seems to be the error
>> causing no display when booting as dom0 on Xen:
>> 
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 
>> A little more context from the logs follows (I did not word wrap
>> the log messages to 72 characters because they are easier to read
>> without word wrapping).
>> 
>> On bare metal:
>> 
>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [    2.535938] [drm] 
>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [    2.536139] 
>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [    2.536274] 
>> exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [    2.536493] 
>> exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [    2.536520] 
>> exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>> ...
>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [    3.493686] 
>> Console: switching to colour frame buffer device 170x48
>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [    3.521747] 
>> exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [    3.522831] [drm] 
>> Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>> 
>> The screen works normally in this case.
>> 
>> On Xen as dom0:
>> 
>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [    2.606812] [drm] 
>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [    2.606884] 
>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [    2.606999] 
>> exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 
>> 14450000.mixer lacks support for IOMMU
>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [    2.607044] 
>> exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [    2.607162] 
>> exynos-drm exynos-drm: adev bind failed: -22
>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [    2.607183] 
>> exynos-dp: probe of 145b0000.dp-controller failed with error -22
>> 
>> There is no display on the screen in this case. The backlight
>> does not even come on.
>> 
>> So the error causing no display is probably:
>> 
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 
>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>> is needed for PCI passthrough to domU guests, but not for dom0 to
>> use such devices. 
> 
> I believe that the IOMMU would be required on x86 when using dom0 PVH. 
> PVH is very similar to an Arm guests.
> 
> On Arm, we don't require the IOMMU because not all Arm platforms have 
> all DMA-capable devices protected by an IOMMU. So dom0 will still have 
> its memory direct mapped (i.e. host physical address = guest physical 
> address) to allow DMA in dom0 with limited modification.
> 
> That said, I thik this is a different situation here (see below).
> 
> 
>> So on arm, why is dom0 trying to use IOMMU for
>> the exynos-mixer/exynos-drm when bare metal does not use it?
> 
> Just to confirm, are you using the same kernel, same config when booting 
> on baremetal? If so, from looking at the code, I would expect that the 
> IOMMU is also used on baremetal.
> 
> The check failing is:
> 
> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
> 
> I am not quite too sure why the check implies the IOMMU is not 
> supported. That said, I vaguely recall that Linux will update the DMA 
> ops when running under Xen. Would you be able to print the two values 
> returned ("%pS" should give the symbol)?

I got those values:

[    2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 
0xc0d662dc

I presume you know how to interpret those. The failed test is that they are not 
equal.

For more context of that message from the kernel log:

[    2.551932] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping 
operations
[    2.551981] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
[    2.552094] [drm] ops(priv->dma_dev): 0xc0d018c0, ops(subdrv_dev): 0xc0d662dc
[    2.552109] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] 
*ERROR* Device 14450000.mixer lacks support for IOMMU
[    2.552152] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 
0xc0d97554): -22
[    2.552266] exynos-drm exynos-drm: adev bind failed: -22
[    2.552288] exynos-dp: probe of 145b0000.dp-controller failed with error -22

Tomorrow I will try the hack of removing support for the IOMMU from Linux and
report the results here.

> 
> Anyway, letting dom0 to use the IOMMU is probably a bad idea as even if 
> dom0 memory is direct mapped, grant mappings are not. So you would end 
> up to see random crashes.
> 
> Right now, if Xen doesn't use the IOMMU (e.g. because it was disabled or 
> there is no driver), then the device will be assigned to dom0. We 
> recently had some discussion to hide the IOMMU from dom0. I expect a 
> patch to be on the ML in the near future.
> 
> As a temporary hack, would you be able to compile out the IOMMU driver 
> from Linux and check if it helps using the GPU?
> 
> Looking at the documentation in the Linux tree, I am under the impresion 
> that the Exynos SMMUs are mainly used to avoid allocating large 
> contiguous buffer. So in the longer run, it might be good to understand 
> the performance impact of hiding them from dom0. ...
> 
> ...> 
> Cheers,
> 
> [1] 
> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>