Re: IOMMU problem on Xen dom0 arm (Was: Re: Xen on arm Chromebook seems to cause no display on screen)

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 10/26/2023 5:24 PM, Julien Grall wrote:
> (+ Bertrand, Stefano)
> 
> Hi Chuck,
> 
> Thanks for the report.
> 
> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>> We also have not yet done a thorough analysis of the differences
>>> in the kernel boot logs when booting on the bare metal vs. booting
>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>> cause of this problem after a quick look at the logs.
>> 
>> After a more careful look at the logs, this seems to be the error
>> causing no display when booting as dom0 on Xen:
>> 
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 
>> A little more context from the logs follows (I did not word wrap
>> the log messages to 72 characters because they are easier to read
>> without word wrapping).
>> 
>> On bare metal:
>> 
>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [    2.535938] [drm] 
>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [    2.536139] 
>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [    2.536274] 
>> exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [    2.536493] 
>> exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [    2.536520] 
>> exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>> ...
>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [    3.493686] 
>> Console: switching to colour frame buffer device 170x48
>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [    3.521747] 
>> exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [    3.522831] [drm] 
>> Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>> 
>> The screen works normally in this case.
>> 
>> On Xen as dom0:
>> 
>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [    2.606812] [drm] 
>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [    2.606884] 
>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [    2.606999] 
>> exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 
>> 14450000.mixer lacks support for IOMMU
>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [    2.607044] 
>> exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [    2.607162] 
>> exynos-drm exynos-drm: adev bind failed: -22
>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [    2.607183] 
>> exynos-dp: probe of 145b0000.dp-controller failed with error -22
>> 
>> There is no display on the screen in this case. The backlight
>> does not even come on.
>> 
>> So the error causing no display is probably:
>> 
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 
>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>> is needed for PCI passthrough to domU guests, but not for dom0 to
>> use such devices. 
> 
> I believe that the IOMMU would be required on x86 when using dom0 PVH. 
> PVH is very similar to an Arm guests.
> 
> On Arm, we don't require the IOMMU because not all Arm platforms have 
> all DMA-capable devices protected by an IOMMU. So dom0 will still have 
> its memory direct mapped (i.e. host physical address = guest physical 
> address) to allow DMA in dom0 with limited modification.
> 
> That said, I thik this is a different situation here (see below).
> 
> 
>> So on arm, why is dom0 trying to use IOMMU for
>> the exynos-mixer/exynos-drm when bare metal does not use it?
> 
> Just to confirm, are you using the same kernel, same config when booting 
> on baremetal? If so, from looking at the code, I would expect that the 
> IOMMU is also used on baremetal.

Hi Julien,

I didn't see that you asked this. The answer is yes, it is the same
kernel on the bare metal. I also thought from my brief look at the
code that on bare metal the IOMMU is also used, or at least, given
the logs for the bare metal case and the Xen case, I don't see any
evidence this check is not made on bare metal, but apparently the
check passes on bare metal.

> 
> The check failing is:
> 
> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
> 
> I am not quite too sure why the check implies the IOMMU is not 
> supported. That said, I vaguely recall that Linux will update the DMA 
> ops when running under Xen. Would you be able to print the two values 
> returned ("%pS" should give the symbol)?

I got those values now:

get_dma_ops(priv->dma_dev): 0xc0d018c0
get_dma_ops(subdrv_dev):    0xc0d662dc

Kind regards,

Chuck

>> 
> 
> Cheers,
> 
> [1] 
> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>