Re: IOMMU problem on Xen dom0 arm (Was: Re: Xen on arm Chromebook seems to cause no display on screen)

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 10/27/2023 12:06 AM, Chuck Zmudzinski wrote:
> On 10/26/2023 5:24 PM, Julien Grall wrote:
>> (+ Bertrand, Stefano)
>> 
>> Hi Chuck,
>> 
>> Thanks for the report.
>> 
>> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>>> We also have not yet done a thorough analysis of the differences
>>>> in the kernel boot logs when booting on the bare metal vs. booting
>>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>>> cause of this problem after a quick look at the logs.
>>> 
>>> After a more careful look at the logs, this seems to be the error
>>> causing no display when booting as dom0 on Xen:
>>> 
>>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>> 
>>> A little more context from the logs follows (I did not word wrap
>>> the log messages to 72 characters because they are easier to read
>>> without word wrapping).
>>> 
>>> On bare metal:
>>> 
>>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [    2.535938] [drm] 
>>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [    2.536139] 
>>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [    2.536274] 
>>> exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [    2.536493] 
>>> exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [    2.536520] 
>>> exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>>> ...
>>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [    3.493686] 
>>> Console: switching to colour frame buffer device 170x48
>>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [    3.521747] 
>>> exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [    3.522831] [drm] 
>>> Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>>> 
>>> The screen works normally in this case.
>>> 
>>> On Xen as dom0:
>>> 
>>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [    2.606812] [drm] 
>>> Exynos DRM: using 14400000.fimd device for DMA mapping operations
>>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [    2.606884] 
>>> exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [    2.606999] 
>>> exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 
>>> 14450000.mixer lacks support for IOMMU
>>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [    2.607044] 
>>> exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [    2.607162] 
>>> exynos-drm exynos-drm: adev bind failed: -22
>>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [    2.607183] 
>>> exynos-dp: probe of 145b0000.dp-controller failed with error -22
>>> 
>>> There is no display on the screen in this case. The backlight
>>> does not even come on.
>>> 
>>> So the error causing no display is probably:
>>> 
>>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>> 
>>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>>> is needed for PCI passthrough to domU guests, but not for dom0 to
>>> use such devices. 
>> 
>> I believe that the IOMMU would be required on x86 when using dom0 PVH. 
>> PVH is very similar to an Arm guests.
>> 
>> On Arm, we don't require the IOMMU because not all Arm platforms have 
>> all DMA-capable devices protected by an IOMMU. So dom0 will still have 
>> its memory direct mapped (i.e. host physical address = guest physical 
>> address) to allow DMA in dom0 with limited modification.
>> 
>> That said, I thik this is a different situation here (see below).
>> 
>> 
>>> So on arm, why is dom0 trying to use IOMMU for
>>> the exynos-mixer/exynos-drm when bare metal does not use it?
>> 
>> Just to confirm, are you using the same kernel, same config when booting 
>> on baremetal? If so, from looking at the code, I would expect that the 
>> IOMMU is also used on baremetal.
> 
> Hi Julien,
> 
> I didn't see that you asked this. The answer is yes, it is the same
> kernel on the bare metal. I also thought from my brief look at the
> code that on bare metal the IOMMU is also used, or at least, given
> the logs for the bare metal case and the Xen case, I don't see any
> evidence this check is not made on bare metal, but apparently the
> check passes on bare metal.
> 
>> 
>> The check failing is:
>> 
>> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
>> 
>> I am not quite too sure why the check implies the IOMMU is not 
>> supported. That said, I vaguely recall that Linux will update the DMA 
>> ops when running under Xen. Would you be able to print the two values 
>> returned ("%pS" should give the symbol)?
> 
> I got those values now:
> 
> get_dma_ops(priv->dma_dev): 0xc0d018c0
> get_dma_ops(subdrv_dev):    0xc0d662dc

On bare metal:

get_dma_ops(priv->dma_dev): 0xc0d018c0
get_dma_ops(subdrv_dev):    0xc0d018c0

So the dma_ops of the subdrv_dev is changed by Xen, causing the check
to fail. I assume the subdrv_dev is the exynos_mixer in this case.

> 
> Kind regards,
> 
> Chuck
> 
>>> 
>> 
>> Cheers,
>> 
>> [1] 
>> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>> 
> 
>