Re: IOMMU problem on Xen dom0 arm (Was: Re: Xen on arm Chromebook seems to cause no display on screen)

[Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx>]

On 10/26/2023 5:24 PM, Julien Grall wrote:
> As a temporary hack, would you be able to compile out the IOMMU driver 
> from Linux and check if it helps using the GPU?

I did a test with most IOMMU support compiled out of
Linux.

Short summary: This does help the GPU on Xen, now dom0 can print
console messages on the built-in display, but X.org did not work
on Xen. The change also caused a new problem on Xen: without IOMMU
support Wifi stopped working on Xen. Everything still works on bare
metal without the IOMMU support in Linux but probably with a
performance hit.

Detailed description of the test and results:

The make *config targets for the Linux build would not allow
allow disabling CONFIG_IOMMU_SUPPORT without also disabling
CONFIG_DRM so I also did not disable these two generic
IOMMU options because it didn't make sense to do a test of
the GPU with a kernel that has DRM disabled:

#
# Generic IOMMU Pagetable Support
#
CONFIG_IOMMU_IO_PGTABLE=y
CONFIG_IOMMU_IO_PGTABLE_LPAE=y

The tested kernel has disabled these IOMMU options:

CONFIG_EXYNOS_IOMMU and several others as shown in the following
diff compared to the kernel I used in the original report:

--- .config.old 2023-10-24 00:41:01.000000000 -0400
+++ .config     2023-10-27 13:15:57.146322091 -0400
@@ -251,8 +251,6 @@
 
 CONFIG_ARM=y
 CONFIG_ARM_HAS_GROUP_RELOCS=y
-CONFIG_ARM_DMA_USE_IOMMU=y
-CONFIG_ARM_DMA_IOMMU_ALIGNMENT=8
 CONFIG_SYS_SUPPORTS_APM_EMULATION=y
 CONFIG_HAVE_PROC_CPU=y
 CONFIG_STACKTRACE_SUPPORT=y
@@ -5151,7 +5148,6 @@
 # end of Clock Source drivers
 
 # CONFIG_MAILBOX is not set
-CONFIG_IOMMU_API=y
 CONFIG_IOMMU_SUPPORT=y
 
 #
@@ -5164,12 +5160,7 @@
 # end of Generic IOMMU Pagetable Support
 
 # CONFIG_IOMMU_DEBUGFS is not set
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
-# CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set
-# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
-CONFIG_OF_IOMMU=y
-CONFIG_EXYNOS_IOMMU=y
-# CONFIG_EXYNOS_IOMMU_DEBUG is not set
+# CONFIG_EXYNOS_IOMMU is not set
 # CONFIG_ARM_SMMU is not set
 
 #
@@ -6558,7 +6549,6 @@
 CONFIG_HAS_IOPORT_MAP=y
 CONFIG_HAS_DMA=y
 CONFIG_DMA_OPS=y
-CONFIG_NEED_SG_DMA_LENGTH=y
 CONFIG_NEED_DMA_MAP_STATE=y
 CONFIG_ARCH_DMA_ADDR_T_64BIT=y
 CONFIG_DMA_DECLARE_COHERENT=y
 
 # CONFIG_GREYBUS is not set
-----------------------------

Logs for case without EXYNOS_IOMMU:

With Xen:

DRM logs:

1999-12-31T20:01:02.777765-05:00 devuan-bunsen kernel: [    2.362733] [drm] 
Exynos DRM: using 14400000.fimd device for DMA mapping operations
1999-12-31T20:01:02.777768-05:00 devuan-bunsen kernel: [    2.362755] 
exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d95d34)
1999-12-31T20:01:02.777771-05:00 devuan-bunsen kernel: [    2.362894] 
exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d96f34)
1999-12-31T20:01:02.777775-05:00 devuan-bunsen kernel: [    2.363116] 
exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d96c58)
1999-12-31T20:01:02.777939-05:00 devuan-bunsen kernel: [    2.363141] 
exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d975b0)

Wifi Logs:

1999-12-31T20:01:04.869178-05:00 devuan-bunsen kernel: [   13.612951] 
mwifiex_sdio mmc2:0001:1: info: trying to associate to bssid 28:74:f5:3e:6e:88
1999-12-31T20:01:04.889185-05:00 devuan-bunsen kernel: [   13.632338] 
mwifiex_sdio mmc2:0001:1: info: associated to bssid xx:xx:xx:xx:xx:xx 
successfully
1999-12-31T20:01:04.899132-05:00 devuan-bunsen kernel: [   13.640107] IPv6: 
ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@...
1999-12-31T20:00:24.635000-05:00 devuan-bunsen kernel: [    0.000000] Booting 
Linux on physical CPU 0x0

Bare metal:

DRM logs:

1999-12-31T20:01:07.261447-05:00 devuan-bunsen kernel: [    2.359431] [drm] 
Exynos DRM: using 14400000.fimd device for DMA mapping operations
1999-12-31T20:01:07.261455-05:00 devuan-bunsen kernel: [    2.359452] 
exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d95d34)
1999-12-31T20:01:07.261489-05:00 devuan-bunsen kernel: [    2.359569] 
exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d96f34)
1999-12-31T20:01:07.261498-05:00 devuan-bunsen kernel: [    2.359769] 
exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d96c58)
1999-12-31T20:01:07.261759-05:00 devuan-bunsen kernel: [    2.359793] 
exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d975b0)

Wifi logs:

1999-12-31T20:01:09.379505-05:00 devuan-bunsen kernel: [   13.599578] 
mwifiex_sdio mmc2:0001:1: info: associated to bssid xx:xx:xx:xx:xx:xx 
successfully
1999-12-31T20:01:09.389524-05:00 devuan-bunsen kernel: [   13.606470] IPv6: 
ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
1999-12-31T20:01:09.389573-05:00 devuan-bunsen kernel: [   13.611797] 
mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:34:40.082622-04:00 devuan-bunsen kernel: [   25.954828] 
mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:34:40.375838-04:00 devuan-bunsen kernel: [   26.238291] 
mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:35:11.332585-04:00 devuan-bunsen kernel: [   57.204105] 
mwifiex_sdio mmc2:0001:1: mlan0: already connected

Notable observations:

1. Without support for the Exynos IOMMU there is no difference
   in the logs for the initialization of the exynos-drm GPU
   between Xen and bare metal. Still, on Xen, Xorg did not start
   but on bare metal Xorg did start, but on Xen there was an
   improvement because now the system is able to display
   console messages from dom0 on the primary display. 

2. Notice that when Wifi works, the Chromebook learns the correct
   date from ntp. It did not get the correct date on Xen because
   on Xen it did not successfully connect to the network, even
   though it successfully associated with the Wifi access point.

3. It appears the check that was failing in the kernel with IOMMU
   support for exynos when running on Xen is skipped in this kernel
   on both Xen and the bare metal. That makes sense because the
   name of the function (drm_iommu_attach_device) where the failed
   test occurred appears to be specific to iommu so it is not used
   in a kernel without exynos IOMMU support.

4. The change in the kernel caused Wifi to not connect on Xen, and it
   looks like on Xen the Wifi driver dumped some binary data into the
   kernel log.

Here is what works without CONFIG_EXYNOS_IOMMU enabled:

         Bare metal    Xen
GPU          Y          Y
X.org        Y          N
Wifi         Y          N

I have not tried to adjust xorg.conf driver settings to
get X.org to work with Xen. If it does work with a different
Xorg driver I expect it would be with a performance hit.

For comparison with the original report of this problem,
here is what works with CONFIG_EXYNOS_IOMMU enabled:

         Bare metal    Xen
GPU          Y          N
X.org        Y          N
Wifi         Y          Y