xen: arm64: Poor block device performance

[Jonas Blixt <jonas.blixt@xxxxxxxx>]

Hello,

I'm investigating a block device performance issue on our system.

Our setup is as follows:
SoC: NXP IMX8DXP (arm64), Dual core Cortex A35
Flash: eMMC, HS400
Xen 4.18.1
Dom0 kernel: 6.1.55
DomU kernel: 6.1.14

Dom0 has two vcpu's and domU has one. We're using the xen-blkfront/back drivers 
to expose block devices to domU.

We measure the following using a simple 'dd' -test (echo 3 > 
/proc/sys/vm/drop_caches && time dd if=/dev/XXX  of=/dev/null bs=1M count=64):

Directly on the eMMC block device in dom 0: 160 MByte/s
On xvd device in domU: 9.5 MByte/s

Ram block device in dom0: 460 MByte/s
On xvd device in domU: 246 MByte/s

The difference between the ram block device and the eMMC might suggest that 
there is some interaction between the xenblk driver and the mmc driver that 
causes this performance degradation. I'm not sure what to expect with the ram 
device but it at least does not have the same penalty.

I would appreciate any help or suggestions on how to debug this. Is 'blktrace' 
the right tool for this job and what should I look for?

Best Regards
Jonas