xen_platform_pci and 4k-blocksize issue on DomU

["Eugene K." <ek@xxxxxxxxxxxxx>]

Hello.

We run several Linux virtual machines with latest Linux kernels.

Most of machines (group A) have xen_platform_pci enabled and single disk
drive emulated by physical LVM residing on SSD. SSD has logical and
physical block size of 512 bytes. Machines have virtual disk /dev/xvda,
gdisk on that /dev/xvda reports logical and physical block of 512 bytes.

Few more machines (group B) are running on the same servers but have
xen_platform_pci disabled. Their LVM resides on either SSD or NVMe,
those NVMe has 4096-byte blocks unlike SSDs. All the machines of this
group regardless of the physical storage under LVM report virtual disk
as /dev/sda and gdisk reports block size of 512 bytes there.

Last group (group C) has two virtual disks connected to them, one is
rootfs residing on SSD and another one is data partition that physically
resides on NVMe.

Machines of this last group C have xen_platform_pci enabled, so disks
are visible as /dev/xvda and /dev/xvdb. Before upgrade to Linux-6.9.3
these machines have had /dev/xvda reporting pysical/logical block size
of 512 bytes and /dev/xvdb with pysical/logical blocks of 4096-byte
size.

Problem 1. Machines of group A can't boot in case of underlying storage
migration to LVM that physically resides on NVMe - as their partitions
starts to report physical/logical blocks of 4096-bytes (instead of 512
logical/4096 physical that could be expected), so it was impossible to
migrate VMs running with xen_platform_pci enabled. This problem doesn't
affect machines of group B (having xen_platform_pci disabled and always
using 512-byte blocks)

Problem 2. On upgrading Linux from kernel 6.8.12 to kernel version 6.9.3
machines of group C started to report block size of both partitions -
the first one that physically runs over SSD and second one over NVMe -
of the same value, so blocks of all the devices became 512-byte (and
amount of blocks is 8x times bigger on data partitions that used to be
reporting 4k-blocks with older linux kernel). Kernel configuration
almost didn't change, just new options introduced in 6.9.3 appeared, but
none of them is disk or xen-related. The issue persists on all the 6.9
branch, the latest one, and never happen on 6.6 or 6.8 kernels, on
reboot to 6.6.30 machine still reports one partition as having 512-byte
block, another one as 4kbyte-blocks, so matches underlying storage block
size.

The problem seems to be related to some change of Xen PVHVM driver in
Linux kernel, but it was not mentioned anywhere explicitly.

In case of either disabling xen_platform_pci or using "sdb" instead of
"hdb" in disk configuration (so guest kernel detects SCSI drive), block
has (and has always had) 512-byte size on any drive regardless of
underlying storage block size.

The question is whether to consider this as a bug (as block size is not
aligned in guest and physical storage anymore while it used to be) or
fix as now all the drivers are not depending from physical storage
parameters? If this is a fix and desired behavior, then how big the
performance impact will be because of block size inside the VM being 8
times smaller than the physical one?..

--
Regards,
Eugene K.