[QUESTION] Credit2 wakeup latency in 1-pCPU/2-vCPU Arm test

[Mykola Kvach <xakep.amatop@xxxxxxxxx>]

Hi all,

While running a wakeup-heavy Arm test, originally used for GICv4
doorbell testing, I noticed an interesting scheduler behavior on clean
Xen master.

The setup is synthetic on purpose:

1 physical CPU
2 Dom0 vCPUs
fio pinned to vCPU1
CPU stress pinned to vCPU0
4 KiB random read
iodepth=1
Credit2 scheduler

The goal is not to measure raw block throughput. The goal is to stress
the wakeup path where an I/O-bound vCPU is blocked, while another vCPU
is CPU-bound on the same pCPU.

On FVP with GICv3, the result is very sensitive to IRQ placement:

virtio IRQ pinned to the fio vCPU:
2223.939 IOPS, p99 = 1122.304 us
no IRQ pinning:
93.508 IOPS, p99 = 17170.432 us

I also ran the same test family on AWS c7g.metal with clean Xen master.
The same low-IOPS class is visible there as well:

normal run, 8 Dom0 vCPUs, Credit2:
about 1753 IOPS, p99 = 643 us
wakeup-heavy run, 2 Dom0 vCPUs, Credit2 default:
about 100 IOPS, p99 = 10158 us
wakeup-heavy run, 2 Dom0 vCPUs, Credit2 sched_ratelimit_us=100:
about 973 IOPS, p99 = 8454 us
wakeup-heavy run, 2 Dom0 vCPUs, Credit2 sched_ratelimit_us=0:
about 748 IOPS, p99 = 13435 us
wakeup-heavy run, 2 Dom0 vCPUs, RTDS:
about 999 IOPS, p99 = 6521 us

The AWS numbers above are medians from 5 repeats of the rr4k_qd1
workload.

I added debug counters around WFI, interrupt injection, vCPU kick, and
schedule-in. In the runs checked so far, the interrupt is delivered and
the target vCPU is kicked. The counter for entering WFI while a virtual
interrupt is already pending stays at zero. Most of the visible delay is
between the kick and the next schedule-in of the target vCPU.

My current interpretation is that this test is valid as a wakeup stress
test, but it is very sensitive to IRQ placement and scheduler preemption
behavior. I would not describe these numbers as raw storage throughput.

Since the same behavior is present on clean Xen master with GICv3, it
does not look specific to the GICv4 changes.

Question:

Does this match the expected Credit2 behavior for this kind of
overcommitted wakeup-heavy workload?

More specifically, when an interrupt wakes a blocked I/O-bound vCPU while
another vCPU is CPU-bound on the same pCPU, is the woken vCPU expected to
wait for the normal Credit2 scheduling/rate-limit window, or is there
an intended fast preemption path for this case?

Regards,
Mykola