Re: Issue: Networking performance in Xen VM on Arm64

[Leo Yan <leo.yan@xxxxxxxxxx>]

On Mon, Oct 17, 2022 at 04:50:05PM -0700, Stefano Stabellini wrote:

[...]

> > Which means it takes 543us to let Dom0 to receive the notification.
> > You could see DomU runs in CPU3 and Dom0 runs on CPU13, there should
> > not have contention for CPU resources.  Seems to me, it's likely Xen
> > hypervisor takes long time to deliver the interrupt, note, it's not
> > take so long time for every skb transferring, sometimes the time for
> > response a notification is short (about ~10us).
> 
> Good find. I think this is worth investigating further. Do you have
> vwfi=native in your Xen command line as well?

Yes, I have added "sched=null" and "vwfi=native" into Xen options:

options=noreboot dom0_mem=4096M bootscrub=0 iommu=on loglvl=error 
guest_loglvl=error sched=null vwfi=native

> After that, I would add printk also in Xen with the timestamp. The event
> channel notification code path is the following:
> 
> # domU side
> xen/arch/arm/vgic-v2.c:vgic_v2_to_sgi
> xen/arch/arm/vgic.c:vgic_to_sgi
> xen/arch/arm/vgic.c:vgic_inject_irq
> xen/arch/arm/vgic.c:vcpu_kick
> xen/arch/arm/gic-v2.c:gicv2_send_SGI
> 
> # dom0 side
> xen/arch/arm/gic.c:do_sgi
> xen/arch/arm/traps.c:leave_hypervisor_to_guest
> 
> It would be good to understand why sometimes it takes ~10us and some
> other times it takes ~540us

Thanks a lot for detailed info.

Just note, in my platform DomU enables GICv3 driver rather than GICv2.
This would be a bit different in the Xen code.  But it should be easy
for me to map to vgic-v3 files.

I have a question for how to trace Xen system.  Outputting chars to UART
is time costy (usually it's millisecond level), it is not friendly to
use console for debugging performance issue.  I searched a bit, either
"xl dmesg" or xentrace can be used for capturing trace logs, one thing
I am not certain is if we can save Xen logs only into log buffer and
doesn't output to UART, so that afterwards we can use "xl dmesg" to
capture the logs.  Could anyone confirm for this is correct usage with
"xl dmesg" or I should use xentrace for this case?

Thanks!

Leo