Re: Issue: Networking performance in Xen VM on Arm64

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 18 Oct 2022, Leo Yan wrote:
> On Mon, Oct 17, 2022 at 04:50:05PM -0700, Stefano Stabellini wrote:
> 
> [...]
> 
> > > Which means it takes 543us to let Dom0 to receive the notification.
> > > You could see DomU runs in CPU3 and Dom0 runs on CPU13, there should
> > > not have contention for CPU resources.  Seems to me, it's likely Xen
> > > hypervisor takes long time to deliver the interrupt, note, it's not
> > > take so long time for every skb transferring, sometimes the time for
> > > response a notification is short (about ~10us).
> > 
> > Good find. I think this is worth investigating further. Do you have
> > vwfi=native in your Xen command line as well?
> 
> Yes, I have added "sched=null" and "vwfi=native" into Xen options:
> 
> options=noreboot dom0_mem=4096M bootscrub=0 iommu=on loglvl=error 
> guest_loglvl=error sched=null vwfi=native
> 
> > After that, I would add printk also in Xen with the timestamp. The event
> > channel notification code path is the following:
> > 
> > # domU side
> > xen/arch/arm/vgic-v2.c:vgic_v2_to_sgi
> > xen/arch/arm/vgic.c:vgic_to_sgi
> > xen/arch/arm/vgic.c:vgic_inject_irq
> > xen/arch/arm/vgic.c:vcpu_kick
> > xen/arch/arm/gic-v2.c:gicv2_send_SGI
> > 
> > # dom0 side
> > xen/arch/arm/gic.c:do_sgi
> > xen/arch/arm/traps.c:leave_hypervisor_to_guest
> > 
> > It would be good to understand why sometimes it takes ~10us and some
> > other times it takes ~540us
> 
> Thanks a lot for detailed info.
> 
> Just note, in my platform DomU enables GICv3 driver rather than GICv2.
> This would be a bit different in the Xen code.  But it should be easy
> for me to map to vgic-v3 files.
> 
> I have a question for how to trace Xen system.  Outputting chars to UART
> is time costy (usually it's millisecond level), it is not friendly to
> use console for debugging performance issue.  I searched a bit, either
> "xl dmesg" or xentrace can be used for capturing trace logs, one thing
> I am not certain is if we can save Xen logs only into log buffer and
> doesn't output to UART, so that afterwards we can use "xl dmesg" to
> capture the logs.  Could anyone confirm for this is correct usage with
> "xl dmesg" or I should use xentrace for this case?

I am not super familiar about this. But if you take away the console
entirely from Xen and give it to dom0 only (no dtuart command line
option to Xen), you should get the behavior you described:
- no console output because Xen has no console
- xl dmesg should still work