RE: [EXT] Re: xen arm64 low power sleep support

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 30 Aug 2023, Anthony Chan wrote:
> On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > Hi all,
> > >
> > > My name is Tony and I've been researching/developing using Xen for
> > potential upcoming uses in our embedded systems.  I started with Xen
> > using Xilinx tools about a year ago and still have lots to learn about what 
> > it
> > can to do in the embedded space.  So far, I've managed to integrate Xen
> > and Linux into an existing product that exclusively runs bare-metal code on
> > a ZynqMP SoC and migrate some of the functionality into custom Linux
> > driver/userspace.
> > >
> > > I'm now looking at low power support, for now at least between Xen
> > (4.16) and Linux (5.15) dom0.  I've tried a few different Linux kernel
> > configs around power management and each time I try to suspend from
> > linux dom0 (via sysfs or systemctl), Xen will watchdog on dom0 guest.
> > AFAIK, Xen should trap on a 'WFI' from guests, but from what I can tell
> > debugging through the linux suspend process is it's spinning in a 'suspend-
> > to-idle' loop before it can get to issuing a 'WFI' or using PSCI interface 
> > to
> > notify Xen.  I'm beginning to suspect that 'low power' support for
> > embedded arm64 just isn't quite there yet, or am I missing something in
> > the configs?
> > >
> > > I realize this could very well be a Linux 'issue' but checking here 
> > > first.  I
> > know Xen presents a flattened device tree to Linux without CPU idle-state
> > nodes and maybe this is causing the linux guest to only do the suspend-
> > to-idle mode?  I should mention that I'm booting up using dom0less
> > feature if that matters.
> >
> >
> > Hi Anthony,
> >
> > Assuming you are using the default Xen command line parameters for
> > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI, the CPU
> > will execute WFI directly and go into low power mode.
> Yes, using these command line params.
> 
> > Given the issue you are describing, I am suspecting the guest is not issuing
> > WFI: that is simple and known to work. Instead, I suspect that Linux might
> > be trying to use PSCI_suspend in a way that is not supported or well-
> > implemented by Xen.
> >
> > Can you check? You can add a printk in Linux
> > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> Instrumented both places it doesn't appear to reach there.  In 
> kernel/power/suspend.c, there's a call to s2idle_loop that it's currently 
> 'stuck' in and I think it doesn't get to the psci suspend your referring till 
> afterwards, when suspend_ops->enter is called.  Unfortunately, without any 
> idle-states nodes in the FDT, the only suspend state Linux is defaults to is 
> 'suspend to idle'.
 
The fact that Linux uses "suspend to idle" is not a problem because as I
mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.

However, if the issue is not PSCI_suspend then I don't have another easy
guess. Please post a full stack trace or more information about the
error in Linux and I might be able to see where it is coming from.