Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for ARM

[Peng Fan <peng.fan@xxxxxxx>]

Hi Edgar,

> -----Original Message-----
> From: Edgar E. Iglesias [mailto:edgar.iglesias@xxxxxxxxxx]
> Sent: 2018年3月26日 19:43
> To: Peng Fan <peng.fan@xxxxxxx>
> Cc: Mirela Simonovic <mirela.simonovic@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxx;
> sstabellini@xxxxxxxxxx; julien.grall@xxxxxxxxxx
> Subject: Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for
> ARM
> 
> On Mon, Mar 26, 2018 at 09:51:40AM +0000, Peng Fan wrote:
> > Hi Mirela,
> >
> > Good to know that you are working suspend/resume support. Currently we
> > are also trying to support this on i.MX8, just wonder do you have any
> > open source available to support suspend to ram?
> >
> > > +
> > > +Suspend to RAM (in the following text 'suspend') for ARM in Xen
> > > +should be coordinated using ARM PSCI standard [1].
> > > +
> > > +Ideally, EL1/2 should suspend in the following order:
> > > +1) Unprivileged guests (DomUs) suspend
> > > +2) Privileged guest (Dom0) suspends
> > > +3) Xen suspends
> > > +
> > > +However, suspending unprivileged guests is not mandatory for
> > > +suspending
> > > +Dom0 and Xen. System suspend initiated by Dom0 (step 2) is
> > > +considered to be an ultimate decision to suspend the physical
> > > +machine. Suspending of Xen (step 3) is triggered whenever Dom0
> > > +completes suspend. Xen suspend leads to the full suspend of EL2.
> > > +
> > > +If an unprivileged guest is not suspended at the moment when Dom0
> > > +initiates its own suspend, the guest will be paused on Xen's
> > > +suspend and unpaused on Xen's resume. That way, a guest which
> > > +doesn't have power management support cannot prevent the physical
> > > +system from suspending when the decision to suspend is made by
> > > +privileged software
> > > (Dom0).
> > > +
> > > +Each guest in the system is responsible for suspending the devices it 
> > > owns.
> > > +If a guest does not suspend a device, the device will continue to
> > > +operate as it is configured at the moment when the system suspends.
> > > +If a device triggers an interrupt while the physical system is
> > > +suspended, the
> > > system will resume.
> > > +
> > > +It is recommended for an unprivileged guest to participate in power
> > > +management in the following scenario:
> > > +Assume unprivileged guest owns a device which will trigger
> > > +interrupt at some point. This interrupt will wake-up the system. If
> > > +such a behavior is not wanted, coordination between Dom0 and the
> > > +guest is required in order to inform the guest about the intended 
> > > physical
> system suspend.
> > > +Then, the guest will have a chance to suspend the device or respond
> > > +to the
> > > request in an abort fashion.
> > > +
> > > +Since this proposal is focused on implementing PSCI-based suspend
> > > +mechanisms in Xen, communication with or among the guests is not
> > > +covered by
> > > this document.
> > > +The order of suspending the guests is assumed to be guaranteed by
> > > +the software running in EL1.
> > > +
> > > +This document covers the following:
> > > +1) Mechanism for suspending/resuming a guest:
> > > + 1.1) Suspend is initiated by the guest
> > > + 1.2) Resume is initiated by a device interrupt
> > > +2) Mechanism for pausing/unpausing running guests when Dom0
> > > +suspends
> >
> > Will this take care of passthroughed devices for DomU?
> 

Thanks for your reply. Sorry for late reply

> Hi Peng,
> 
> The ZynqMP uses the EEMI Firmware interface to do power-management.
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.x
> ilinx.com%2Fsupport%2Fdocumentation%2Fuser_guides%2Fug1200-eemi-api.p
> df&data=02%7C01%7Cpeng.fan%40nxp.com%7C021307a245394e945cbf08d59
> 30ebb3c%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C6365766138
> 46476140&sdata=xwyil1ar7VXXYPJb2yXxYPWJvR5mVEb6wokggdt0ZH4%3D&re
> served=0

Yes. I see. 

> 
> In our case, we've implemented an EEMI mediator in Xen that traps EEMI
> requests from domU's and makes sure that the guest owns the device it is 
> trying
> to operate on.
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2FXilinx%2Fxen%2Fblob%2Fxilinx%2Fstable-4.9%2Fxen%2Farch%2Farm%
> 2Fplatforms%2Fxilinx-zynqmp-eemi.c&data=02%7C01%7Cpeng.fan%40nxp.com
> %7C021307a245394e945cbf08d5930ebb3c%7C686ea1d3bc2b4c6fa92cd99c5c3
> 01635%7C0%7C0%7C636576613846476140&sdata=33AdCyBLxUYIR6h%2BtZzx
> TrnYpOZ86IMFySmjHA2%2Fits%3D&reserved=0
> 
> So domU will first issue the usual EEMI calls as it would in a 
> non-virtualized case
> to suspend all it's devices. Once that has happened, the guest will issue 
> PSCI calls
> to suspend the VM. So, Mirela please shim in if I missed something.
> 
> The EEMI mediator has been posted to the ML but is currently sitting in our 
> tree
> waiting for us to go through the upstreaming effort.

So if Dom0 and DomU both running Linux, in DomU, "echo mem >/sys/power/state" 
to suspend
DomU, then in Dom0 "echo mem >/sys/power/state" to suspend Dom0?

Thanks,
Peng.

> 
> Cheers,
> Edgar
> 
> 
> 
> 
> >
> > Thanks,
> > Peng.
> >
> > > +3) Mechanism for suspending/resuming Xen when Dom0 completes
> > > +suspend
> > > +4) Resuming from any state on a wake-up event (device interrupt):
> > > + 4.1) Resume DomU on wake-up event when Dom0 is still running
> > > + 4.2) Resume DomU on wake-up event when Xen is suspended
> > > + 4.3) Resume Dom0 on wake-up event
> > > +
> > > +Mechanisms enumerated above will allow different kind of policies
> > > +and coordination among guests to be implemented in EL1. That is out
> > > +of the scope of this document.
> > > +
> > > +-----------------
> > > +Suspending Guests
> > > +-----------------
> > > +
> > > +Suspend procedure for a guest consists of the following:
> > > +1) Suspending devices
> > > +2) Suspending non-boot CPUs (based on hotplug/PSCI)
> > > +3) System suspend, performed by the boot CPU
> > > +
> > > +Each guest should suspend the devices it owns just like it would
> > > +when running without Xen.
> > > +
> > > +Guests should suspend their non-boot vCPUs using the hotplug mechanism.
> > > +Virtual CPUs should be put offline using the already implemented
> > > +PSCI vCPU_OFF call (prefix 'v' is added to distinguish PSCI calls
> > > +made by guests to Xen, which affect virtual machines; as opposed to
> > > +PSCI calls made by Xen to the EL3, which can affect power state of
> > > +the physical
> > > machine).
> > > +
> > > +After suspending its non-boot vCPUs a guest should finalize the
> > > +suspend by making the vSYSTEM_SUSPEND PSCI call. The resume address
> > > +is specified by the guest via the vSYSTEM_SUSPEND
> > > +entry_point_address argument. The vSYSTEM_SUSPEND call is currently
> not implemented in Xen.
> > > +
> > > +It is expected that a guest leaves enabled all interrupts that
> > > +should wake it up. Other interrupts should be disabled by the guest
> > > +prior to calling vSYSTEM_SUSPEND.
> > > +
> > > +After an unprivileged guest suspends, Xen will not suspend. Xen
> > > +would suspend only after the Dom0 completes the system suspend.
> > > +
> > > +--------------
> > > +Suspending Xen
> > > +--------------
> > > +
> > > +Xen should start suspending itself upon receiving the
> > > +vSYSTEM_SUSPEND call from the last running guest (Dom0). At that
> > > +moment all physical CPUs are still online (taking offline a vCPU or
> > > +suspending a VM does not affect
> > > physical CPUs).
> > > +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF
> > > +PSCI call to EL3. The CPU_OFF PSCI function is currently not implemented 
> > > in
> Xen.
> > > +
> > > +After putting offline the non-boot cores Xen must save the context
> > > +and finalize suspend by invoking SYSTEM_SUSPEND PSCI call, which is
> passed to EL3.
> > > +The resume point of Xen is specified by the entry_point_address
> > > +argument of the SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function
> > > +and context saving is not implemented in Xen for ARM today.
> > > +
> > > +------------
> > > +Resuming Xen
> > > +------------
> > > +
> > > +Xen must be resumed prior to any software running in EL1. Starting
> > > +from the resume point, Xen should restore the context and resume
> > > +Dom0. Dom0 shall always be resumed whenever Xen resumes.
> > > +
> > > +---------------
> > > +Resuming Guests
> > > +---------------
> > > +
> > > +Resume of the privileged guest (Dom0) is always following the Xen resume.
> > > +
> > > +An unprivileged guest shall resume once a device it owns triggers a
> > > +wake-up interrupt, regardless of whether Xen was suspended when the
> > > +wake-up interrupt was triggered. If Xen was suspended, it is
> > > +assumed that Dom0 will be running before the DomU guest starts to
> > > +resume. The synchronization mechanism to enforce the assumed condition
> is TBD.
> > > +
> > > +If the ARM's GIC was powered down after the ARM subsystem
> > > +suspended, it is assumed that Xen needs to restore the GIC
> > > +interface for a VM prior to handing over control to the guest.
> > > +However, the guest should restore its own context upon entering the
> > > +resume point, just like it would when
> > > running without Xen.
> > > +
> > > +===============
> > > +Implementation
> > > +===============
> > > +
> > > +--------
> > > +Overview
> > > +--------
> > > +
> > > +In order to enable the suspend/resume of VMs and Xen itself, the
> > > +following PSCI calls have to be implemented and integrated in Xen:
> > > +1) vSYSTEM_SUSPEND
> > > +2) CPU_OFF
> > > +3) SYSTEM_SUSPEND
> > > +
> > > +In addition, the following have to be implemented:
> > > +* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
> > > +* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0),
> > > including:
> > > + * Disable wathdog on suspend, enable it on resume
> > > + * Pause domains on suspend, unpause them on resume
> > > + * Disable non-boot pCPUs on suspend, enable them on resume
> > > + * Save/restore of GIC configuration
> > > + * Suspend/resume timer
> > > + * Save/restore of EL2 context
> > > + * Implement resume entry point in Xen, including MMU configuration
> > > +
> > > +Implementation details are provided in the sections below. Function
> > > +names and paths used below are consistent within the document but
> > > +may not always match the names used in future implementation.
> > > +Existing functions and paths are named as in Xen source tree.
> > > +
> > > +-------------------------------------
> > > +Suspend/Resume Implementation Details
> > > +-------------------------------------
> > > +
> > > +PSCI Implementation and Integration
> > > +-----------------------------------
> > > +vSYSTEM_SUSPEND
> > > +---------------
> > > +vSYSTEM_SUSPEND shall be implemented in
> > > +* do_psci_system_suspend() in arch/arm/vpsci.c
> > > +* Code independent from PSCI interface will be added in
> > > +arch/arm/suspend.c
> > > +
> > > +The implementation shall include the following steps:
> > > +* Suspend the current (calling) vCPU. Consists of 2 major steps:
> > > +1) Reset context of vCPU and save entry point into PC and context
> > > +ID into X0 (entry point and context ID are provided via
> > > +vSYSTEM_SUSPEND
> > > +arguments)
> > > +2) Block vCPU to ensure that it is not scheduled until it is
> > > +unblocked by an interrupt.
> > > +In step 1) above, the context is reset in order to prepare the vCPU
> > > +for resume, i.e. to save vCPU context that matches reset values as
> > > +expected by software on resume. This doesn't hold for PC and X0,
> > > +since the PC contains resume entry point and X0 contains context ID, as
> defined by PSCI.
> > > +* If the hardware domain made the call trigger Xen suspend, i.e.
> > > +  call machine_suspend() which will be implemented in
> > > +arch/arm/suspend.c  (similar as the machine_restart() is
> > > +implemented in
> > > +arch/arm/shutdown.c)
> > > +
> > > +The function do_psci_system_suspend() shall be called from
> > > +* do_trap_psci() in arch/arm/traps.c
> > > +
> > > +CPU_OFF (physical CPUs)
> > > +-----------------------
> > > +The CPU_OFF function shall be implemented in
> > > +* call_psci_cpu_off() in arch/arm/psci.c
> > > +
> > > +The implementation shall consist just of making the SMC call to EL3.
> > > +
> > > +This function needs to be called when Xen generic code disables a
> > > +non-boot
> > > CPU.
> > > +When a CPU is disabled it will loop forever in while loop
> > > +(stop_cpu() function which is already implemented in
> > > +xen/arch/arm/smpboot.c). Call to
> > > +call_psci_cpu_off() shall be made before the CPU enters infinite loop.
> > > +
> > > +SYSTEM_SUSPEND (physical)
> > > +-------------------------
> > > +The SYSTEM_SUSPEND function shall be implemented in
> > > +* call_psci_system_suspend() in arch/arm/psci.c
> > > +
> > > +The implementation shall consist just of making the SMC call to EL3.
> > > +The entry_point_address argument of the SMC call needs to be an ARM
> > > +architecture resume address, which shall be implemented, e.g. as
> > > +hyp_resume() in arch/arm/arm64/entry.S. The
> > > +call_psci_system_suspend()
> > > function does not return.
> > > +On the resume, the execution flow continues from hyp_resume.
> > > +
> > > +The function needs to be called from machine_suspend() to finalize
> > > +the suspend procedure.
> > > +
> > > +------------------
> > > +Additional Changes
> > > +------------------
> > > +
> > > +Suspend Flow
> > > +------------
> > > +The suspend procedure shall be implemented in
> > > +* machine_suspend() in arch/arm/suspend.c
> > > +
> > > +The implementation shall include the following steps:
> > > +* Move the execution to boot pCPU
> > > +* Set the system_state variable to SYS_STATE_suspend
> > > +* Disable watchdog
> > > +* Freeze domains by calling domain_pause() for each domain
> > > +* Disable non-boot CPUs by calling disable_nonboot_cpus()
> > > +* Disable interrupts
> > > +* Suspend timer
> > > +* Save GIC context. Shall be implemented in arch/arm/gic.c,
> > > +  include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be
> supported).
> > > +* Save CPU context. This shall be implemented in assembly, in
> > > +hyp_suspend()
> > > +  in arch/arm/arm64/entry.S. The context consists of callee-saved
> > > +general
> > > +  purpose registers, as well as few system registers. Context of
> > > +registers shall
> > > +  be saved in a statically allocated structure.
> > > +* Finalize the suspend by calling call_psci_system_suspend()
> > > +
> > > +Resume Flow
> > > +------------
> > > +The resume entry point shall be implemented in
> > > +* hyp_resume() in arch/arm/arm64/entry.S The very beginning of the
> > > +resume procedure has to be implemented in assembly.
> > > +It shall contain the following:
> > > +* Enable the MMU so that the structure containing CPU context which
> > > +was saved on suspend can be accessed
> > > +* Restore CPU context (to match the values saved on suspend) and
> > > +return into C
> > > +* Set the system_state variable to SYS_STATE_resume
> > > +* Restore GIC context
> > > +* Resume timer
> > > +* Enable interrupts
> > > +* Enable non-boot CPUs by calling enable_nonboot_cpus()
> > > +* Thaw domains by calling domain_unpause() for each domain
> > > +* Enable watchdog
> > > +* Set the system_state variable to SYS_STATE_active
> > > +* Resume Dom0
> > > +
> > > +==========
> > > +References
> > > +==========
> > > +
> > > +[1] Power State Coordination Interface (ARM):
> > > +https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fi
> > > +nfoc
> > >
> +enter.arm.com%2Fhelp%2Ftopic%2Fcom.arm.doc.den0022d%2FPower_State
> > > _Coord
> > >
> +ination_Interface_PDD_v1_1_DEN0022D.pdf&data=02%7C01%7Cpeng.fan%4
> > > 0nxp.c
> > >
> +om%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c6fa92cd99
> > > c5c30163
> > >
> +5%7C0%7C1%7C636495614074885940&sdata=3ycqEZR9XgcqdvrmJKY86aukt
> > > %2BQv%2BS
> > > +BSZMxbCrpraEY%3D&reserved=0
> > > --
> > > 2.13.0
> > >
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxxxxxxxxx
> > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> > > ists.xe
> > >
> nproject.org%2Fmailman%2Flistinfo%2Fxen-devel&data=02%7C01%7Cpeng.fa
> > >
> n %40nxp.com%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c
> 6f
> > >
> a92cd99c5c301635%7C0%7C0%7C636495614074885940&sdata=YLuJhbx%2B1
> > > tDvblYbgtOZZBhsG36%2BUhpRc4VpSpHHM%2FU%3D&reserved=0
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel