Re: [Xen-devel] [RFC] xen/arm: Suspend to RAM Support in Xen for ARM

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 9 Aug 2017, Mirela Simonovic wrote:
> This document contains our draft proposal for implementing "suspend to RAM"
> support for ARM in Xen, as discussed during the last Xen ARM community call.
> It covers the basic suspend to RAM mechanism based on ARM PSCI standard,
> that would allow individual guests and Xen itself to suspend.
> 
> We would appreciate your feedback.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@xxxxxxxxxx>

I am really sorry, for the very late reply. This email fell through the
cracks. In general, if I forget about an email, please send a follow up
to help me remember :-) Do you have an more up-to-date proposal or is
this still the latest?

From an high level, it looks good, only one minor comment: don't call it
"hotplug" because usually in the Xen community hotplug means adding more
vcpus or removing vcpus at runtime (removing entirely, not suspending).

But the approach looks good to me.


> ---
>  docs/misc/arm/suspend-to-ram.txt | 210 
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 210 insertions(+)
>  create mode 100644 docs/misc/arm/suspend-to-ram.txt
> 
> diff --git a/docs/misc/arm/suspend-to-ram.txt 
> b/docs/misc/arm/suspend-to-ram.txt
> new file mode 100644
> index 0000000000..ec8080fc64
> --- /dev/null
> +++ b/docs/misc/arm/suspend-to-ram.txt
> @@ -0,0 +1,210 @@
> +% Suspend to RAM Support in Xen for ARM
> +% Revision 1.0
> +
> +========
> +Overview
> +========
> +
> +Suspend to RAM (in the following text 'suspend') for ARM in Xen should be
> +coordinated using ARM PSCI standard [1].
> +
> +EL1/2 should suspend in the following order:
> +1) Unprivileged guests (DomUs) suspend
> +2) Privileged guest (Dom0) suspends
> +3) Xen suspends
> +
> +Since this proposal is focused on implementing PSCI-based suspend mechanisms 
> in
> +Xen, communication with or among the guests is not covered by this document.
> +The order of suspending the guests is assumed to be guaranteed by the 
> software
> +running in EL1.
> +
> +-----------------
> +Suspending Guests
> +-----------------
> +
> +Suspend procedure for a guest consists of the following:
> +1) Suspending devices
> +2) Suspending non-boot CPUs
> +3) System suspend, performed by the boot CPU
> +
> +Each guest should suspend the devices it owns. Suspending of devices is not
> +covered by this document. The document covers only mechanisms for suspending
> +non-boot CPUs, as well as the system suspend.
> +
> +Guests should suspend their non-boot vCPUs using the hotplug mechanism.
> +Virtual CPUs should be put offline using the already implemented PSCI 
> vCPU_OFF
> +call (prefix 'v' is added to distinguish PSCI calls made by guests to Xen, 
> which
> +affect virtual machines; as opposed to PSCI calls made by Xen to the EL3, 
> which
> +can affect power state of the physical machine).
> +
> +After suspending its non-boot vCPUs a guest should finalize the suspend by
> +making the vSYSTEM_SUSPEND PSCI call. The resume address is specified by the
> +guest via the vSYSTEM_SUSPEND entry_point_address argument. The 
> vSYSTEM_SUSPEND
> +call is currently not implemented in Xen.
> +
> +It is expected that a guest leaves enabled all interrupts that should wake it
> +up. Other interrupts should be disabled by the guest prior to calling
> +vSYSTEM_SUSPEND.
> +
> +After an unprivileged guest suspends, Xen will not suspend. Xen would suspend
> +only after the Dom0 completes the system suspend.
> +
> +--------------
> +Suspending Xen
> +--------------
> +
> +Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND call
> +from the last running guest (Dom0). At that moment all physical CPUs are 
> still
> +online (taking offline a vCPU or suspending a VM does not affect physical 
> CPUs).
> +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI call
> +to EL3. The CPU_OFF PSCI function is currently not implemented in Xen.
> +
> +After putting offline the non-boot cores Xen must save the context and 
> finalize
> +suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3.
> +The resume point of Xen is specified by the entry_point_address argument of 
> the
> +SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is not
> +implemented in Xen for ARM today.
> +
> +------------
> +Resuming Xen
> +------------
> +
> +Xen must be resumed prior to any software running in EL1. Starting from the
> +resume point, Xen should restore the context and resume Dom0. Dom0 shall 
> always
> +be resumed whenever Xen resumes.
> +The whole Xen resume flow for the ARM architecture has to be implemented.
> +
> +---------------
> +Resuming Guests
> +---------------
> +
> +Resume of the privileged guest (Dom0) is always following the Xen resume.
> +
> +An unprivileged guest shall resume once a device it owns triggers a wake-up
> +interrupt, regardless of whether Xen was suspended when the wake-up interrupt
> +was triggered. If Xen was suspended, it is assumed that Dom0 will be running
> +before the DomU guest starts to resume. The synchronization mechanism to
> +enforce the assumed condition is TBD.
> +
> +If the ARM's GIC was powered down after the ARM subsystem suspended, it is
> +assumed that Xen needs to restore the GIC interface for a VM prior to handing
> +over control to the guest. However, the guest should restore its own context
> +upon entering the resume point (out of scope of this document).
> +
> +=======================
> +Implementation Proposal
> +=======================
> +
> +--------
> +Overview
> +--------
> +
> +In order to enable the suspend/resume of VMs and Xen itself, the following 
> PSCI
> +calls have to be implemented and integrated in Xen:
> +1) vSYSTEM_SUSPEND
> +2) CPU_OFF
> +3) SYSTEM_SUSPEND
> +
> +In addition, the following have to be implemented:
> +* Save/restore of EL2 context
> +* Save/restore of GIC configuration for each VM
> +
> +Implementation details are provided in the sections below. Function names and
> +paths used below are consistent within the document but may not always match 
> the
> +names used in future implementation. Existing functions and paths are named 
> as
> +in Xen source tree.
> +
> +Note: The proposal is still incomplete and shall be refined in future 
> revisions.
> +Specific issues that are not addressed are marked as "TBD".
> +
> +-------------------------------------
> +Suspend/Resume Implementation Details
> +-------------------------------------
> +
> +PSCI Implementation and Integration
> +-----------------------------------
> +vSYSTEM_SUSPEND
> +---------------
> +vSYSTEM_SUSPEND shall be implemented in
> +* do_psci_system_suspend() in arch/arm/vpsci.c
> +
> +The implementation shall include the following steps:
> +* Block the current vCPU
> +* If the hardware domain made the call trigger Xen suspend, i.e.
> +  call machine_suspend() which will be implemented in arch/arm/suspend.c
> + (similar as the machine_restart() is implemented in arch/arm/shutdown.c)
> +
> +The function do_psci_system_suspend() shall be called from
> +* do_trap_psci() in arch/arm/traps.c
> +
> +CPU_OFF (physical CPUs)
> +-----------------------
> +The CPU_OFF function shall be implemented in
> +* call_psci_cpu_off() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3.
> +
> +This function needs to be called when Xen generic code disables non-boot 
> CPUs,
> +which is done by
> +* disable_nonboot_cpus() in common/cpu.c
> +This function calls architecture specific
> +* __cpu_die() implemented in arch/arm/smpboot.c
> +The call_psci_cpu_off() shall be invoked when the respective CPU dies. To 
> make
> +that happen, the
> +* arch_cpu_die() would be implemented in arch/arm/arm64/smpboot.c
> +and called from __cpu_die().
> +Finally the call_psci_cpu_off() shall be invoked from arch_cpu_die().
> +
> +Such a control flow would be similar to the already existing flow for 
> enabling
> +non-boot CPUs, which looks like this:
> +enable_nonboot_cpus() -> cpu_up() -> __cpu_up() -> arch_cpu_up() ->
> +call_psci_cpu_on()
> +
> +SYSTEM_SUSPEND (physical)
> +-------------------------
> +The SYSTEM_SUSPEND function shall be implemented in
> +* call_psci_system_suspend() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3. The
> +entry_point_address argument of the SMC call needs to be an ARM architecture
> +resume address. The call_psci_system_suspend() function does not return.
> +
> +The function needs to be called from machine_suspend() to finalize the 
> suspend
> +procedure.
> +
> +------------------
> +Additional Changes
> +------------------
> +
> +Suspend Flow
> +------------
> +The suspend procedure shall be implemented in
> +* machine_suspend() in arch/arm/suspend.c
> +
> +The implementation shall include the following steps:
> +* Set the system_state variable to SYS_STATE_suspend
> +* Freeze domains by calling domain_pause() for each domain (we assume this 
> needs
> +  to be done)
> +* Disable non-boot CPUs by calling disable_nonboot_cpus()
> +* Save ARM specific context
> +
> +Resume Flow
> +------------
> +The resume procedure shall be implemented in
> +* machine_resume() in arch/arm/suspend.c
> +
> +The machine_resume() implementation shall include the following steps:
> +* Restore ARM specific context
> +* Enable non-boot CPUs by calling enable_nonboot_cpus()
> +* Thaw domains by calling domain_unpause() for each domain (we assume this 
> needs
> +  to be done)
> +* Set the system_state variable to SYS_STATE_resume
> +* TBD: how to resume Dom0, i.e. how to hand over control to Dom0?
> +
> +==========
> +References
> +==========
> +
> +[1] Power State Coordination Interface (ARM):
> +http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf
> +
> -- 
> 2.13.0
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel