Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for ARM

[Mirela Simonovic <mirela.simonovic@xxxxxxxxxx>]

Hi Peng,

Sorry for late response, this email got buried and I accidentally saw it now.

On Thu, Apr 12, 2018 at 4:26 AM, Peng Fan <peng.fan@xxxxxxx> wrote:
> Hi Edgar,
>
>> -----Original Message-----
>> From: Edgar E. Iglesias [mailto:edgar.iglesias@xxxxxxxxxx]
>> Sent: 2018年3月26日 19:43
>> To: Peng Fan <peng.fan@xxxxxxx>
>> Cc: Mirela Simonovic <mirela.simonovic@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxx;
>> sstabellini@xxxxxxxxxx; julien.grall@xxxxxxxxxx
>> Subject: Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for
>> ARM
>>
>> On Mon, Mar 26, 2018 at 09:51:40AM +0000, Peng Fan wrote:
>> > Hi Mirela,
>> >
>> > Good to know that you are working suspend/resume support. Currently we
>> > are also trying to support this on i.MX8, just wonder do you have any
>> > open source available to support suspend to ram?
>> >

We don't have suspend to RAM support available in open source yet.
However, we are starting the upstream (just the first series covering
CPU hotplug is submitted so far).

>> > > +
>> > > +Suspend to RAM (in the following text 'suspend') for ARM in Xen
>> > > +should be coordinated using ARM PSCI standard [1].
>> > > +
>> > > +Ideally, EL1/2 should suspend in the following order:
>> > > +1) Unprivileged guests (DomUs) suspend
>> > > +2) Privileged guest (Dom0) suspends
>> > > +3) Xen suspends
>> > > +
>> > > +However, suspending unprivileged guests is not mandatory for
>> > > +suspending
>> > > +Dom0 and Xen. System suspend initiated by Dom0 (step 2) is
>> > > +considered to be an ultimate decision to suspend the physical
>> > > +machine. Suspending of Xen (step 3) is triggered whenever Dom0
>> > > +completes suspend. Xen suspend leads to the full suspend of EL2.
>> > > +
>> > > +If an unprivileged guest is not suspended at the moment when Dom0
>> > > +initiates its own suspend, the guest will be paused on Xen's
>> > > +suspend and unpaused on Xen's resume. That way, a guest which
>> > > +doesn't have power management support cannot prevent the physical
>> > > +system from suspending when the decision to suspend is made by
>> > > +privileged software
>> > > (Dom0).
>> > > +
>> > > +Each guest in the system is responsible for suspending the devices it 
>> > > owns.
>> > > +If a guest does not suspend a device, the device will continue to
>> > > +operate as it is configured at the moment when the system suspends.
>> > > +If a device triggers an interrupt while the physical system is
>> > > +suspended, the
>> > > system will resume.
>> > > +
>> > > +It is recommended for an unprivileged guest to participate in power
>> > > +management in the following scenario:
>> > > +Assume unprivileged guest owns a device which will trigger
>> > > +interrupt at some point. This interrupt will wake-up the system. If
>> > > +such a behavior is not wanted, coordination between Dom0 and the
>> > > +guest is required in order to inform the guest about the intended 
>> > > physical
>> system suspend.
>> > > +Then, the guest will have a chance to suspend the device or respond
>> > > +to the
>> > > request in an abort fashion.
>> > > +
>> > > +Since this proposal is focused on implementing PSCI-based suspend
>> > > +mechanisms in Xen, communication with or among the guests is not
>> > > +covered by
>> > > this document.
>> > > +The order of suspending the guests is assumed to be guaranteed by
>> > > +the software running in EL1.
>> > > +
>> > > +This document covers the following:
>> > > +1) Mechanism for suspending/resuming a guest:
>> > > + 1.1) Suspend is initiated by the guest
>> > > + 1.2) Resume is initiated by a device interrupt
>> > > +2) Mechanism for pausing/unpausing running guests when Dom0
>> > > +suspends
>> >
>> > Will this take care of passthroughed devices for DomU?
>>
>
> Thanks for your reply. Sorry for late reply
>
>> Hi Peng,
>>
>> The ZynqMP uses the EEMI Firmware interface to do power-management.
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.x
>> ilinx.com%2Fsupport%2Fdocumentation%2Fuser_guides%2Fug1200-eemi-api.p
>> df&data=02%7C01%7Cpeng.fan%40nxp.com%7C021307a245394e945cbf08d59
>> 30ebb3c%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C6365766138
>> 46476140&sdata=xwyil1ar7VXXYPJb2yXxYPWJvR5mVEb6wokggdt0ZH4%3D&re
>> served=0
>
> Yes. I see.
>
>>
>> In our case, we've implemented an EEMI mediator in Xen that traps EEMI
>> requests from domU's and makes sure that the guest owns the device it is 
>> trying
>> to operate on.
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
>> com%2FXilinx%2Fxen%2Fblob%2Fxilinx%2Fstable-4.9%2Fxen%2Farch%2Farm%
>> 2Fplatforms%2Fxilinx-zynqmp-eemi.c&data=02%7C01%7Cpeng.fan%40nxp.com
>> %7C021307a245394e945cbf08d5930ebb3c%7C686ea1d3bc2b4c6fa92cd99c5c3
>> 01635%7C0%7C0%7C636576613846476140&sdata=33AdCyBLxUYIR6h%2BtZzx
>> TrnYpOZ86IMFySmjHA2%2Fits%3D&reserved=0
>>
>> So domU will first issue the usual EEMI calls as it would in a 
>> non-virtualized case
>> to suspend all it's devices. Once that has happened, the guest will issue 
>> PSCI calls
>> to suspend the VM. So, Mirela please shim in if I missed something.
>>
>> The EEMI mediator has been posted to the ML but is currently sitting in our 
>> tree
>> waiting for us to go through the upstreaming effort.
>
> So if Dom0 and DomU both running Linux, in DomU, "echo mem >/sys/power/state" 
> to suspend
> DomU, then in Dom0 "echo mem >/sys/power/state" to suspend Dom0?

That is correct. Xen suspend will be triggered when Dom0 completes suspend.

>
> Thanks,
> Peng.
>
>>
>> Cheers,
>> Edgar
>>
>>
>>
>>
>> >
>> > Thanks,
>> > Peng.
>> >
>> > > +3) Mechanism for suspending/resuming Xen when Dom0 completes
>> > > +suspend
>> > > +4) Resuming from any state on a wake-up event (device interrupt):
>> > > + 4.1) Resume DomU on wake-up event when Dom0 is still running
>> > > + 4.2) Resume DomU on wake-up event when Xen is suspended
>> > > + 4.3) Resume Dom0 on wake-up event
>> > > +
>> > > +Mechanisms enumerated above will allow different kind of policies
>> > > +and coordination among guests to be implemented in EL1. That is out
>> > > +of the scope of this document.
>> > > +
>> > > +-----------------
>> > > +Suspending Guests
>> > > +-----------------
>> > > +
>> > > +Suspend procedure for a guest consists of the following:
>> > > +1) Suspending devices
>> > > +2) Suspending non-boot CPUs (based on hotplug/PSCI)
>> > > +3) System suspend, performed by the boot CPU
>> > > +
>> > > +Each guest should suspend the devices it owns just like it would
>> > > +when running without Xen.
>> > > +
>> > > +Guests should suspend their non-boot vCPUs using the hotplug mechanism.
>> > > +Virtual CPUs should be put offline using the already implemented
>> > > +PSCI vCPU_OFF call (prefix 'v' is added to distinguish PSCI calls
>> > > +made by guests to Xen, which affect virtual machines; as opposed to
>> > > +PSCI calls made by Xen to the EL3, which can affect power state of
>> > > +the physical
>> > > machine).
>> > > +
>> > > +After suspending its non-boot vCPUs a guest should finalize the
>> > > +suspend by making the vSYSTEM_SUSPEND PSCI call. The resume address
>> > > +is specified by the guest via the vSYSTEM_SUSPEND
>> > > +entry_point_address argument. The vSYSTEM_SUSPEND call is currently
>> not implemented in Xen.
>> > > +
>> > > +It is expected that a guest leaves enabled all interrupts that
>> > > +should wake it up. Other interrupts should be disabled by the guest
>> > > +prior to calling vSYSTEM_SUSPEND.
>> > > +
>> > > +After an unprivileged guest suspends, Xen will not suspend. Xen
>> > > +would suspend only after the Dom0 completes the system suspend.
>> > > +
>> > > +--------------
>> > > +Suspending Xen
>> > > +--------------
>> > > +
>> > > +Xen should start suspending itself upon receiving the
>> > > +vSYSTEM_SUSPEND call from the last running guest (Dom0). At that
>> > > +moment all physical CPUs are still online (taking offline a vCPU or
>> > > +suspending a VM does not affect
>> > > physical CPUs).
>> > > +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF
>> > > +PSCI call to EL3. The CPU_OFF PSCI function is currently not 
>> > > implemented in
>> Xen.
>> > > +
>> > > +After putting offline the non-boot cores Xen must save the context
>> > > +and finalize suspend by invoking SYSTEM_SUSPEND PSCI call, which is
>> passed to EL3.
>> > > +The resume point of Xen is specified by the entry_point_address
>> > > +argument of the SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function
>> > > +and context saving is not implemented in Xen for ARM today.
>> > > +
>> > > +------------
>> > > +Resuming Xen
>> > > +------------
>> > > +
>> > > +Xen must be resumed prior to any software running in EL1. Starting
>> > > +from the resume point, Xen should restore the context and resume
>> > > +Dom0. Dom0 shall always be resumed whenever Xen resumes.
>> > > +
>> > > +---------------
>> > > +Resuming Guests
>> > > +---------------
>> > > +
>> > > +Resume of the privileged guest (Dom0) is always following the Xen 
>> > > resume.
>> > > +
>> > > +An unprivileged guest shall resume once a device it owns triggers a
>> > > +wake-up interrupt, regardless of whether Xen was suspended when the
>> > > +wake-up interrupt was triggered. If Xen was suspended, it is
>> > > +assumed that Dom0 will be running before the DomU guest starts to
>> > > +resume. The synchronization mechanism to enforce the assumed condition
>> is TBD.
>> > > +
>> > > +If the ARM's GIC was powered down after the ARM subsystem
>> > > +suspended, it is assumed that Xen needs to restore the GIC
>> > > +interface for a VM prior to handing over control to the guest.
>> > > +However, the guest should restore its own context upon entering the
>> > > +resume point, just like it would when
>> > > running without Xen.
>> > > +
>> > > +===============
>> > > +Implementation
>> > > +===============
>> > > +
>> > > +--------
>> > > +Overview
>> > > +--------
>> > > +
>> > > +In order to enable the suspend/resume of VMs and Xen itself, the
>> > > +following PSCI calls have to be implemented and integrated in Xen:
>> > > +1) vSYSTEM_SUSPEND
>> > > +2) CPU_OFF
>> > > +3) SYSTEM_SUSPEND
>> > > +
>> > > +In addition, the following have to be implemented:
>> > > +* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
>> > > +* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0),
>> > > including:
>> > > + * Disable wathdog on suspend, enable it on resume
>> > > + * Pause domains on suspend, unpause them on resume
>> > > + * Disable non-boot pCPUs on suspend, enable them on resume
>> > > + * Save/restore of GIC configuration
>> > > + * Suspend/resume timer
>> > > + * Save/restore of EL2 context
>> > > + * Implement resume entry point in Xen, including MMU configuration
>> > > +
>> > > +Implementation details are provided in the sections below. Function
>> > > +names and paths used below are consistent within the document but
>> > > +may not always match the names used in future implementation.
>> > > +Existing functions and paths are named as in Xen source tree.
>> > > +
>> > > +-------------------------------------
>> > > +Suspend/Resume Implementation Details
>> > > +-------------------------------------
>> > > +
>> > > +PSCI Implementation and Integration
>> > > +-----------------------------------
>> > > +vSYSTEM_SUSPEND
>> > > +---------------
>> > > +vSYSTEM_SUSPEND shall be implemented in
>> > > +* do_psci_system_suspend() in arch/arm/vpsci.c
>> > > +* Code independent from PSCI interface will be added in
>> > > +arch/arm/suspend.c
>> > > +
>> > > +The implementation shall include the following steps:
>> > > +* Suspend the current (calling) vCPU. Consists of 2 major steps:
>> > > +1) Reset context of vCPU and save entry point into PC and context
>> > > +ID into X0 (entry point and context ID are provided via
>> > > +vSYSTEM_SUSPEND
>> > > +arguments)
>> > > +2) Block vCPU to ensure that it is not scheduled until it is
>> > > +unblocked by an interrupt.
>> > > +In step 1) above, the context is reset in order to prepare the vCPU
>> > > +for resume, i.e. to save vCPU context that matches reset values as
>> > > +expected by software on resume. This doesn't hold for PC and X0,
>> > > +since the PC contains resume entry point and X0 contains context ID, as
>> defined by PSCI.
>> > > +* If the hardware domain made the call trigger Xen suspend, i.e.
>> > > +  call machine_suspend() which will be implemented in
>> > > +arch/arm/suspend.c  (similar as the machine_restart() is
>> > > +implemented in
>> > > +arch/arm/shutdown.c)
>> > > +
>> > > +The function do_psci_system_suspend() shall be called from
>> > > +* do_trap_psci() in arch/arm/traps.c
>> > > +
>> > > +CPU_OFF (physical CPUs)
>> > > +-----------------------
>> > > +The CPU_OFF function shall be implemented in
>> > > +* call_psci_cpu_off() in arch/arm/psci.c
>> > > +
>> > > +The implementation shall consist just of making the SMC call to EL3.
>> > > +
>> > > +This function needs to be called when Xen generic code disables a
>> > > +non-boot
>> > > CPU.
>> > > +When a CPU is disabled it will loop forever in while loop
>> > > +(stop_cpu() function which is already implemented in
>> > > +xen/arch/arm/smpboot.c). Call to
>> > > +call_psci_cpu_off() shall be made before the CPU enters infinite loop.
>> > > +
>> > > +SYSTEM_SUSPEND (physical)
>> > > +-------------------------
>> > > +The SYSTEM_SUSPEND function shall be implemented in
>> > > +* call_psci_system_suspend() in arch/arm/psci.c
>> > > +
>> > > +The implementation shall consist just of making the SMC call to EL3.
>> > > +The entry_point_address argument of the SMC call needs to be an ARM
>> > > +architecture resume address, which shall be implemented, e.g. as
>> > > +hyp_resume() in arch/arm/arm64/entry.S. The
>> > > +call_psci_system_suspend()
>> > > function does not return.
>> > > +On the resume, the execution flow continues from hyp_resume.
>> > > +
>> > > +The function needs to be called from machine_suspend() to finalize
>> > > +the suspend procedure.
>> > > +
>> > > +------------------
>> > > +Additional Changes
>> > > +------------------
>> > > +
>> > > +Suspend Flow
>> > > +------------
>> > > +The suspend procedure shall be implemented in
>> > > +* machine_suspend() in arch/arm/suspend.c
>> > > +
>> > > +The implementation shall include the following steps:
>> > > +* Move the execution to boot pCPU
>> > > +* Set the system_state variable to SYS_STATE_suspend
>> > > +* Disable watchdog
>> > > +* Freeze domains by calling domain_pause() for each domain
>> > > +* Disable non-boot CPUs by calling disable_nonboot_cpus()
>> > > +* Disable interrupts
>> > > +* Suspend timer
>> > > +* Save GIC context. Shall be implemented in arch/arm/gic.c,
>> > > +  include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be
>> supported).
>> > > +* Save CPU context. This shall be implemented in assembly, in
>> > > +hyp_suspend()
>> > > +  in arch/arm/arm64/entry.S. The context consists of callee-saved
>> > > +general
>> > > +  purpose registers, as well as few system registers. Context of
>> > > +registers shall
>> > > +  be saved in a statically allocated structure.
>> > > +* Finalize the suspend by calling call_psci_system_suspend()
>> > > +
>> > > +Resume Flow
>> > > +------------
>> > > +The resume entry point shall be implemented in
>> > > +* hyp_resume() in arch/arm/arm64/entry.S The very beginning of the
>> > > +resume procedure has to be implemented in assembly.
>> > > +It shall contain the following:
>> > > +* Enable the MMU so that the structure containing CPU context which
>> > > +was saved on suspend can be accessed
>> > > +* Restore CPU context (to match the values saved on suspend) and
>> > > +return into C
>> > > +* Set the system_state variable to SYS_STATE_resume
>> > > +* Restore GIC context
>> > > +* Resume timer
>> > > +* Enable interrupts
>> > > +* Enable non-boot CPUs by calling enable_nonboot_cpus()
>> > > +* Thaw domains by calling domain_unpause() for each domain
>> > > +* Enable watchdog
>> > > +* Set the system_state variable to SYS_STATE_active
>> > > +* Resume Dom0
>> > > +
>> > > +==========
>> > > +References
>> > > +==========
>> > > +
>> > > +[1] Power State Coordination Interface (ARM):
>> > > +https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fi
>> > > +nfoc
>> > >
>> +enter.arm.com%2Fhelp%2Ftopic%2Fcom.arm.doc.den0022d%2FPower_State
>> > > _Coord
>> > >
>> +ination_Interface_PDD_v1_1_DEN0022D.pdf&data=02%7C01%7Cpeng.fan%4
>> > > 0nxp.c
>> > >
>> +om%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c6fa92cd99
>> > > c5c30163
>> > >
>> +5%7C0%7C1%7C636495614074885940&sdata=3ycqEZR9XgcqdvrmJKY86aukt
>> > > %2BQv%2BS
>> > > +BSZMxbCrpraEY%3D&reserved=0
>> > > --
>> > > 2.13.0
>> > >
>> > >
>> > > _______________________________________________
>> > > Xen-devel mailing list
>> > > Xen-devel@xxxxxxxxxxxxxxxxxxxx
>> > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>> > > ists.xe
>> > >
>> nproject.org%2Fmailman%2Flistinfo%2Fxen-devel&data=02%7C01%7Cpeng.fa
>> > >
>> n %40nxp.com%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c
>> 6f
>> > >
>> a92cd99c5c301635%7C0%7C0%7C636495614074885940&sdata=YLuJhbx%2B1
>> > > tDvblYbgtOZZBhsG36%2BUhpRc4VpSpHHM%2FU%3D&reserved=0

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel