Re: [Xen-devel] [PATCH v5 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes

[Julien Grall <julien.grall@xxxxxxx>]

Hi,

Thank you for the series. I have pushed the series in a branch "next" 
and will merge it once Xen 4.11 has been released.
One thing that has not been handled in this series is moving the 
interrupts used by Xen (e.g UART, SMMU...) away from the core that is 
powerdown. This is required by the PSCI-spec (see 5.5 in DEN0022C) to 
avoid errorneous state.
I think this will be quite critical for suspend/resume as we want the 
wake-up even to happen on CPU0 (the one who called PSCI_suspend).
Cheers,

On 01/06/18 14:17, Mirela Simonovic wrote:
> This patch set contains fixes that are required as precondition for suspend to
> RAM support, including the CPU hotplug which is required to suspend non-boot
> CPUs.
> The first two patches in this series:
> 1) xen/arm64: Added handling of the trapped access to OSLSR register
> 2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
> are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
> patch set does not include the implementation of virtual PSCI system suspend
> call that would allow guests to finalize their suspend procedures. This will
> be submitted in the following series.
>
> Remaining of the patches are related to enabling CPU hotplug for non-boot
> CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for suspend
> to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot CPUs
> will be triggered with disable_nonboot_cpus(), while the hotplug is triggered
> with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs could
> be powered down/up on suspend/resume, respectively, if the underlying firmware
> allows so. Calls to enable/disable_nonboot_cpus() functions currently do not
> exist in Xen ARM code. This will be added with the suspend to RAM support for
> ARM.
>
> When non-boot pCPUs are hot-unplugged their interrupts are migrated to the boot
> pCPU. This series also includes a fix that would restore the interrupts affinity
> once non-boot pCPUs are hotplugged. Here only SPIs used by guests are covered.
> Migration of Xen internal SPIs is not covered. According to my understanding
> Xen internal SPIs are routed to the boot CPU which initializes the respective
> devices. Therefore, there is no need to migrate Xen internal SPIs.
>
> The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
> physical power down/up of non-boot CPUs). The testing requires additional
> patches for issuing system suspend. These patches and instructions for testing
> will be submitted later, when we get closer to the final version of the series.
>
> ---
> Changes in v2:
> -Rename cover-letter title and emphasize that 2 patches from this series are not
> specific to CPU hotplug (my initial fault, splitting it now could be confusing)
> -Fix cover-letter explanations
> -Address all the issues and comments as discussed on mailing list for v1
> -Add 3 patches to ensure that suspend/resume does not cause any memory leaks.
> All the memory allocated when a CPU was hotplugged is now freed when the CPU is
> hot-unplugged.
> -Remove from the v1 series the patch which incorrectly dealt with an issue:
> [PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
> One solution to the issue addressed by the patch above is to add rcu_barrier()
> prior to calling enable_nonboot_cpus() during the suspend. This is how it is
> done in x86 suspend implementation. Until the discussion here
> https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
> doesn't conclude differently, I need to assume that adding rcu_barrier() prior
> to calling enable_nonboot_cpus() as it is done for x86 is the right way to go.
> Therefore, the fix to the issue will be part of the suspend to RAM series.
>
> Changes in v3:
> -Add acked-by where needed
> -Fix CPU_OFF PSCI implementation (physical interface)
> -Use notifiers to implement freeing memory and releasing interrupts on CPU
> hotplug
> -Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
> hotplug
> -Add enabling errata workarounds on CPU hotplug, also based on a notifier
> -Remove patch:
> [PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU only on 
> boot
>
> Changes in v4:
> -Add acked-by/reviewed-by where needed
> -Cleanup: use smp_processor_id() instead of get_processor_id(), fixed
>   indentation, add includes alphabetically, add newline before return, etc.
> -Disable timers prior to releasing timer interrupts
> -Initialize cpu_smpboot notifier at presmp_init rather than init phase
> -In the last patch of the series errata notifier now returns an error
>
> Changes in v5:
> -Introduce handle_ro_read_val() to handle traps as read-only as fixed value
> -Fix handling accesses to OSLSR_EL1
> -Fix variable type in 5th patch
>
> ---
> CC: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> CC: Julien Grall <julien.grall@xxxxxxx>
> CC: George Dunlap <george.dunlap@xxxxxxxxxxxxx>
> CC: Dario Faggioli <dfaggioli@xxxxxxxx>
> ---
>
> Mirela Simonovic (10):
>    xen/arm64: Added handling of the trapped access to OSLSR register
>    xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
>    xen/arm: Implement CPU_OFF PSCI call (physical interface)
>    xen/arm: Remove __initdata and __init to enable CPU hotplug
>    xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume
>    xen/common: Restore IRQ affinity when hotplugging a pCPU
>    xen/arm: Release maintenance interrupt when CPU is hot-unplugged
>    xen/arm: Disable timers and release their interrupts on CPU hot-unplug
>    xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug
>    xen/arm: Enable errata for secondary CPU on hotplug after the boot
>
>   xen/arch/arm/arm64/smpboot.c     |  2 +-
>   xen/arch/arm/arm64/vsysreg.c     |  4 ++-
>   xen/arch/arm/cpuerrata.c         | 49 +++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/cpufeature.c        | 29 ++++++++++++++++++++++
>   xen/arch/arm/gic.c               | 31 +++++++++++++++++++++++
>   xen/arch/arm/irq.c               |  2 +-
>   xen/arch/arm/p2m.c               | 53 ++++++++++++++++++++++++++++++++++++----
>   xen/arch/arm/processor.c         |  2 +-
>   xen/arch/arm/psci.c              | 13 ++++++++++
>   xen/arch/arm/smpboot.c           | 42 +++++++++++++++++++++++++++++--
>   xen/arch/arm/time.c              | 45 ++++++++++++++++++++++++++++++++++
>   xen/arch/arm/traps.c             | 26 ++++++++++++++------
>   xen/arch/arm/vgic-v2.c           |  2 ++
>   xen/common/schedule.c            |  4 +++
>   xen/include/asm-arm/cpufeature.h |  1 +
>   xen/include/asm-arm/procinfo.h   |  4 +--
>   xen/include/asm-arm/psci.h       |  1 +
>   xen/include/asm-arm/traps.h      |  4 +++
>   18 files changed, 293 insertions(+), 21 deletions(-)
--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel