|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [RFC PATCH 00/31] CPUFreq on ARM
Hi Oleksandr, I just wanted to tell you that the patch series is very well organized and the patches very nicely split. Thank you! - Stefano On Thu, 9 Nov 2017, Oleksandr Tyshchenko wrote: > From: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx> > > Hi, all. > > The purpose of this RFC patch series is to add CPUFreq support to Xen on ARM. > Motivation of hypervisor based CPUFreq is to enable one of the main PM > use-cases in virtualized system powered by Xen hypervisor. Rationale behind > this activity is that CPU virtualization is done by hypervisor and the guest > OS doesn't actually know anything about physical CPUs because it is running > on virtual CPUs. It is quite clear that a decision about frequency change > should be taken by hypervisor as only it has information about actual CPU > load. Although these required components (CPUFreq core, governors, etc) > already exist in Xen, it is worth to mention that they are ACPI specific. So, > a part of the current patch series makes them more generic in order to make > possible a CPUFreq usage on architectures without ACPI support in. > But, the main question we have to answer is about frequency changing > interface in virtualized system. The frequency changing interface and all > dependent components which needed CPUFreq to be functional on ARM are not > present in Xen these days. The list of required components is quite big and > may change across different ARM SoC vendors. As an example, the following > components are involved in DVFS on Renesas Salvator-X board which has R-Car > Gen3 SoC installed: generic clock, regulator and thermal frameworks, Vendor’s > CPG, PMIC, AVS, THS drivers, i2c support, etc. > > We were considering a few possible approaches of hypervisor based CPUFreqs on > ARM and came to conclusion to base this solution on popular at the moment, > already upstreamed to Linux, ARM System Control and Power Interface(SCPI) > protocol [1]. We chose SCPI protocol instead of newer ARM System Control and > Management Interface (SCMI) protocol [2] since it is widely spread in Linux, > there are good examples how to use it, the range of capabilities it has is > enough for implementing hypervisor based CPUFreq and, what is more, upstream > Linux support for SCMI is missed so far, but SCMI could be used as well. > > Briefly speaking, the SCPI protocol is used between the System Control > Processor(SCP) and the Application Processors(AP). The mailbox feature > provides a mechanism for inter-processor communication between SCP and AP. > The main purpose of SCP is to offload different PM related tasks from AP and > one of the services that SCP provides is Dynamic voltage and frequency > scaling (DVFS), it is what we actually need for CPUFreq. I will describe this > approach in details down the text. > > Let me explain a bit more what these possible approaches are: > > 1. “Xen+hwdom” solution. > GlobalLogic team proposed split model [3], where “hwdom-cpufreq” frontend > driver in Xen interacts with the “xen-cpufreq” backend driver in Linux hwdom > (possibly dom0) in order to scale physical CPUs. This solution hasn’t been > accepted by Xen community yet and seems it is not going to be accepted > without taking into the account still unanswered major questions and proving > that “all-in-Xen” solution, which Xen community considered as more > architecturally cleaner option, would be unworkable in practice. > The other reasons why we decided not to stick to this approach are complex > communication interface between Xen and hwdom: event channel, hypercalls, > syscalls, passing CPU info via DT, etc and possible synchronization issues > with a proposed solution. > Although it is worth to mention that the beauty of this approach was that > there wouldn’t be a need to port a lot of things to Xen. All frequency > changing interface and all dependent components which needed CPUFreq to be > functional were already in place. > Although this approach is not used, still I picked a few already acked > patches which made ACPI specific CPUFreq stuff more generic. > > 2. “all-in-Xen” solution. > This implies that all CPUFreq related stuff should be located in Xen. > Community considered this solution as more architecturally cleaner option > than “Xen+hwdom” one. No layering violation comparing with the previous > approach (letting guest OS manage one or more physical CPUs is more of a > layering violation). > This solution looks better, but to be honest, we are not in favor of this > solution as well. We expect enormous developing effort to get this support in > (the scope of required components looks unreal) and maintain it. So, we > decided not to stick to this approach as well. > > 3. “Xen+SCP(ARM TF)” solution. > It is yet another solution based on ARM SCPI protocol. The generic idea here > is that there is a firmware, which being a server runs on some dedicated IP > core (server), provides different PM services (DVFS, sensors, etc). On the > other side there is a CPUFreq driver in Xen, which is running on the AP > (client), consumes these services. CPUFreq driver neither changes the CPU > frequency/voltage by itself nor cooperates with Linux in order to do such > job. It just communicates with SCP directly using SCPI protocol. As I said > before, some integrated into a SoC mailbox IP need to be used for IPC > (doorbell for triggering action and shared memory region for commands). > CPUFreq driver doesn’t even need to know what should be physically changed > for the new frequency to take effect. It is a certainly SCP’s responsibility. > This all avoid CPUFreq infrastructure in Xen on ARM from diving into each > supported SoC internals and as the result having a lot of code. > > The possible issue here could be in SCP, the problem is that some dedicated > IP core may be absent at all or performs other than PM tasks. Fortunately, > there is a brilliant solution to teach firmware running in the EL3 exception > level (ARM TF) to perform SCP functions and use SMC calls for communications > [4]. Exactly this transport implementation I want to bring to Xen the first. > Such solution is going to be generic across all ARM platforms that do have > firmware running in the EL3 exception level and don’t have candidate for > being SCP. > > Here we have completely synchronous case because of SMC calls nature. SMC > triggered mailbox driver emulates a mailbox which signals transmitted data > via Secure Monitor Call (SMC) instruction [5]. The mailbox receiver is > implemented in firmware and synchronously returns data when it returns > execution to the non-secure world again. This would allow us both to trigger > a request and transfer execution to the firmware code in a safe and > architected way. Like PSCI requests. > As you can see this method is free from synchronization issues. What is more, > this solution is more architecturally cleaner solution than split model > “Xen+hwdom” one. From the security point of view, I hope, everything will be > much more correct since the ARM TF, which we want to see in charge of > controlling CPU frequency/voltage, is a trusted SW layer. Moreover, ARM TF is > responsible for enabling/disabling CPU (PSCI) and nobody complains about it, > so let it do DVFS too. > > I have to admit that I have checked this solution only due to a lack of > candidate for being SCP. But, I hope, that other ARM SoCs where dedicated SCP > is present (asynchronous case) will work too, but with some limitations. The > mailbox IPs for these ARM SoCs must have TX/RX-done irqs. I have described in > the corresponding patches why this limitation is present. > > To be honest I have Renesas R-Car Gen3 SoCs in mind as our nearest target, > but I would like to make this solution as generic as possible. I don’t treat > proposed solution as world-wide generic, but I hope, this solution may be > suitable for other ARM SoCs which meet such requirements. Anyway, having > something which works, but doesn’t cover all cases is better than having > nothing. > > I would like to notice that the patches are POC state and I post them just to > illustrate in more detail of what I am talking about. Patch series consist of > the following parts: > 1. GL’s patches which make ACPI specific CPUFreq stuff more generic. Although > these patches has been already acked by Xen community and the CPUFreq code > base hasn’t changed in a last few years I drop all A-b. > 2. A bunch of device-tree helpers and macros. > 3. Direct ported SCPI protocol, mailbox infrastructure and the ARM SMC > triggered mailbox driver. All components except mailbox driver are in > mainline Linux. > 4. Xen changes to direct ported code for making it compilable. These changes > don’t change functionality. > 5. Some modification to direct ported code which slightly change > functionality, I would say to restrict it. > 6. SCPI based CPUFreq driver and CPUFreq interface component. > 7. Misc patches mostly to ARM subsystem. > 8. Patch from Volodymyr Babchuk which adds SMC wrapper. > > Most important TODOs regarding the whole patch series: > 1. Handle devm in the direct ported code. Currently, in case of any errors > previously allocated resources are left unfreed. > 2. Thermal management integration. > 3. Don't pass CPUFreq related nodes to dom0. Xen owns SCPI completely. > 4. Handle CPU_TURBO frequencies if they are supported by HW. > > You can find the whole patch series here: > repo: https://github.com/otyshchenko1/xen.git branch: cpufreq-devel1 > > P.S. There is no need to modify xenpm tool. It works out of the box on ARM. > > [1] > Linux code: > http://elixir.free-electrons.com/linux/v4.14-rc6/source/drivers/firmware/arm_scpi.c > http://elixir.free-electrons.com/linux/v4.14-rc6/source/include/linux/scpi_protocol.h > http://elixir.free-electrons.com/linux/v4.14-rc6/source/Documentation/devicetree/bindings/arm/arm,scpi.txt > > Recent protocol version: > http://infocenter.arm.com/help/topic/com.arm.doc.dui0922g/scp_message_interface_v1_2_DUI0922G_en.pdf > > [2] > Xen part: > https://lists.xen.org/archives/html/xen-devel/2014-11/msg00940.html > Linux part: > https://lists.xen.org/archives/html/xen-devel/2014-11/msg00944.html > > [3] > http://infocenter.arm.com/help/topic/com.arm.doc.den0056a/DEN0056A_System_Control_and_Management_Interface.pdf > > [4] > http://linux-sunxi.narkive.com/qYWJqjXU/patch-v2-0-3-mailbox-arm-introduce-smc-triggered-mailbox > > [5] > http://infocenter.arm.com/help/topic/com.arm.doc.den0028b/ARM_DEN0028B_SMC_Calling_Convention.pdf > > Oleksandr Dmytryshyn (6): > cpufreq: move cpufreq.h file to the xen/include/xen location > pm: move processor_perf.h file to the xen/include/xen location > pmstat: move pmstat.c file to the xen/drivers/pm/stat.c location > cpufreq: make turbo settings to be configurable > pmstat: make pmstat functions more generalizable > cpufreq: make cpufreq driver more generalizable > > Oleksandr Tyshchenko (24): > xenpm: Clarify xenpm usage > xen/device-tree: Add dt_count_phandle_with_args helper > xen/device-tree: Add dt_property_for_each_string macros > xen/device-tree: Add dt_property_read_u32_index helper > xen/device-tree: Add dt_property_count_elems_of_size helper > xen/device-tree: Add dt_property_read_string_helper and friends > xen/arm: Add driver_data field to struct device > xen/arm: Add DEVICE_MAILBOX device class > xen/arm: Store device-tree node per cpu > xen/arm: Add ARM System Control and Power Interface (SCPI) protocol > xen/arm: Add mailbox infrastructure > xen/arm: Introduce ARM SMC based mailbox > xen/arm: Add common header file wrappers.h > xen/arm: Add rxdone_auto flag to mbox_controller structure > xen/arm: Add Xen changes to SCPI protocol > xen/arm: Add Xen changes to mailbox infrastructure > xen/arm: Add Xen changes to ARM SMC based mailbox > xen/arm: Use non-blocking mode for SCPI protocol > xen/arm: Don't set txdone_poll flag for ARM SMC mailbox > cpufreq: hack: perf->states isn't a real guest handle on ARM > xen/arm: Introduce SCPI based CPUFreq driver > xen/arm: Introduce CPUFreq Interface component > xen/arm: Build CPUFreq components > xen/arm: Enable CPUFreq on ARM > > Volodymyr Babchuk (1): > arm: add SMC wrapper that is compatible with SMCCC > > MAINTAINERS | 4 +- > tools/misc/xenpm.c | 6 +- > xen/arch/arm/Kconfig | 2 + > xen/arch/arm/Makefile | 1 + > xen/arch/arm/arm32/Makefile | 1 + > xen/arch/arm/arm32/smc.S | 32 + > xen/arch/arm/arm64/Makefile | 1 + > xen/arch/arm/arm64/smc.S | 29 + > xen/arch/arm/cpufreq/Makefile | 5 + > xen/arch/arm/cpufreq/arm-smc-mailbox.c | 248 ++++++ > xen/arch/arm/cpufreq/arm_scpi.c | 1191 > ++++++++++++++++++++++++++ > xen/arch/arm/cpufreq/cpufreq_if.c | 522 +++++++++++ > xen/arch/arm/cpufreq/mailbox.c | 562 ++++++++++++ > xen/arch/arm/cpufreq/mailbox.h | 28 + > xen/arch/arm/cpufreq/mailbox_client.h | 69 ++ > xen/arch/arm/cpufreq/mailbox_controller.h | 161 ++++ > xen/arch/arm/cpufreq/scpi_cpufreq.c | 328 +++++++ > xen/arch/arm/cpufreq/scpi_protocol.h | 116 +++ > xen/arch/arm/cpufreq/wrappers.h | 239 ++++++ > xen/arch/arm/smpboot.c | 5 + > xen/arch/x86/Kconfig | 2 + > xen/arch/x86/acpi/cpu_idle.c | 2 +- > xen/arch/x86/acpi/cpufreq/cpufreq.c | 2 +- > xen/arch/x86/acpi/cpufreq/powernow.c | 2 +- > xen/arch/x86/acpi/power.c | 2 +- > xen/arch/x86/cpu/mwait-idle.c | 2 +- > xen/arch/x86/platform_hypercall.c | 2 +- > xen/common/device_tree.c | 124 +++ > xen/common/sysctl.c | 2 +- > xen/drivers/Kconfig | 2 + > xen/drivers/Makefile | 1 + > xen/drivers/acpi/Makefile | 1 - > xen/drivers/acpi/pmstat.c | 526 ------------ > xen/drivers/cpufreq/Kconfig | 3 + > xen/drivers/cpufreq/cpufreq.c | 102 ++- > xen/drivers/cpufreq/cpufreq_misc_governors.c | 2 +- > xen/drivers/cpufreq/cpufreq_ondemand.c | 4 +- > xen/drivers/cpufreq/utility.c | 13 +- > xen/drivers/pm/Kconfig | 3 + > xen/drivers/pm/Makefile | 1 + > xen/drivers/pm/stat.c | 538 ++++++++++++ > xen/include/acpi/cpufreq/cpufreq.h | 245 ------ > xen/include/acpi/cpufreq/processor_perf.h | 63 -- > xen/include/asm-arm/device.h | 2 + > xen/include/asm-arm/processor.h | 4 + > xen/include/public/platform.h | 1 + > xen/include/xen/cpufreq.h | 254 ++++++ > xen/include/xen/device_tree.h | 158 ++++ > xen/include/xen/pmstat.h | 2 + > xen/include/xen/processor_perf.h | 69 ++ > 50 files changed, 4822 insertions(+), 862 deletions(-) > create mode 100644 xen/arch/arm/arm32/smc.S > create mode 100644 xen/arch/arm/arm64/smc.S > create mode 100644 xen/arch/arm/cpufreq/Makefile > create mode 100644 xen/arch/arm/cpufreq/arm-smc-mailbox.c > create mode 100644 xen/arch/arm/cpufreq/arm_scpi.c > create mode 100644 xen/arch/arm/cpufreq/cpufreq_if.c > create mode 100644 xen/arch/arm/cpufreq/mailbox.c > create mode 100644 xen/arch/arm/cpufreq/mailbox.h > create mode 100644 xen/arch/arm/cpufreq/mailbox_client.h > create mode 100644 xen/arch/arm/cpufreq/mailbox_controller.h > create mode 100644 xen/arch/arm/cpufreq/scpi_cpufreq.c > create mode 100644 xen/arch/arm/cpufreq/scpi_protocol.h > create mode 100644 xen/arch/arm/cpufreq/wrappers.h > delete mode 100644 xen/drivers/acpi/pmstat.c > create mode 100644 xen/drivers/pm/Kconfig > create mode 100644 xen/drivers/pm/Makefile > create mode 100644 xen/drivers/pm/stat.c > delete mode 100644 xen/include/acpi/cpufreq/cpufreq.h > delete mode 100644 xen/include/acpi/cpufreq/processor_perf.h > create mode 100644 xen/include/xen/cpufreq.h > create mode 100644 xen/include/xen/processor_perf.h > > -- > 2.7.4 > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |