|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] RE: Proposal for Porting Xen to Armv8-R64 - DraftB
Hi Stefano, > -----Original Message----- > From: Stefano Stabellini <sstabellini@xxxxxxxxxx> > Sent: 2022年4月15日 8:41 > To: Wei Chen <Wei.Chen@xxxxxxx> > Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx; julien@xxxxxxx; Stefano Stabellini > <sstabellini@xxxxxxxxxx>; Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; > Penny Zheng <Penny.Zheng@xxxxxxx> > Subject: Re: Proposal for Porting Xen to Armv8-R64 - DraftB > > On Fri, 25 Mar 2022, Wei Chen wrote: > > # Proposal for Porting Xen to Armv8-R64 > > > > This proposal will introduce the PoC work of porting Xen to Armv8-R64, > > which includes: > > - The changes of current Xen capability, like Xen build system, memory > > management, domain management, vCPU context switch. > > - The expanded Xen capability, like static-allocation and direct-map. > > > > ***Notes:*** > > 1. ***This proposal only covers the work of porting Xen to Armv8-R64*** > > ***single CPU.Xen SMP support on Armv8-R64 relates to Armv8-R*** > > ***Trusted-Frimware (TF-R). This is an external dependency,*** > > ***so we think the discussion of Xen SMP support on Armv8-R64*** > > ***should be started when single-CPU support is complete.*** > > 2. ***This proposal will not touch xen-tools. In current stange,*** > > ***Xen on Armv8-R64 only support dom0less, all guests should*** > > ***be booted from device tree.*** > > > > ## Changelogs > > Draft-A -> Draft-B: > > 1. Update Kconfig options usage. > > 2. Update the section for XEN_START_ADDRESS. > > 3. Add description of MPU initialization before parsing device tree. > > 4. Remove CONFIG_ARM_MPU_EL1_PROTECTION_REGIONS. > > 5. Update the description of ioremap_nocache/cache. > > 6. Update about the free_init_memory on Armv8-R. > > 7. Describe why we need to switch the MPU configuration later. > > 8. Add alternative proposal in TODO. > > 9. Add use tool to generate Xen Armv8-R device tree in TODO. > > 10. Add Xen PIC/PIE discussion in TODO. > > 11. Add Xen event channel support in TODO. > > > > ## Contributors: > > Wei Chen <Wei.Chen@xxxxxxx> > > Penny Zheng <Penny.Zheng@xxxxxxx> > > > > ## 1. Essential Background > > > > ### 1.1. Armv8-R64 Profile > > The Armv-R architecture profile was designed to support use cases that > > have a high sensitivity to deterministic execution. (e.g. Fuel Injection, > > Brake control, Drive trains, Motor control etc) > > > > Arm announced Armv8-R in 2013, it is the latest generation Arm > architecture > > targeted at the Real-time profile. It introduces virtualization at the > highest > > security level while retaining the Protected Memory System Architecture > (PMSA) > > based on a Memory Protection Unit (MPU). In 2020, Arm announced Cortex- > R82, > > which is the first Arm 64-bit Cortex-R processor based on Armv8-R64. > > > > - The latest Armv8-R64 document can be found here: > > [Arm Architecture Reference Manual Supplement - Armv8, for Armv8-R > AArch64 architecture > profile](https://developer.arm.com/documentation/ddi0600/latest/). > > > > - Armv-R Architecture progression: > > Armv7-R -> Armv8-R AArch32 -> Armv8 AArch64 > > The following figure is a simple comparison of "R" processors based on > > different Armv-R Architectures. > >  > > > > - The Armv8-R architecture evolved additional features on top of Armv7-R: > > - An exception model that is compatible with the Armv8-A model > > - Virtualization with support for guest operating systems > > - PMSA virtualization using MPUs In EL2. > > - The new features of Armv8-R64 architecture > > - Adds support for the 64-bit A64 instruction set, previously Armv8- > R > > only supported A32. > > - Supports up to 48-bit physical addressing, previously up to 32-bit > > addressing was supported. > > - Optional Arm Neon technology and Advanced SIMD > > - Supports three Exception Levels (ELs) > > - Secure EL2 - The Highest Privilege, MPU only, for firmware, > hypervisor > > - Secure EL1 - RichOS (MMU) or RTOS (MPU) > > - Secure EL0 - Application Workloads > > - Optionally supports Virtual Memory System Architecture at S-EL1/S- > EL0. > > This means it's possible to run rich OS kernels - like Linux - > either > > bare-metal or as a guest. > > - Differences with the Armv8-A AArch64 architecture > > - Supports only a single Security state - Secure. There is not Non- > Secure > > execution state supported. > > - EL3 is not supported, EL2 is mandatory. This means secure EL2 is > the > > highest EL. > > - Supports the A64 ISA instruction > > - With a small set of well-defined differences > > - Provides a PMSA (Protected Memory System Architecture) based > > virtualization model. > > - As opposed to Armv8-A AArch64's VMSA based Virtualization > > - Can support address bits up to 52 if FEAT_LPA is enabled, > > otherwise 48 bits. > > - Determines the access permissions and memory attributes of > > the target PA. > > - Can implement PMSAv8-64 at EL1 and EL2 > > - Address translation flat-maps the VA to the PA for EL2 > Stage 1. > > - Address translation flat-maps the VA to the PA for EL1 > Stage 1. > > - Address translation flat-maps the IPA to the PA for EL1 > Stage 2. > > - PMSA in EL1 & EL2 is configurable, VMSA in EL1 is configurable. > > > > ### 1.2. Xen Challenges with PMSA Virtualization > > Xen is PMSA unaware Type-1 Hypervisor, it will need modifications to run > > with an MPU and host multiple guest OSes. > > > > - No MMU at EL2: > > - No EL2 Stage 1 address translation > > - Xen provides fixed ARM64 virtual memory layout as basis of EL2 > > stage 1 address translation, which is not applicable on MPU > system, > > where there is no virtual addressing. As a result, any > operation > > involving transition from PA to VA, like ioremap, needs > modification > > on MPU system. > > - Xen's run-time addresses are the same as the link time addresses. > > - Enable PIC/PIE (position-independent code) on a real-time > target > > processor probably very rare. Further discussion in 2.1 and > TODO > > sections. > > - Xen will need to use the EL2 MPU memory region descriptors to > manage > > access permissions and attributes for accesses made by VMs at > EL1/0. > > - Xen currently relies on MMU EL1 stage 2 table to manage these > > accesses. > > - No MMU Stage 2 translation at EL1: > > - A guest doesn't have an independent guest physical address space > > - A guest can not reuse the current Intermediate Physical Address > > memory layout > > - A guest uses physical addresses to access memory and devices > > - The MPU at EL2 manages EL1 stage 2 access permissions and > attributes > > - There are a limited number of MPU protection regions at both EL2 and > EL1: > > - Architecturally, the maximum number of protection regions is 256, > > typical implementations have 32. > > - By contrast, Xen does not need to consider the number of page > table > > entries in theory when using MMU. > > - The MPU protection regions at EL2 need to be shared between the > hypervisor > > and the guest stage 2. > > - Requires careful consideration - may impact feature 'fullness' of > both > > the hypervisor and the guest > > - By contrast, when using MMU, Xen has standalone P2M table for > guest > > stage 2 accesses. > > > > ## 2. Proposed changes of Xen > > ### **2.1. Changes of build system:** > > > > - ***Introduce new Kconfig options for Armv8-R64***: > > Unlike Armv8-A, because lack of MMU support on Armv8-R64, we may not > > expect one Xen binary to run on all machines. Xen images are not > common > > across Armv8-R64 platforms. Xen must be re-built for different Armv8- > R64 > > platforms. Because these platforms may have different memory layout > and > > link address. > > - `ARM64_V8R`: > > This option enables Armv8-R profile for Arm64. Enabling this > option > > results in selecting MPU. This Kconfig option is used to gate some > > Armv8-R64 specific code except MPU code, like some code for Armv8- > R64 > > only system ID registers access. > > > > - `ARM_MPU` > > This option enables MPU on Armv8-R architecture. Enabling this > option > > results in disabling MMU. This Kconfig option is used to gate some > > ARM_MPU specific code. Once when this Kconfig option has been > enabled, > > the MMU relate code will not be built for Armv8-R64. The reason > why > > not depends on runtime detection to select MMU or MPU is that, we > don't > > think we can use one image for both Armv8-R64 and Armv8-A64. > Another > > reason that we separate MPU and V8R in provision to allow to > support MPU > > on 32bit Arm one day. > > > > ***Try to use `if ( IS_ENABLED(CONFIG_ARMXXXX) )` instead of > spreading*** > > ***`#ifdef CONFIG_ARMXXXX` everywhere, if it is possible.*** > > > > - ***About Xen start address for Armv8-R64***: > > On Armv8-A, Xen has a fixed virtual start address (link address too) > on all > > Armv8-A platforms. In an MMU based system, Xen can map its loaded > address > > to this virtual start address. On Armv8-A platforms, the Xen start > address > > does not need to be configurable. But on Armv8-R platforms, they don't > have > > MMU to map loaded address to a fixed virtual address. And different > platforms > > will have very different address space layout, so it's impossible for > Xen to > > specify a fixed physical address for all Armv8-R platforms' start > address. > > > > - `XEN_START_ADDRESS` > > This option allows to set the custom address at which Xen will be > > linked. This address must be aligned to a page size. Xen's run-time > > addresses are the same as the link time addresses. > > ***Notes: Fixed link address means the Xen binary could not be*** > > ***relocated by EFI loader. So in current stage, Xen could not*** > > ***be launched as an EFI application on Armv8-R64.(TODO#3.3)*** > > > > - Provided by platform files. > > We can reuse the existed arm/platforms store platform specific > files. > > And `XEN_START_ADDRESS` is one kind of platform specific > information. > > So we can use platform file to define default `XEN_START_ADDRESS` > for > > each platform. > > > > - Provided by Kconfig. > > This option can be an independent or a supplymental option. Users > can > > define a customized `XEN_START_ADDRESS` to override the default > value > > in platform's file. > > > > - Generated from device tree by build scripts (optional) > > Vendors who want to enable Xen on their Armv8-R platforms, they > can > > use some tools/scripts to parse their boards device tree to > generate > > the basic platform information. These tools/scripts do not > necessarily > > need to be integrated in Xen, but Xen can give some recommended > > configuration. For example, Xen can recommend Armv8-R platforms to > use > > lowest ram start address + 2MB as the default Xen start address. > > The generated platform files can be placed to arm/platforms for > > maintenance. > > > > - Enable Xen PIC/PIE (optional) > > We have mentioned about PIC/PIE in section 1.2. With PIC/PIE > support, > > Xen can run from everywhere it has been loaded. But it's rare to > use > > PIC/PIE on a real-time system (code size, more memory access). So > a > > partial PIC/PIE image maybe better (see 3. TODO section). But > partial > > PIC/PIE image may not solve this Xen start address issue. > > I like the description of the XEN_START_ADDRESS problem and solutions. > > For the initial implementation, a platform file is fine. We need to > start easy. > > Afterwards, I think it would be far better to switch to a script that > automatically generates XEN_START_ADDRESS from the host device tree. > Also, if we provide a way to customize the start address via Kconfig, > then the script that reads the device tree could simply output the right > CONFIG_* option for Xen to build. It wouldn't even have to generate an > header file. > > > > - ***About MPU initialization before parsing device tree***: > > Before Xen can start parsing information from device tree and use > > this information to setup MPU, Xen need an initial MPU state. This > > is because: > > 1. More deterministic: Arm MPU supports background regions, if we > > don't configure the MPU regions and don't enable MPU. The > default > > MPU background attributes will take effect. The default > background > > attributes are `IMPLEMENTATION DEFINED`. That means all RAM > regions > > may be configured to device memory and RWX. Random values in > RAM or > > maliciously embedded data can be exploited. > > 2. More compatible: On some Armv8-R64 platforms, if MPU is > disabled, > > the `dc zva` instruction will make the system halt (This is one > > side effect of MPU background attributes, the RAM has been > configured > > as device memory). And this instruction will be embedded in > some > > built-in functions, like `memory set`. If we use `- > ddont_use_dc` to > > rebuild GCC, the built-in functions will not contain `dc zva`. > > However, it is obviously unlikely that we will be able to > recompile > > all GCC for ARMv8-R64. > > > > - Reuse `XEN_START_ADDRESS` > > In the very beginning of Xen boot, Xen just need to cover a > limited > > memory range and very few devices (actually only UART device). So > we > > can use two MPU regions to map: > > 1. `XEN_START_ADDRESS` to `XEN_START_ADDRESS + 2MB` or. > > `XEN_START_ADDRESS` to `XEN_START_ADDRESS + image_size`as > > normal memory. > > 2. `UART` MMIO region base to `UART` MMIO region end to device > memory. > > These two are enough to support Xen run in boot time. And we don't > need > > to provide additional platform information for initial normal > memory > > and device memory regions. In current PoC we have used this option > > for implementation, and it's the same as Armv8-A. > > > > - Additional platform information for initial MPU state > > Introduce some macros to allow users to set initial normal > > memory regions: > > `ARM_MPU_NORMAL_MEMORY_START` and `ARM_MPU_NORMAL_MEMORY_END` > > and device memory: > > `ARM_MPU_DEVICE_MEMORY_START` and `ARM_MPU_DEVICE_MEMORY_END` > > These macros are the same platform specific information as > > `XEN_START_ADDRESS`, so the options#1/#2/#3 of generating > > `XEN_START_ADDRESS` also can be applied to these macros. > > ***From our current PoC work, we think these macros may*** > > ***not be necessary. But we still place them here to see*** > > ***whether the community will have some different scenarios*** > > ***that we haven't considered.*** > > I think it is fine for now. And their values could be automatically > generated by the same script that will automatically generate > XEN_START_ADDRESS from the host device tree. > > > > - ***Define new system registers for compiliers***: > > Armv8-R64 is based on Armv8.4. That means we will use some Armv8.4 > > specific system registers. As Armv8-R64 only have secure state, so > > at least, `VSTCR_EL2` and `VSCTLR_EL2` will be used for Xen. And the > > first GCC version that supports Armv8.4 is GCC 8.1. In addition to > > these, PMSA of Armv8-R64 introduced lots of MPU related system > registers: > > `PRBAR_ELx`, `PRBARx_ELx`, `PRLAR_ELx`, `PRLARx_ELx`, `PRENR_ELx` and > > `MPUIR_ELx`. But the first GCC version to support these system > registers > > is GCC 11. So we have two ways to make compilers to work properly with > > these system registers. > > 1. Bump GCC version to GCC 11. > > The pros of this method is that, we don't need to encode these > > system registers in macros by ourselves. But the cons are that, > > we have to update Makefiles to support GCC 11 for Armv8-R64. > > 1.1. Check the GCC version 11 for Armv8-R64. > > 1.2. Add march=armv8r to CFLAGS for Armv8-R64. > > 1.3. Solve the confliction of march=armv8r and mcpu=generic > > These changes will affect common Makefiles, not only Arm Makefiles. > > And GCC 11 is new, lots of toolchains and Distro haven't supported > it. > > > > 2. Encode new system registers in macros ***(preferred)*** > > ``` > > /* Virtualization Secure Translation Control Register */ > > #define VSTCR_EL2 S3_4_C2_C6_2 > > /* Virtualization System Control Register */ > > #define VSCTLR_EL2 S3_4_C2_C0_0 > > /* EL1 MPU Protection Region Base Address Register encode */ > > #define PRBAR_EL1 S3_0_C6_C8_0 > > ... > > /* EL2 MPU Protection Region Base Address Register encode */ > > #define PRBAR_EL2 S3_4_C6_C8_0 > > ... > > ``` > > If we encode all above system registers, we don't need to bump GCC > > version. And the common CFLAGS Xen is using still can be applied to > > Armv8-R64. We don't need to modify Makefiles to add specific CFLAGS. > > ***Notes:*** > > ***Armv8-R AArch64 supports the A64 ISA instruction set with*** > > ***some modifications:*** > > ***Redefines DMB, DSB, and adds an DFB. But actually, the*** > > ***encodings of DMB and DSB are still the same with A64.*** > > ***And DFB is an alias of DSB #12. In this case, we think*** > > ***we don't need a new architecture specific flag to*** > > ***generate new instructions for Armv8-R.*** > > I think that for the initial implementation either way is fine. I agree > that macros would be better than requiring GCC 11. > > > > ### **2.2. Changes of the initialization process** > > In general, we still expect Armv8-R64 and Armv8-A64 to have a consistent > > initialization process. In addition to some architecutre differences, > there > > is no more than reusable code that we will distinguish through > CONFIG_ARM_MPU > > or CONFIG_ARM64_V8R. We want most of the initialization code to be > reusable > > between Armv8-R64 and Armv8-A64. > > > > - We will reuse the original head.s and setup.c of Arm. But replace the > > MMU and page table operations in these files with configuration > operations > > for MPU and MPU regions. > > > > - We provide a boot-time MPU configuration. This MPU configuration will > > support Xen to finish its initialization. And this boot-time MPU > > configuration will record the memory regions that will be parsed from > > device tree. > > > > In the end of Xen initialization, we will use a runtime MPU > configuration > > to replace boot-time MPU configuration. The runtime MPU configuration > will > > merge and reorder memory regions to save more MPU regions for guests. > >  > > > > - Defer system unpausing domain after free_init_memory. > > When Xen initialization is about to end, Xen unpauses guests created > > during initialization. But this will cause some issues. The unpause > > action occurs before free_init_memory, however the runtime MPU > > configuration is built after free_init_memory. In Draft-A, we had > > discussed whether a zeroing operation for init code and data is > > enough or not. Because I had just given a security reason for doing > > free_init_memory on Armv8-R (free_init_memory will drop the Xen init > > code & data, this will reduce the code an attacker can exploit). > > But I forgot other very important reasons: > > 1. Init code and data will occupy two MPU regions, because they > > have different memory attributes. > > 2. It's not easy to zero init code section, because it's readonly. > > We have to update its MPU region to make this section RW. This > > operation doesn't do much less than free_init_memory. > > 3. Zeroing init code and data will not release the two MPU regions > > they are using. This would be a very big waste of a limited MPU > > regions resource. > > 4. Current free_init_memory operation is reusing lots of Armv8-A > > codes, except re-add init memory to Xen heap. Becuase we're using > > static heap on Armv8-R. > > > > So if the unpaused guests start executing the context switch at this > > point, then its MPU context will base on the boot-time MPU > configuration. > > Probably it will be inconsistent with runtime MPU configuration, this > > will cause unexpected problems (This may not happen in a single core > > system, but on SMP systems, this problem is forseeable, so we hope to > > solve it at the beginning). > > > > Why we need to switch the MPU configuration that late? > > Because we need to re-order the MPU regions to reduce complexity of > runtime > > MPU regions management. > > 1. In the boot stage, we allocate MPU regions in sequence until the > max. > > Since a few MPU regions will get removed along the way, they will > leave > > holes there. For example, when heap is ready, fdt will be > reallocated > > in the heap, which means the MPU region for device tree is never > needed. > > And also in free_init_memory, although we do not add init memory to > heap, > > we still reclaim the MPU regions they are using. Without ordering, > we > > may need a bitmap to record such information. > > > > In context switch, the memory layout is quite different for guest > mode > > and hypervisor mode. When switching to guest mode, only guest RAM, > > emulated/passthrough devices, etc could be seen, but in hypervisor > mode, > > all Xen used devices and guests RAM shall be seen. And without > reordering, > > we need to iterate all MPU regions to find according regions to > disable > > during runtime context switch, that's definitely a overhead. > > > > So we propose an ordering at the tail of the boot time, to put all > fixed > > MPU regions in the head, like xen text/data, etc, and put all > flexible > > ones at tail, like device memory, guests RAM. > > > > Then later in runtime, like context switch, we could easily just > disable > > ones from tail and inserts new ones in the tail. > > > > ### **2.3. Changes to reduce memory fragmentation** > > > > In general, memory in Xen system can be classified to 4 classes: > > `image sections`, `heap sections`, `guest RAM`, `boot modules (guest > Kernel, > > initrd and dtb)` > > > > Currently, Xen doesn't have any restriction for users how to allocate > > memory for different classes. That means users can place boot modules > > anywhere, can reserve Xen heap memory anywhere and can allocate guest > > memory anywhere. > > > > In a VMSA system, this would not be too much of a problem, since the > > MMU can manage memory at a granularity of 4KB after all. But in a > > PMSA system, this will be a big problem. On Armv8-R64, the max MPU > > protection regions number has been limited to 256. But in typical > > processor implementations, few processors will design more than 32 > > MPU protection regions. Add in the fact that Xen shares MPU protection > > regions with guest's EL1 Stage 2. It becomes even more important > > to properly plan the use of MPU protection regions. > > > > - An ideal of memory usage layout restriction: > >  > > 1. Reserve proper MPU regions for Xen image (code, rodata and data + > bss). > > 2. Reserve one MPU region for boot modules. > > That means the placement of all boot modules, include guest kernel, > > initrd and dtb, will be limited to this MPU region protected area. > > 3. Reserve one or more MPU regions for Xen heap. > > On Armv8-R64, the guest memory is predefined in device tree, it will > > not be allocated from heap. Unlike Armv8-A64, we will not move all > > free memory to heap. We want Xen heap is dertermistic too, so Xen on > > Armv8-R64 also rely on Xen static heap feature. The memory for Xen > > heap will be defined in tree too. Considering that physical memory > > can also be discontinuous, one or more MPU protection regions needs > > to be reserved for Xen HEAP. > > 4. If we name above used MPU protection regions PART_A, and name left > > MPU protection regions PART_B: > > 4.1. In hypervisor context, Xen will map left RAM and devices to > PART_B. > > This will give Xen the ability to access whole memory. > > 4.2. In guest context, Xen will create EL1 stage 2 mapping in PART_B. > > In this case, Xen just need to update PART_B in context switch, > > but keep PART_A as fixed. > > > > ***Notes: Static allocation will be mandatory on MPU based systems*** > > > > **A sample device tree of memory layout restriction**: > > ``` > > chosen { > > ... > > /* > > * Define a section to place boot modules, > > * all boot modules must be placed in this section. > > */ > > mpu,boot-module-section = <0x10000000 0x10000000>; > > /* > > * Define a section to cover all guest RAM. All guest RAM must be > located > > * within this section. The pros is that, in best case, we can only > have > > * one MPU protection region to map all guest RAM for Xen. > > */ > > mpu,guest-memory-section = <0x20000000 0x30000000>; > > /* > > * Define a memory section that can cover all device memory that > > * will be used in Xen. > > */ > > mpu,device-memory-section = <0x80000000 0x7ffff000>; > > /* Define a section for Xen heap */ > > xen,static-mem = <0x50000000 0x20000000>; > > > > domU1 { > > ... > > #xen,static-mem-address-cells = <0x01>; > > #xen,static-mem-size-cells = <0x01>; > > /* Statically allocated guest memory, within mpu,guest-memory- > section */ > > xen,static-mem = <0x30000000 0x1f000000>; > > > > module@11000000 { > > compatible = "multiboot,kernel\0multiboot,module"; > > /* Boot module address, within mpu,boot-module-section */ > > reg = <0x11000000 0x3000000>; > > ... > > }; > > > > module@10FF0000 { > > compatible = "multiboot,device-tree\0multiboot,module"; > > /* Boot module address, within mpu,boot-module-section > */ > > reg = <0x10ff0000 0x10000>; > > ... > > }; > > }; > > }; > > ``` > > It's little hard for users to compose such a device tree by hand. Based > > on the discussion of Draft-A, Xen community suggested users to use some > > tools like [imagebuilder](https://gitlab.com/ViryaOS/imagebuilder/- > /blob/master/scripts/uboot-script-gen#L390) to generate the above device > tree properties. > > Please goto TODO#3.3 section to get more details of this suggestion. > > Yes, I think we'll need an ImageBuilder script to populate these entries > automatically. With George's help, I moved ImageBuilder to Xen Project. > This is the new repository: https://gitlab.com/xen-project/imagebuilder > > The script to generate mpu,boot-module-section and the other mpu > addresses could be the same ImageBuilder script that generates also > XEN_START_ADDRESS. > > > > ### **2.4. Changes of memory management** > > Xen is coupled with VMSA, in order to port Xen to Armv8-R64, we have to > > decouple Xen from VMSA. And give Xen the ablity to manage memory in PMSA. > > > > 1. ***Use buddy allocator to manage physical pages for PMSA*** > > From the view of physical page, PMSA and VMSA don't have any > difference. > > So we can reuse buddy allocator on Armv8-R64 to manage physical pages. > > The difference is that, in VMSA, Xen will map allocated pages to > virtual > > addresses. But in PMSA, Xen just convert the pages to physical > address. > > > > 2. ***Can not use virtual address for memory management*** > > As Armv8-R64 only has PMSA in EL2, Xen loses the ability of using > virtual > > address to manage memory. This brings some problems, some virtual > address > > based features could not work well on Armv8-R64, like `FIXMAP`, > `vmap/vumap`, > > `ioremap` and `alternative`. > > > > But the functions or macros of these features are used in lots of > common > > code. So it's not good to use `#ifdef CONFIG_ARM_MPU` to gate relate > code > > everywhere. In this case, we propose to use stub helpers to make the > changes > > transparently to common code. > > 1. For `FIXMAP`, we will use `0` in `FIXMAP_ADDR` for all fixmap > operations. > > This will return physical address directly of fixmapped item. > > 2. For `vmap/vumap`, we will use some empty inline stub helpers: > > ``` > > static inline void vm_init_type(...) {} > > static inline void *__vmap(...) > > { > > return NULL; > > } > > static inline void vunmap(const void *va) {} > > static inline void *vmalloc(size_t size) > > { > > return NULL; > > } > > static inline void *vmalloc_xen(size_t size) > > { > > return NULL; > > } > > static inline void vfree(void *va) {} > > ``` > > > > 3. For `ioremap`, it depends on `vmap`. As we have make `vmap` to > always > > return `NULL`, they could not work well on Armv8-R64 without > changes. > > `ioremap` will return input address directly. But if some extended > > functions like `ioremap_nocache`, `ioremap_cache`, need to ask a > new > > memory attributes. As Armv8-R doesn't have infinite MPU regions > for > > Xen to split the memory area from its located MPU region and > assign > > the new attributes to it. So in `ioremap_nocache`, `ioremap_cache`, > > if the input attributes are different from current memory > attributes, > > these functions will return `NULL`. > > ``` > > static inline void *ioremap_attr(...) > > { > > /* We don't have the ability to change input PA cache > attributes */ > > if ( CACHE_ATTR_need_change ) > > return NULL; > > return (void *)pa; > > } > > static inline void __iomem *ioremap_nocache(...) > > { > > return ioremap_attr(start, len, PAGE_HYPERVISOR_NOCACHE); > > } > > static inline void __iomem *ioremap_cache(...) > > { > > return ioremap_attr(start, len, PAGE_HYPERVISOR); > > } > > static inline void __iomem *ioremap_wc(...) > > { > > return ioremap_attr(start, len, PAGE_HYPERVISOR_WC); > > } > > void *ioremap(...) > > { > > return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE); > > } > > > > ``` > > 4. For `alternative`, it has been listed in TODO, we will simply > disable > > it on Armv8-R64 in current stage. But simply disable > `alternative` > > will make `cpus_have_const_cap` always return false. > > ``` > > * System capability check for constant cap */ > > #define cpus_have_const_cap(num) ({ \ > > register_t __ret; \ > > \ > > asm volatile (ALTERNATIVE("mov %0, #0", \ > > "mov %0, #1", \ > > num) \ > > : "=r" (__ret)); \ > > \ > > unlikely(__ret); \ > > }) > > ``` > > So, before we have an PMSA `alternative` implementation, we have > to > > implement a separate `cpus_have_const_cap` for Armv8-R64: > > ``` > > #define cpus_have_const_cap(num) cpus_have_cap(num) > > ``` > > > > ### **2.5. Changes of guest management** > > Armv8-R64 only supports PMSA in EL2, but it supports configurable > > VMSA or PMSA in EL1. This means Xen will have a new type guest on > > Armv8-R64 - MPU based guest. > > > > 1. **Add a new domain type - MPU_DOMAIN** > > When user want to create a guest that will be using MPU in EL1, user > > should add a `mpu` property in device tree `domU` node, like > following > > example: > > ``` > > domU2 { > > compatible = "xen,domain"; > > direct-map; > > mpu; --> Indicates this domain will use PMSA in EL1. > > ... > > }; > > ``` > > Corresponding to `mpu` property in device tree, we also need to > introduce > > a new flag `XEN_DOMCTL_CDF_INTERNAL_mpu` for domain to mark itself > as an > > MPU domain. This flag will be used in domain creation and domain > doing > > vCPU context switch. > > 1. Domain creation need this flag to decide enable PMSA or VMSA in > EL1. > > 2. vCPU context switch need this flag to decide save/restore MMU or > MPU > > related registers. > > > > 2. **Add MPU registers for vCPU to save EL1 MPU context** > > Current Xen only supports MMU based guest, so it hasn't considered to > > save/restore MPU context. In this case, we need to add MPU registers > > to `arch_vcpu`: > > ``` > > struct arch_vcpu > > { > > ... > > #ifdef CONFIG_ARM_MPU > > /* Virtualization Translation Control Register */ > > register_t vtcr_el2; > > > > /* EL1 MPU regions' registers */ > > pr_t *mpu_regions; > > #endif > > ... > > } > > ``` > > Armv8-R64 can support max to 256 MPU regions. But that's just > theoretical. > > So we don't want to embed `pr_t mpu_regions[256]` in `arch_vcpu` > directly, > > this will be a memory waste in most cases. Instead we use a pointer > in > > `arch_vcpu` to link with a dynamically allocated `mpu_regions`: > > ``` > > p->arch.mpu_regions = _xzalloc(sizeof(pr_t) * mpu_regions_count_el1, > SMP_CACHE_BYTES); > > ``` > > As `arch_vcpu` is used very frequently in context switch, so Xen > defines > > `arch_vcpu` as a cache alignment data structure. `mpu_regions` also > will > > be used very frequently in Armv8-R context switch. So we use > `_xzalloc` > > to allocate `SMP_CACHE_BYTES` alignment memory for `mpu_regions`. > > > > `mpu_regions_count_el1` can be detected from `MPUIR_EL1` system > register > > in Xen boot stage. The limitation is that, if we define a static > > `arch_vcpu`, we have to allocate `mpu_regions` before using it. > > > > 3. **MPU based P2M table management** > > Armv8-R64 EL2 doesn't have EL1 stage 2 address translation. But > through > > PMSA, it still has the ability to control the permissions and > attributes > > of EL1 stage 2. In this case, we still hope to keep the interface > > consistent with MMU based P2M as far as possible. > > > > p2m->root will point to an allocated memory. In Armv8-A64, this > memory > > is used to save the EL1 stage 2 translation table. But in Armv8-R64, > > this memory will be used to store EL2 MPU protection regions that are > > used by guest. During domain creation, Xen will prepare the data in > > this memory to make guest can access proper RAM and devices. When the > > guest's vCPU will be scheduled in, this data will be written to MPU > > protection region registers. > > > > ### **2.6. Changes of exception trap** > > As Armv8-R64 has compatible excetpion mode with Armv8-A64, so we can > reuse most > > of Armv8-A64's exception trap & handler code. But except the trap based > on EL1 > > stage 2 translation abort. > > > > In Armv8-A64, we use `FSC_FLT_TRANS` > > ``` > > case FSC_FLT_TRANS: > > ... > > if ( is_data ) > > { > > enum io_state state = try_handle_mmio(regs, hsr, gpa); > > ... > > } > > ``` > > But for Armv8-R64, we have to use `FSC_FLT_PERM` > > ``` > > case FSC_FLT_PERM: > > ... > > if ( is_data ) > > { > > enum io_state state = try_handle_mmio(regs, hsr, gpa); > > ... > > } > > ``` > > > > ### **2.5. Changes of device driver** > > Because Armv8-R64 only has single secure state, this will affect some > > devices that have two secure state, like GIC. But fortunately, most > > vendors will not link a two secure state GIC to Armv8-R64 processors. > > Current GIC driver can work well with single secure state GIC for Armv8- > R64. > > > > ### **2.7. Changes of virtual device** > > Currently, we only support pass-through devices in guest. Becuase event > > channel, xen-bus, xen-storage and other advanced Xen features haven't > been > > enabled in Armv8-R64. > > > > ## 3. TODO > > This section describes some features that are not currently implemented > in > > the PoC. Those features are things that should be looked in a second > stage > > and will not be part of the initial support of MPU/Armv8-R. Those jobs > could > > be done by Arm or any Xen contributors. > > > > ### 3.1. Alternative framework support > > On Armv8-A system, `alternative` is depending on `VMAP` function to > remap > > a code section to a new read/write virtual address. But on Armv8-R, > we do > > not have virtual address to do remap. So as an alternative method, > we will > > disable the MPU to make all RAM `RWX` in "apply alternative all > patches" > > progress temporarily. > > > > 1. Disable MPU -> Code section becomes RWX. > > 2. Apply alternative patches to Xen text. > > 3. Enable MPU -> Code section restores to RX. > > > > All memory is RWX, there may be some security risk. But, because > > "alternative apply patches" happens in Xen init stage, it propoably > > doesn't matter as much. > > > > ### 3.2. Xen Event Channel Support > > In Current RFC patches we haven't enabled the event channel support. > > But I think it's good opportunity to do some discussion in advanced. > > On Armv8-R, all VMs are native direct-map, because there is no > stage2 > > MMU translation. Current event channel implementation depends on > some > > shared pages between Xen and guest: `shared_info` and per-cpu > `vcpu_info`. > > > > For `shared_info`, in current implementation, Xen will allocate a > page > > from heap for `shared_info` to store initial meta data. When guest > is > > trying to setup `shared_info`, it will allocate a free gfn and use a > > hypercall to setup P2M mapping between gfn and `shared_info`. > > > > For direct-mapping VM, this will break the direct-mapping concept. > > And on an MPU based system, like Armv8-R system, this operation will > > be very unfriendly. Xen need to pop `shared_info` page from Xen heap > > and insert it to VM P2M pages. If this page is in the middle of > > Xen heap, this means Xen need to split current heap and use extra > > MPU regions. Also for the P2M part, this page is unlikely to form > > a new continuous memory region with the existing p2m pages, and Xen > > is likely to need another additional MPU region to set it up, which > > is obviously a waste for limited MPU regions. And This kind of > dynamic > > is quite hard to imagine on an MPU system. > > Yeah, it doesn't make any sense for MPU systems > > > > For `vcpu_info`, in current implementation, Xen will store > `vcpu_info` > > meta data for all vCPUs in `shared_info`. When guest is trying to > setup > > `vcpu_info`, it will allocate memory for `vcpu_info` from guest side. > > And then guest will use hypercall to copy meta data from > `shared_info` > > to guest page. After that both Xen `vcpu_info` and guest `vcpu_info` > > are pointed to the same page that allocated by guest. > > > > This implementation has serval benifits: > > 1. There is no waste memory. No extra memory will be allocated from > Xen heap. > > 2. There is no P2M remap. This will not break the direct-mapping, > and > > is MPU system friendly. > > So, on Armv8-R system, we can still keep current implementation for > > per-cpu `vcpu_info`. > > > > So, our proposal is that, can we reuse current implementation idea > of > > `vcpu_info` for `shared_info`? We still allocate one page for > > `d->shared_info` at domain construction for holding some initial > meta-data, > > using alloc_domheap_pages instead of alloc_xenheap_pages and > > share_xen_page_with_guest. And when guest allocates a page for > > `shared_info` and use hypercall to setup it, We copy the initial > data from > > `d->shared_info` to it. And after copy we can update `d- > >shared_info` to point > > to guest allocated 'shared_info' page. In this case, we don't have > to think > > about the fragmentation of Xen heap and p2m and the extra MPU > regions. > > Yes, I think that would work. > > Also I think it should be possible to get rid of the initial > d->shared_info allocation in Xen, given that d->shared_info is for the > benefit of the guest and the guest cannot access it until it makes the > XENMAPSPACE_shared_info hypercall. > While we're working on event channel PoC work on Xen Armv8-R, we found another issue after we dropped d->shared_info allocation in Xen. Both shared_info and vcpu_info are allocated from Guest in runtime. That means the addresses of shared_info and vcpu_info are random. For MMU system, this is OK, because Xen has a full view of system memory in runtime. But for MPU system, the situation becomes a little tricky. We have to setup extra MPU regions for remote domains' shared_info and vcpu_info in event channel hypercall runtime. That's because in current Xen hypercall concept, hypercall will not cause vCPU context switch. When hypercall trap to EL2, it will keep vCPU's P2M view. For MMU system, we have vttbr_el2 for vCPU P2M view and ttbr_el2 for Xen view. So in EL2 Xen has full permissions to access any memory it wants. But for MPU system, we only have one EL2 MPU. Before entering guest, Xen will setup vCPU P2M view in EL2 MPU. In this case, when system entry EL2 through hypercall, the EL2 MPU still keeps current vCPU P2M view and with Xen essential memory (code, data, heap) access permissions. But current EL2 MPU doesn't have the access permissions for EL2 to access other domain's memory. For an event channel hypercall, if we want to update the pending bitmap in remote domain's vcpu_info, it will cause a dataabort in EL2. To solve this dataabort, we may have two methods: 1. Map remote domain's whole memory or pages for shared_info + vcpu_info in EL2 MPU temporarily for hypercall to update pending bits or other accesses. This method doesn't need to do context switch for EL2 MPU, But this method has some disadvantages: 1. We have to reserve MPU regions for hypercall. 2. Different hypercall may have different reservation of MPU regions. 3. We have to handle hypercall one by one for existed and new in future. 2. Switch to Xen's memory view in EL2 MPU when trap from EL1 to EL2. In this case, Xen will have full memory access permissions to update pending bits in EL2. This only changes the EL2 MPU context, does not need to do vCPU context switch. Because the trapped vCPU will be used in the full flow of hypercall. After the hypercall, before returning to EL2, the EL2 MPU will switch to scheduled vCPU' P2M view. This method needs to do EL2 MPU context switch, but: 1. We don't need to reserve MPU regions for Xen's memory view. (Xen's memory view has been setup while initialization) 2. We don't need to handle pages' mapping in hypercall level. 3. Apply to other EL1 to EL2 traps, like dataabort, IRQ, etc. But this method introduces another security concern: Giving full memory view to Xen at any EL2 runtime will cause any security issue? Although Xen has done in this way in MMU system already. > > > But here still has some concerns: > > `d->shared_info` in Xen is accessed without any lock. So it will not > be > > that simple to update `d->shared_info`. It might be possible to > protect > > d->shared_info (or other structure) with a read-write lock. > > > > Do we need to add PGT_xxx flags to make it global and stay as much > the > > same with the original op, a simple investigation tells us that it > only > > be referred in `get_page_type`. Since ARM doesn't care about > typecounts > > and always return 1, it doesn't have too much impact. > > > > ### 3.3. Xen Partial PIC/PIE > > As we have described in `XEN_START_ADDRESS` section. PIC/PIE can > solve > > different platforms have different `XEN_START_ADDRESS` issue. But we > > also describe some issues to use PIC/PIE in real time systems like > > Armv8-R platforms. > > > > But a partial PIC/PIE support may be needed for Armv8-R. Because Arm > > [EBBR](https://arm-software.github.io/ebbr/index.html) require Xen > > on Armv8-R to support EFI boot service. Due to lack of relocation > > capability, EFI loader could not launch xen.efi on Armv8-R. So maybe > > we still need a partially supported PIC/PIE. Only some boot code > > support PIC/PIE to make EFI relocation happy. This boot code will > > help Xen to check its loaded address and relocate Xen image to Xen's > > run-time address if need. > > > > ### 3.4. A tool to generate Armv8-R Xen device tree > > 1. Use a tool to generate above device tree property. > > This tool will have some similar inputs as below: > > --- > > DEVICE_TREE="fvp_baremetal.dtb" > > XEN="4.16-2022.1/xen" > > > > NUM_DOMUS=1 > > DOMU_KERNEL[0]="4.16-2022.1/Image-domU" > > DOMU_RAMDISK[0]="4.16-2022.1/initrd.cpio" > > DOMU_PASSTHROUGH_DTB[0]="4.16-2022.1/passthrough-example-dev.dtb" > > DOMU_RAM_BASE[0]=0x30000000 > > DOMU_RAM_SIZE[0]=0x1f000000 > > --- > > Using above inputs, the tool can generate a device tree similar as > > we have described in sample. > > > > - `mpu,guest-memory-section`: > > This section will cover all guests' RAM (`xen,static-mem` defined > regions > > in all DomU nodes). All guest RAM must be located within this section. > > In the best case, we can only have one MPU protection region to map > all > > guests' RAM for Xen. > > > > If users set `DOMU_RAM_BASE` and `DOMU_RAM_SIZE`, these will be > converted > > to the base and size of `xen,static-mem`. This tool will scan all > > `xen, static-mem` in DomU nodes to determin the base and size of > > `mpu,guest-memory-section`. If there is any other kind of memory > usage > > has been detected in this section, this tool can report an error. > > Except build time check, Xen also need to do runtime check to prevent > a > > bad device tree that generated by malicious tools. > > > > If users set `DOMU_RAM_SIZE` only, this will be converted to the size > of > > `xen,static-mem` only. Xen will allocate the guest memory in runtime, > but > > not from Xen heap. `mpu,guest-memory-section` will be caculated in > runtime > > too. The property in device tree doesn't need or will be ignored by > Xen. > > I am fine with this. You should also know that there was a recent > discussion about adding something like: > > # address size address size ... > DOMU_STATIC_MEM_RANGES[0]="0xe000000 0x1000000 0xa0000000 0x30000000" > > to the ImageBuilder config file. > > > > - `mpu,boot-module-section`: > > This section will be used to store the boot modules like DOMU_KERNEL, > > DOMU_RAMDISK, and DOMU_PASSTHROUGH_DTB. Xen keeps all boot modules in > > this section to meet the requirment of DomU restart on Armv8-R. In > > current stage, we don't have a privilege domain like Dom0 that can > > access filesystem to reload DomU images. > > > > And in current Xen code, the base and size are mandatory for boot > modules > > If users don't specify the base of each boot module, the tool will > > allocte a base for each module. And the tool will generate the > > `mpu,boot-module-section` region, when it finishs boot module memory > > allocation. > > > > Users also can specify the base and size of each boot module, these > will > > be converted to the base and size of module's `reg` directly. The > tool > > will scan all modules `reg` in DomU nodes to generate the base and > size of > > `mpu,boot-module-section`. If there is any kind of other memory usage > > has been detected in this section, this tool can report an error. > > Except build time check, Xen also need to do runtime check to prevent > a > > bad device tree that generated by malicious tools. > > Xen should always check for the validity of its input. However I should > point out that there is no "malicious tool" in this picture because a > malicious entity with access to the tool would also have access to Xen > directly, so they might as well replace the Xen binary. > > > > - `mpu,device-memory-section`: > > This section will cover all device memory that will be used in Xen. > Like > > `UART`, `GIC`, `SMMU` and other devices. We haven't considered > multiple > > `mpu,device-memory-section` scenarios. The devices' memory and RAM > are > > interleaving in physical address space, it would be required to use > > multiple `mpu,device-memory-section` to cover all devices. This > layout > > is common on Armv8-A system, especially in server. But it's rare in > > Armv8-R. So in current stage, we don't want to allow multiple > > `mpu,device-memory-section`. The tool can scan baremetal device tree > > to sort all devices' memory ranges. And calculate a proper region for > > `mpu,device-memory-section`. If it find Xen need multiple > > `mpu,device-memory-section`, it can report an unsupported error. > > > > 2. Use a tool to generate device tree property and platform files > > This opinion still uses the same inputs as opinion#1. But this tool > only > > generates `xen,static-mem` and `module` nodes in DomU nodes, it will > not > > generate `mpu,guest-memory-section`, `mpu,boot-module-section` and > > `mpu,device-memory-section` properties in device tree. This will > > generate following macros: > > `MPU_GUEST_MEMORY_SECTION_BASE`, `MPU_GUEST_MEMORY_SECTION_SIZE` > > `MPU_BOOT_MODULE_SECTION_BASE`, `MPU_BOOT_MODULE_SECTION_SIZE` > > `MPU_DEVICE_MEMORY_SECTION_BASE`, `MPU_DEVICE_MEMORY_SECTION_SIZE` > > in platform files in build time. In runtime, Xen will skip the device > > tree parsing for `mpu,guest-memory-section`, `mpu,boot-module- > section` > > and `mpu,device-memory-section`. And instead Xen will use these > macros > > to do runtime check. > > But, this also means these macros only exist in local build system, > > these macros will not be maintained in Xen repo. > > Yes this makes sense to me. > > I think we should add both scripts to the imagebuilder repository. This > way, they could share code easily, and we can keep the documentation in > a single place.
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |