|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [XEN][PATCH v5 0/8] xen/arm: narrow stage-2 TLB invalidation and batch physmap updates
From: Haseeb Ashraf <haseeb.ashraf@xxxxxxxxxxx> Base commit: ada7fa3491668bbcf1afe525057c470903625d26 Previous version: https://patchew.org/Xen/cover.1765197209.git.haseeb.ashraf@xxxxxxxxxxx/ This series depends on a complementary Linux KVM change [1][2], "KVM: arm64: Resume stage-2 range walks where they stopped". Without it, a DomU cannot be run in practice on Xen as a guest (L1) of KVM. The cost of emulating TLBI VMALLS12E1IS and ALLE1IS is the reason: each trap walks the whole IPA space of the shadow stage-2 in 2^18 separate walks, and on the AmpereOne host used below it took 905.8 ms on average. One create and destroy cycle of an 8GB guest caused 226 of them, about 205 s of emulation (measurements in [1]). With the Linux change a trap takes a few milliseconds (4.06 ms on average in [1], 1.8 ms in Table 2 below). All the results below were taken with it applied on L0, so it is the baseline. This Xen series does not depend on it to be correct. It addresses a separate cost that remains: a trap still takes milliseconds, and many full invalidations are not optimal when a narrower one is enough. [1] https://lore.kernel.org/all/20261118051648.74580-1-haseebashraf091@xxxxxxxxx/ (cover letter) [2] https://lore.kernel.org/all/20261118051648.74580-2-haseebashraf091@xxxxxxxxx/ (patch) Updates of the Arm stage-2 page tables are followed by a TLB invalidation that is much broader than the update. Most paths issue VMALLS12E1IS, which discards all the stage-1 and stage-2 translations of the VMID on every CPU, whatever was changed. Removing guest pages one at a time also costs a hypercall, a p2m lock round trip and an invalidation per page. This series narrows the invalidation to what changed where Xen knows it, drops an invalidation that has no work to do, uses FEAT_TLBIRANGE and FEAT_nTLBPA when the CPU has them, and adds batched interfaces that spread the per-page cost of hypercalls, locking and invalidation. These are general improvements of the Arm p2m code and do not depend on how Xen is run. The effect is largest where TLB maintenance traps, for example when Xen itself runs as a guest (L1) of another hypervisor: the results below show it on the creation and the destruction of an L2 guest. Three common cases are affected: (a) vCPU migration on a pCPU, where Xen flushes the combined stage-1 and stage-2 translations of the guest, although the stage-2 regime is shared by all the vCPUs of the domain. (b) teardown of DomU mappings exported into Dom0, where XENMEM_remove_from_physmap flushes all the TLBs of the guest, although the affected IPA range is known. (c) device models that map and unmap many guest pages: removing pages individually takes a hypercall, a p2m lock round trip and a TLB invalidation per page, and XENMEM_add_to_physmap_batch copies three guest values per entry. Patches: # Change Condition and bound -- --------------------------------- -------------------------------- 1 Invalidate by IPA when the range Fewer than 512 pages (4KB is known; local flush on a vCPU granule), else the whole VMID switch on the same pCPU (stage-1 only on Arm64) 2 Skip the stage-1 invalidation FEAT_nTLBPA only that follows an IPA invalidation 3 IPA-based invalidation on Arm32 FEAT_nTLBPA and TLBI by IPA, same 512 page bound 4 Range invalidation (RIPAS2E1IS) FEAT_TLBIRANGE, at most four range operations 5 Drop the flush when resolving a None translation fault 6 Arm helper: remove a list of Arm with an MMU pages under one p2m lock, one (HAS_PHYSMAP_REMOVE_BATCH) range flush per run of GFNs 7 XENMEM_remove_from_physmap_batch Opt-in for callers, Arm with an hypercall using patch 6 MMU, -ENOSYS elsewhere 8 Copy add_to_physmap_batch Common code, 128 entries per entries in chunks chunk Results: Xen as L1, creation and destruction of an L2 guest Setup: an AmpereOne host runs Linux (L0) with KVM nested virtualization and the Linux change above, in the baseline and in v5. Xen 4.23-dev and its Dom0 run as an L1 guest, and a DomU is the L2 guest (8GiB of RAM, 4 vCPUs). Five back-to-back cycles per run: create the DomU, boot it to a login prompt, then "xl destroy". "Baseline" is Xen master and "v5" is master with this series. A bpftrace script on L0 times the KVM handlers (handle_vmalls12e1is, handle_ripas2e1is and handle_tlbi_el1) of the TLBI instructions that L1 traps on, over all five cycles. Table 1: L2 guest lifecycle, wall clock and system time Cycle VM creation xl destroy (wall) xl destroy (sys) base v5 base v5 base v5 ------- --------- -------- ----------- ----------- ---------- -------- 1 27 s 9 s 106.411 s 2.693 s 87.877 s 0.627 s 2 20 s 3 s 107.549 s 2.690 s 88.966 s 0.635 s 3 23 s 2 s 114.707 s 2.703 s 95.182 s 0.654 s 4 27 s 2 s 107.946 s 2.702 s 89.189 s 0.651 s 5 20 s 3 s 92.778 s 2.697 s 76.813 s 0.645 s ------- --------- -------- ----------- ----------- ---------- -------- mean 23.4 s 3.8 s 105.878 s 2.697 s 87.605 s 0.642 s median 23 s 3 s 107.549 s 2.697 s 88.966 s 0.645 s change 6.2x faster 39x faster 136x less Table 2: TLBI instructions emulated by L0 KVM (five cycles) Run Instruction Entries Cumulative Mean Max --------- ------------- --------- ------------ ---------- ---------- Baseline VMALLS12E1IS 448,959 * 824.771 s 1.843 ms 39.429 ms v5 VMALLS12E1IS 15 26.863 ms 1.791 ms 3.787 ms v5 RIPAS2E1IS 24,854 525.967 ms 21.16 us 30.475 ms v5 VMALLE1IS 24,812 66.215 ms 2.67 us 0.019 ms v5 all 49,681 619.045 ms * 447,539 of these calls returned in the trace. The cumulative time and the mean are over the returned calls. Traps: 9.0x fewer (448,959 against 49,681). Cumulative handler time: about 1,330x less (824.8 s against 0.619 s). Table 3: number of handler calls by duration (five cycles) Duration Baseline v5 v5 v5 VMALLS12E1IS RIPAS2E1IS VMALLE1IS VMALLS12E1IS ----------------- ------------- ------------- ----------- ------------ below 4 us 0 871 22,831 0 4 us to 16 us 3 18,982 1,967 1 16 us to 64 us 1 4,981 14 1 64 us to 1 ms 149 0 0 3 1 ms to 2 ms 247,974 0 0 6 2 ms to 4 ms 197,756 9 0 4 4 ms and more 1,656 11 0 0 How to read it: the baseline needs about 90,000 emulated VMALLS12E1IS per cycle, each about 1.8 ms, and this is what the 106 s of "xl destroy" and most of the creation time are spent on. With the series the same work needs about 5,000 RIPAS2E1IS and 5,000 VMALLE1IS per cycle. A RIPAS2E1IS takes about 21 us and is followed by a VMALLE1IS of about 2.7 us, because the CPU has no FEAT_nTLBPA: the pair costs about 24 us, against 1.8 ms for the full flush. "xl destroy" takes the same 2.7 s in every cycle of the v5 run, while the baseline takes between 93 s and 115 s. The 15 remaining VMALLS12E1IS (three per cycle) are for ranges that are too large for an IPA invalidation. They still take about 1.8 ms each, and are 4% of the cumulative handler time of v5. The lifecycle runs do not exercise the batch interfaces in patches 6 to 8. The tables therefore provide no performance result for those patches; dedicated device-model workloads using the batch interfaces are needed to quantify their benefit. Impact when Xen is not nested: - The series changes common Arm p2m invalidations. No bare metal performance measurement is available yet. - Without FEAT_TLBIRANGE, Arm64 ranges under 512 pages can issue up to 511 broadcast IPA TLBIs, and ranges of 512 pages or more use the whole VMID. Arm32 uses the same bound only with FEAT_nTLBPA and TLBI by IPA, and the whole VMID otherwise. With FEAT_TLBIRANGE, ranges that can be encoded use at most four range operations, and larger ranges use the whole VMID. This bounds the number of instructions, not the latency. - The flushes that are deferred to p2m_write_unlock() are unchanged. The FEAT_nTLBPA and FEAT_TLBIRANGE paths depend on the capability. - The batched removal hypercall is opt-in for callers and exists for Arm with an MMU. Patch 8 changes common code, so its effect on bare metal also needs a measurement. Testing: the runs above were done on Arm64 (AmpereOne) with this series. An x86_64 build completed with CONFIG_COMPAT=y. The Arm64 Cppcheck MISRA add-on reported no diagnostics on changed lines. Not yet validated: Xen on bare metal, the FEAT_nTLBPA path, Arm32 at run time. Notes: - Like XENMEM_add_to_physmap_batch, the batched removal reports per-entry errors in errs[] and is not transactional: -EFAULT can follow removals that completed. - Patch 8 keeps that semantics, but reads and writes the entries in chunks of 128, so more entries may be processed before an -EFAULT. Changes in v5: - Replace the 256MB limit, measured on one platform, with TLBI_MAX_PAGE_OPS, the limit of Linux, and use the Linux limit for range operations (patches 1 and 4). - Keep the deferred flush of removals: p2m_remove_mapping() flushes its range once, other removals are flushed by p2m_write_unlock() as before. Keep the deferred flush of permission-only changes with memaccess (patch 1). - Keep patch 1 independently usable with the MPU stub for p2m_force_tlb_flush_range_sync(); defer the locked-helper refactor until patch 6, its first user. - Move Arm32 nTLBPA and UniTLB capability detection from patch 2 to patch 3, where the Arm32 IPA-based invalidation first uses it. - Invalidate the whole old block when a superpage is split (patch 1). - Do not skip the stage-1 invalidation of the vCPU switch with FEAT_nTLBPA: it separates the vCPUs, it is not about walk caches (patch 2). - Complete the stage-2 invalidation before the stage-1 one, as the Arm ARM requires, and complete the broadcast sequence before the repeat-TLBI workaround (patch 1). - Fix the Arm64 helper macros of patch 1 so that it builds on its own. - Put the TLBI instructions in one-instruction helpers (MISRA C Directive 4.3) and drop the unused inner-shareable stage-1 helper. - Add patches 4 to 8. Cite DDI 0487 M.d. Changes in v4: - Rebased on master branch - Reworked the Arm64 IPA-based TLBI sequence to make completion of the broadcast TLBI explicit instead of relying on a stage-1 helper for the final barriers/workaround. - Reused __tlb_repeat_sync() for repeat-TLBI workaround handling in the Arm64 completion path. - Fixed the Arm32 IPA-based TLBI helper to fall back to a full guest TLB flush when the required capability combination is unavailable, instead of becoming a no-op. - Added a small robustness check in p2m_force_tlb_flush_range_sync(). - Refined comments and commit messages to make the architectural intent clearer. Changes in v3: - Mainly reworked handling of the repeat-TLBI workaround with IPA-based TLBI so that the extra TLBI and DSB are applied only at completion of the overall sequence. - Updated code comments as per feedback. Further details are available in each commit's changelog. - Minor code updates as per feedback. Further details are available in each commit's changelog. Changes in v2: - Split the original change into three commits. The first commit implements the baseline implementation without adding new CPU capabilities. CPU capability handling is introduced in follow-up commits to make each optimization clearer. - Moved Arm32 and Arm64 specific TLBI implementations to the respective architecture-specific flushtlb.h. - Added ARM ARM references in code comments. - Added a threshold to select between IPA-based invalidation and fallback to full guest invalidation above that threshold. - Introduced ARM_HAS_NTLBPA CPU capability for both Arm32 and Arm64. - Introduced ARM32_HAS_TLB_IPA CPU capability for Arm32 IPA-based TLBI. Haseeb Ashraf (8): xen/arm/p2m: perform IPA-based TLBI when IPA is known xen/arm: skip the stage-1 TLBI after IPA flush with FEAT_nTLBPA xen/arm32: use IPA-based TLBI when supported with FEAT_nTLBPA xen/arm64: use range TLBI for IPA flush with FEAT_TLBIRANGE xen/arm: do not flush the TLBs when resolving a translation fault xen/arm: batch stage-2 removal with range flushes xen/memory: add XENMEM_remove_from_physmap_batch xen/memory: copy add_to_physmap batch entries in bulk CHANGELOG.md | 5 + xen/arch/arm/Kconfig | 1 + xen/arch/arm/cpufeature.c | 45 +++++ xen/arch/arm/include/asm/arm32/flushtlb.h | 67 +++++++ xen/arch/arm/include/asm/arm64/flushtlb.h | 178 ++++++++++++++++++ xen/arch/arm/include/asm/cpregs.h | 1 + xen/arch/arm/include/asm/cpufeature.h | 26 ++- xen/arch/arm/include/asm/flushtlb.h | 12 ++ xen/arch/arm/include/asm/mmu/p2m.h | 2 + xen/arch/arm/include/asm/mpu/p2m.h | 4 + xen/arch/arm/include/asm/processor.h | 9 + xen/arch/arm/mmu/p2m.c | 114 ++++++++++-- xen/arch/arm/p2m.c | 111 +++++++++-- xen/common/Kconfig | 3 + xen/common/memory.c | 215 ++++++++++++++++++++-- xen/include/public/memory.h | 35 ++++ xen/include/xen/p2m-common.h | 16 ++ 17 files changed, 799 insertions(+), 45 deletions(-)
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |