[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[XEN][PATCH v5 1/8] xen/arm/p2m: perform IPA-based TLBI when IPA is known



From: Haseeb Ashraf <haseeb.ashraf@xxxxxxxxxxx>

VMALLS12E1IS invalidates all stage-1 and stage-2 translations for the
current VMID across the inner-shareable domain, regardless of which
mapping changed. This can be particularly costly when every TLBI is
emulated by an outer hypervisor.

Narrow the invalidation where Xen knows what changed:

- When a pCPU switches between vCPUs of a domain, use a local flush to
drop the prior vCPU's stage-1 translations. Arm64 uses a stage-1-only
TLBI; on Arm32 the available local guest-TLB instruction can also
invalidate stage-2 entries, depending on execution context.
- When p2m_remove_mapping() removes a range, invalidate that IPA range
instead of the whole VMID.
- When a valid entry is replaced, invalidate its IPA range. When a
superpage is split, invalidate the whole old block at its aligned base.

The IPA sequence issues one broadcast TLBI per page and is limited to
fewer than TLBI_MAX_PAGE_OPS pages (512 with 4KB pages), following
Linux's MAX_DVM_OPS limit. A larger range keeps the whole-VMID
invalidation. The limit bounds the number of broadcast operations; it is
not a performance crossover.

Existing deferred flushes remain unchanged: teardown of all mappings of
a dying domain, permission-only memaccess changes, and removals by
callers that do not know the range are still flushed by
p2m_write_unlock().

Complete stage-2 invalidation before stage-1 invalidation, and complete
the broadcast sequence before the repeat-TLBI workaround.

On Arm32, the range helper keeps the full flush; patch 3 adds the
capability-gated IPA implementation.

Linux references:
- arm64: KVM: Implement TLB handling, for the IPA TLBI followed by a
  stage-1 invalidation:
  
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=5eec0a91e32a2862e86265532ae773820e0afd77
- arm64: tlbi: Set MAX_TLBI_OPS to PTRS_PER_PTE, for the per-flush
  operation limit:
  
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=3d65b6bbc01ecece8142e62a8a5f1d48ba41a240
- The later rename of that limit to MAX_DVM_OPS:
  
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ec1c3b9ff16082f880b304be40992568f4eee6a7

Suggested-by: Julien Grall <julien@xxxxxxx>
Signed-off-by: Haseeb Ashraf <haseeb.ashraf@xxxxxxxxxxx>
---
Changes in v5:
- Replace the 256MB limit with TLBI_MAX_PAGE_OPS, following Linux.
- Flush the known range in p2m_remove_mapping(); other removals and
  permission-only memaccess changes remain deferred.
- Invalidate the whole old block when a superpage is split.
- Complete stage-2 invalidation before stage-1 invalidation, and complete
  the first DSB before the repeat-TLBI workaround.
- Fix the Arm64 helper macros so this patch builds on its own; they did
  not build on their own in v4.
- Keep TLBI instructions in one-instruction helpers and add the MPU stub
  for p2m_force_tlb_flush_range_sync(). Cite DDI 0487 M.d.

 xen/arch/arm/include/asm/arm32/flushtlb.h | 33 ++++++++
 xen/arch/arm/include/asm/arm64/flushtlb.h | 78 ++++++++++++++++++
 xen/arch/arm/include/asm/flushtlb.h       | 12 +++
 xen/arch/arm/include/asm/mmu/p2m.h        |  2 +
 xen/arch/arm/include/asm/mpu/p2m.h        |  4 +
 xen/arch/arm/mmu/p2m.c                    | 99 +++++++++++++++++++----
 xen/arch/arm/p2m.c                        | 16 +++-
 7 files changed, 226 insertions(+), 18 deletions(-)

diff --git a/xen/arch/arm/include/asm/arm32/flushtlb.h 
b/xen/arch/arm/include/asm/arm32/flushtlb.h
index 5483be08fb..374b8bfe54 100644
--- a/xen/arch/arm/include/asm/arm32/flushtlb.h
+++ b/xen/arch/arm/include/asm/arm32/flushtlb.h
@@ -45,6 +45,23 @@ TLB_HELPER(flush_xen_tlb_local, TLBIALLH, nsh)
 
 #undef TLB_HELPER
 
+/*
+ * Flush TLB of local processor. Use when flush for only stage-1 is intended.
+ *
+ * The following function should be used where intention is to clear only
+ * stage-1 TLBs.
+ */
+static inline void flush_guest_tlb_s1_local(void)
+{
+    /*
+     * Same instruction can invalidate both stage-1 and stage-2 TLBs depending
+     * upon the execution context.
+     *
+     * See Arm ARM (DDI 0487 M.d), G5.9.5, Table G5-23.
+     */
+    flush_guest_tlb_local();
+}
+
 /* Flush TLB of local processor for address va. */
 static inline void __flush_xen_tlb_one_local(vaddr_t va)
 {
@@ -60,6 +77,22 @@ static inline void __flush_xen_tlb_one(vaddr_t va)
 /* Only for ARM64_WORKAROUND_REPEAT_TLBI */
 static inline void __tlb_repeat_sync(void) {}
 
+/*
+ * Flush a range of IPA's mappings from the TLB of all processors in the
+ * inner-shareable domain.
+ */
+static inline void flush_guest_tlb_range_ipa(paddr_t ipa,
+                                             unsigned long size)
+{
+    /*
+     * Following can invalidate both stage-1 and stage-2 TLBs depending upon
+     * the execution mode.
+     *
+     * See Arm ARM (DDI 0487 M.d), G5.9.5, Table G5-23.
+     */
+    flush_guest_tlb();
+}
+
 #endif /* __ASM_ARM_ARM32_FLUSHTLB_H__ */
 /*
  * Local variables:
diff --git a/xen/arch/arm/include/asm/arm64/flushtlb.h 
b/xen/arch/arm/include/asm/arm64/flushtlb.h
index 1606b26bf2..b3bce7baf2 100644
--- a/xen/arch/arm/include/asm/arm64/flushtlb.h
+++ b/xen/arch/arm/include/asm/arm64/flushtlb.h
@@ -67,6 +67,9 @@ TLB_HELPER_LOCAL(flush_guest_tlb_local, vmalls12e1)
 /* Flush innershareable TLBs, current VMID only */
 TLB_HELPER(flush_guest_tlb, vmalls12e1is)
 
+/* Flush local TLBs, current VMID, stage-1 only */
+TLB_HELPER_LOCAL(flush_guest_tlb_s1_local, vmalle1)
+
 /* Flush local TLBs, all VMIDs, non-hypervisor mode */
 TLB_HELPER_LOCAL(flush_all_guests_tlb_local, alle1)
 
@@ -116,6 +119,81 @@ static inline void __tlb_repeat_sync(void)
         : : : "memory");
 }
 
+/*
+ * Complete a broadcast TLBI sequence.
+ *
+ * The first DSB completes the TLBIs issued so far. For all relevant errata
+ * it is then only necessary to execute a single additional TLBI;DSB sequence,
+ * which __tlb_repeat_sync() does when the workaround is enabled.
+ */
+static inline void tlbi_complete_broadcast(void)
+{
+    dsb(ish);
+    __tlb_repeat_sync();
+    isb();
+}
+
+/*
+ * TLBI IPAS2E1IS: invalidate the stage-2 and combined entries of one IPA of
+ * the current VMID, in the inner-shareable domain. The caller issues the
+ * barriers.
+ */
+static inline void tlbi_ipas2e1is(paddr_t ipa)
+{
+    asm_inline volatile(
+        "tlbi ipas2e1is, %0"
+        : : "r" (ipa >> PAGE_SHIFT) : "memory");
+}
+
+/* TLBI VMALLE1IS: invalidate the stage-1 entries of the current VMID. */
+static inline void tlbi_vmalle1is(void)
+{
+    asm_inline volatile("tlbi vmalle1is" : : : "memory");
+}
+
+/*
+ * Flush a range of IPA's mappings from the TLB of all processors in the
+ * inner-shareable domain.
+ *
+ * A range of TLBI_MAX_PAGE_OPS pages or more is flushed with the whole
+ * VMID. Each page of the range costs a broadcast TLBI, and the limit bounds
+ * their number like MAX_DVM_OPS does in Linux. It is not a tuned value.
+ */
+static inline void flush_guest_tlb_range_ipa(paddr_t ipa, unsigned long size)
+{
+    unsigned long pages = size >> PAGE_SHIFT;
+    paddr_t end = ipa + size;
+
+    if ( pages >= TLBI_MAX_PAGE_OPS )
+    {
+        flush_guest_tlb();
+        return;
+    }
+
+    /*
+     * See Arm ARM (DDI 0487 M.d), D8.17.6.1 (Invalidating TLB entries from
+     * stage 2 translations).
+     */
+    dsb(ishst); /* Ensure prior page-table updates have completed */
+    for ( ; ipa < end; ipa += PAGE_SIZE )
+    {
+        /* Flush stage-2 TLBs for IPA address */
+        tlbi_ipas2e1is(ipa);
+    }
+
+    /*
+     * The stage-1 entries and walks that went through the removed stage-2
+     * entries have to be invalidated too. The stage-2 invalidation has to be
+     * complete first, or a walk on another CPU could still refill the TLBs
+     * with stage-1 and the old stage-2 translation. Linux does the same (see
+     * __kvm_tlb_flush_vmid_ipa()).
+     */
+    dsb(ish);
+    tlbi_vmalle1is();
+
+    tlbi_complete_broadcast();
+}
+
 #endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */
 /*
  * Local variables:
diff --git a/xen/arch/arm/include/asm/flushtlb.h 
b/xen/arch/arm/include/asm/flushtlb.h
index c292c3c00d..2711b76611 100644
--- a/xen/arch/arm/include/asm/flushtlb.h
+++ b/xen/arch/arm/include/asm/flushtlb.h
@@ -17,6 +17,18 @@ static inline void page_set_tlbflush_timestamp(struct 
page_info *page)
     page->tlbflush_timestamp = tlbflush_current_time();
 }
 
+/*
+ * Upper bound on the number of broadcast TLBI operations issued, one per
+ * page, to invalidate a range. A range of this many pages or more is
+ * invalidated with the whole VMID instead.
+ *
+ * This is the number of entries of a page table, which is the limit Linux
+ * uses for the same purpose (MAX_DVM_OPS in arch/arm64/include/asm/
+ * tlbflush.h). It bounds the burst of broadcast operations, and the time
+ * spent with the p2m lock held. It is not a tuned value.
+ */
+#define TLBI_MAX_PAGE_OPS       (1UL << (PAGE_SHIFT - 3))
+
 #if defined(CONFIG_ARM_32)
 # include <asm/arm32/flushtlb.h>
 #elif defined(CONFIG_ARM_64)
diff --git a/xen/arch/arm/include/asm/mmu/p2m.h 
b/xen/arch/arm/include/asm/mmu/p2m.h
index 58496c0b09..8a16722b82 100644
--- a/xen/arch/arm/include/asm/mmu/p2m.h
+++ b/xen/arch/arm/include/asm/mmu/p2m.h
@@ -10,6 +10,8 @@ extern unsigned int p2m_root_level;
 
 struct p2m_domain;
 void p2m_force_tlb_flush_sync(struct p2m_domain *p2m);
+void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start_ipa,
+                                    uint64_t page_count);
 void p2m_tlb_flush_sync(struct p2m_domain *p2m);
 
 void p2m_clear_root_pages(struct p2m_domain *p2m);
diff --git a/xen/arch/arm/include/asm/mpu/p2m.h 
b/xen/arch/arm/include/asm/mpu/p2m.h
index 1484c75b55..b799738314 100644
--- a/xen/arch/arm/include/asm/mpu/p2m.h
+++ b/xen/arch/arm/include/asm/mpu/p2m.h
@@ -17,6 +17,10 @@ static inline void p2m_clear_root_pages(struct p2m_domain 
*p2m) {}
 
 static inline void p2m_tlb_flush_sync(struct p2m_domain *p2m) {}
 
+static inline void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m,
+                                                  uint64_t start_ipa,
+                                                  uint64_t page_count) {}
+
 #endif /* __ARM_MPU_P2M_H__ */
 
 /*
diff --git a/xen/arch/arm/mmu/p2m.c b/xen/arch/arm/mmu/p2m.c
index 53d28935af..b93ead6c46 100644
--- a/xen/arch/arm/mmu/p2m.c
+++ b/xen/arch/arm/mmu/p2m.c
@@ -233,37 +233,33 @@ void p2m_restore_state(struct vcpu *n)
     isb();
 
     /*
-     * Flush local TLB for the domain to prevent wrong TLB translation
-     * when running multiple vCPU of the same domain on a single pCPU.
+     * All vCPUs of the domain share the same p2m and the same effective
+     * stage-2 translation regime. The stage-1 translations of the vCPU that
+     * ran last on this pCPU must not be used by this one.
      */
     if ( *last_vcpu_ran != INVALID_VCPU_ID && *last_vcpu_ran != n->vcpu_id )
-        flush_guest_tlb_local();
+        flush_guest_tlb_s1_local();
 
     *last_vcpu_ran = n->vcpu_id;
 }
 
 /*
- * Force a synchronous P2M TLB flush.
+ * Loads VTTBR from given P2M.
  *
  * Must be called with the p2m lock held.
+ *
+ * This returns switched out VTTBR.
  */
-void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
+static uint64_t p2m_load_vttbr(struct p2m_domain *p2m, unsigned long *flags)
 {
-    unsigned long flags = 0;
     uint64_t ovttbr;
 
-    ASSERT(p2m_is_write_locked(p2m));
-
-    /*
-     * ARM only provides an instruction to flush TLBs for the current
-     * VMID. So switch to the VTTBR of a given P2M if different.
-     */
     ovttbr = READ_SYSREG64(VTTBR_EL2);
     if ( ovttbr != p2m->vttbr )
     {
         uint64_t vttbr;
 
-        local_irq_save(flags);
+        local_irq_save(*flags);
 
         /*
          * ARM64_WORKAROUND_AT_SPECULATE: We need to stop AT to allocate
@@ -282,8 +278,14 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
         isb();
     }
 
-    flush_guest_tlb();
+    return ovttbr;
+}
 
+/*
+ * Restores VTTBR which was switched out as a result of p2m_load_vttbr().
+ */
+static void p2m_restore_vttbr(uint64_t ovttbr, unsigned long flags)
+{
     if ( ovttbr != READ_SYSREG64(VTTBR_EL2) )
     {
         WRITE_SYSREG64(ovttbr, VTTBR_EL2);
@@ -291,10 +293,64 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
         isb();
         local_irq_restore(flags);
     }
+}
+
+/*
+ * Force a synchronous P2M TLB flush.
+ *
+ * Must be called with the p2m lock held.
+ */
+void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
+{
+    unsigned long flags = 0;
+    uint64_t ovttbr;
+
+    ASSERT(p2m_is_write_locked(p2m));
+
+    /*
+     * ARM only provides an instruction to flush TLBs for the current
+     * VMID. So switch to the VTTBR of a given P2M if different.
+     */
+    ovttbr = p2m_load_vttbr(p2m, &flags);
+
+    flush_guest_tlb();
+
+    p2m_restore_vttbr(ovttbr, flags);
 
     p2m->need_flush = false;
 }
 
+/*
+ * Force a synchronous P2M TLB flush on a range of addresses.
+ *
+ * Must be called with the p2m lock held.
+ */
+void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start_ipa,
+                                    uint64_t page_count)
+{
+    unsigned long flags = 0;
+    uint64_t ovttbr;
+
+    ASSERT(p2m_is_write_locked(p2m));
+
+    if ( !page_count )
+        return;
+
+    /*
+     * ARM only provides an instruction to flush TLBs for the current
+     * VMID. So switch to the VTTBR of a given P2M if different.
+     */
+    ovttbr = p2m_load_vttbr(p2m, &flags);
+
+    /* Invalidate TLB entries by IPA range */
+    if ( unlikely(page_count > ULONG_MAX / PAGE_SIZE) )
+        flush_guest_tlb();
+    else
+        flush_guest_tlb_range_ipa(start_ipa, page_count * PAGE_SIZE);
+
+    p2m_restore_vttbr(ovttbr, flags);
+}
+
 void p2m_tlb_flush_sync(struct p2m_domain *p2m)
 {
     if ( p2m->need_flush )
@@ -1013,6 +1069,7 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
     {
         /* We need to split the original page. */
         lpae_t split_pte = *entry;
+        unsigned long nr = 1UL << XEN_PT_LEVEL_ORDER(level);
 
         ASSERT(p2m_is_superpage(*entry, level));
 
@@ -1036,7 +1093,9 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
          * For more details see (D4.7.1 in ARM DDI 0487A.j).
          */
         p2m_remove_pte(entry, p2m->clean_pte);
-        p2m_force_tlb_flush_sync(p2m);
+        p2m_force_tlb_flush_range_sync(p2m,
+                                       pfn_to_paddr(gfn_x(sgfn) & ~(nr - 1)),
+                                       nr);
 
         p2m_write_pte(entry, split_pte, p2m->clean_pte);
 
@@ -1092,7 +1151,12 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
         p2m_remove_pte(entry, p2m->clean_pte);
 
     if ( removing_mapping )
-        /* Flush can be deferred if the entry is removed */
+        /*
+         * Flush can be deferred if the entry is removed. A caller that knows
+         * the range it removes flushes it by IPA when it is done, see
+         * p2m_remove_mapping(). Otherwise p2m_write_unlock() flushes all
+         * the TLBs of the domain, once for all the entries removed.
+         */
         p2m->need_flush |= !!lpae_is_valid(orig_pte);
     else
     {
@@ -1112,7 +1176,8 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
         {
             if ( likely(!p2m->mem_access_enabled) ||
                  P2M_CLEAR_PERM(pte) != P2M_CLEAR_PERM(orig_pte) )
-                p2m_force_tlb_flush_sync(p2m);
+                p2m_force_tlb_flush_range_sync(p2m, gfn_to_gaddr(sgfn),
+                                               1UL << page_order);
             else
                 p2m->need_flush = true;
         }
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index fb03978a19..a39351ede9 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -133,6 +133,7 @@ static inline int p2m_remove_mapping(struct domain *d,
     int rc;
 
     p2m_write_lock(p2m);
+
     /*
      * Before removing the GFN - MFN mapping for any RAM pages make sure
      * that there is no difference between what is already mapped and what
@@ -161,7 +162,20 @@ static inline int p2m_remove_mapping(struct domain *d,
     rc = p2m_set_entry(p2m, start_gfn, nr, INVALID_MFN,
                        p2m_invalid, p2m_access_rwx);
 
-out:
+    /*
+     * The range that may have changed is known. Invalidate it instead of
+     * all the TLBs of the domain, which p2m_write_unlock() would do. The
+     * whole domain is still flushed when the range is too large for an
+     * invalidation by IPA. A failure after some entries were removed is
+     * covered by the same range.
+     */
+    if ( p2m->need_flush )
+    {
+        p2m_force_tlb_flush_range_sync(p2m, gfn_to_gaddr(start_gfn), nr);
+        p2m->need_flush = false;
+    }
+
+ out:
     p2m_write_unlock(p2m);
 
     return rc;



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.