[Xen-devel] [PATCH v2 0/5] mitigate the per-pCPU blocking list may be too long

[Chao Gao <chao.gao@xxxxxxxxx>]

VT-d PI introduces a per-pCPU blocking list to track the blocked vCPU
running on the pCPU. Theoretically, there are 32K domain on single
host, 128 vCPUs per domain. If all vCPUs are blocked on the same pCPU,
4M vCPUs are in the same list. Traversing this list consumes too
much time. More discussion can be found in [1,2,3].

To mitigate this issue, this series implements the following two methods:
1. put vcpus to another pcpu's list when the local pcpu's list length
reachs an upper bound. The upper bound is determined by total vcpu
count and total pcpu count in the system.
2. Don't put the blocked vCPUs which won't be woken by the wakeup
interrupt into the list.

PATCH 1/5 tracks the event, adding entry to PI blocking list. With the
patch, some data can be acquired to help to validate the following
patches. 

PATCH 2/5 uses a global variable to track how many hvm vcpus on this
system. It is used to calculate the number limit of blocked vcpu on a
given pcpu.

In patch 3/5, a policy is used to restrict the vcpu count on a given
pcpu's pi blocking list in case the list grows too long.

Patch 4/5 adds a refcount to vcpu's pi_desc. If the pi_desc is
recorded in one IRTE, the refcount increases by 1 and If the pi_desc is
cleared in one IRTE, the refcount decreases by 1.

In Patch 5/5, one vCPU is added to PI blocking list only if its
pi_desc is referred by at least one IRTE.

[1] 
https://lists.gt.net/xen/devel/422661?search_string=VT-d%20posted-interrupt%20core%20logic%20handling;#422661
[2] 
https://lists.gt.net/xen/devel/422567?search_string=%20The%20length%20of%20the%20list%20depends;#422567
[3] 
https://lists.gt.net/xen/devel/472749?search_string=enable%20vt-d%20pi%20by%20default;#472749

Chao Gao (5):
  xentrace: add TRC_HVM_PI_LIST_ADD
  vcpu: track hvm vcpu number on the system
  VT-d PI: restrict the vcpu number on a given pcpu
  VT-d PI: Adding reference count to pi_desc
  VT-d PI: Don't add vCPU to PI blocking list for a case

 tools/xentrace/formats                 |   1 +
 xen/arch/x86/hvm/vmx/vmx.c             | 133 ++++++++++++++++++++++++++++++---
 xen/common/domain.c                    |   8 ++
 xen/drivers/passthrough/io.c           |   2 +-
 xen/drivers/passthrough/vtd/intremap.c |  59 ++++++++++++++-
 xen/include/asm-x86/hvm/domain.h       |   6 ++
 xen/include/asm-x86/hvm/trace.h        |   1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h     |   3 +
 xen/include/asm-x86/iommu.h            |   2 +-
 xen/include/asm-x86/msi.h              |   2 +-
 xen/include/public/trace.h             |   1 +
 xen/include/xen/sched.h                |   2 +
 12 files changed, 203 insertions(+), 17 deletions(-)

-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel