[Xen-devel] [PATCH v4 0/4] mitigate the per-pCPU blocking list may be too long

[Chao Gao <chao.gao@xxxxxxxxx>]

Changes in v4:
 - In Patch 3, use a new lock to avoid adding a blocked vcpu to a
 offline pcpu's blocking list.
 - divide Patch 1 in v3 to trace part (Patch 4) and non-trace part
 (Patch 1)
 - move the place we increase/decrease the hvm vcpu number to
 hvm_vcpu_{initialise, destory}

VT-d PI introduces a per-pCPU blocking list to track the blocked vCPU
on a given pCPU. Theoretically, there are 32K domain on single host,
128 vCPUs per domain. If all vCPUs are blocked on the same pCPU,
4M vCPUs are in the same list. Traversing this list consumes too
much time. More discussion can be found in [1,2,3].

To mitigate this issue, this series put vcpus to another pcpu's list
when the local pcpu's list length reachs an upper bound which is the
average vcpus per pcpu ratio plus a constant. 

PATCH 1/4 adds a counter to track the per-pCPU blocking list's length
to compare with the upper bound every time adding a entry to the list.

PATCH 2/4 uses a global variable to track how many hvm vcpus on this
system. It is used to calculate the number limit of blocked vcpu on a
given pcpu, namely the upper bound.

patch 3/4 employs a policy to restrict the vcpu count on a given
pcpu's pi blocking list in case the list grows too long. In one work,
If list length is smaller than the upper bound, the vcpu is added to
the pi blocking list of the pcpu which it is running on. Otherwise,
another online pcpu is chosen to accept the vcpu.

patch 4/4 adds some relevant events to xentrace to aid analysis and
obtain the list length. With this patch, some data can be acquired to
validate patch 3/4. 

[1] 
https://lists.gt.net/xen/devel/422661?search_string=VT-d%20posted-interrupt%20core%20logic%20handling;#422661
[2] 
https://lists.gt.net/xen/devel/422567?search_string=%20The%20length%20of%20the%20list%20depends;#422567
[3] 
https://lists.gt.net/xen/devel/472749?search_string=enable%20vt-d%20pi%20by%20default;#472749


Chao Gao (4):
  VT-d PI: track the vcpu number on pi blocking list
  x86/vcpu: track hvm vcpu number on the system
  VT-d PI: restrict the vcpu number on a given pcpu
  Xentrace: add support for HVM's PI blocking list operation

 tools/xentrace/formats          |   3 +
 tools/xentrace/xenalyze.c       | 154 +++++++++++++++++++++++++++++++++---
 xen/arch/x86/hvm/hvm.c          |   6 ++
 xen/arch/x86/hvm/vmx/vmx.c      | 169 +++++++++++++++++++++++++++++++++-------
 xen/include/asm-x86/hvm/hvm.h   |   3 +
 xen/include/asm-x86/hvm/trace.h |   1 +
 xen/include/public/trace.h      |   5 ++
 7 files changed, 305 insertions(+), 36 deletions(-)

-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel