[Xen-devel] [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c

[Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>]

Hi all,

We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen. It has been enabled
in Linux Kernel.

Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible and easily to extend to add new features. We abstract the general
operations of all features and encapsulate them into a structure. Then,
the development of new feature is simple to mainly implement these callback
functions.

The patch set can be found at:
https://github.com/yisun-git/xen.git l2_cat_v9

---
Acked and Reviewed list before V9:

a - Acked-by
r - Reviewed-by

  r  patch 1  - docs: create Cache Allocation Technology (CAT) and Code and
                Data Prioritization (CDP) feature document
  ar patch 2  - x86: refactor psr: remove L3 CAT/CDP codes.
  r  patch 3  - x86: refactor psr: implement main data structures.
  r  patch 6  - x86: refactor psr: L3 CAT: implement Domain init/free and
                schedule flows.
  r  patch 7  - x86: refactor psr: L3 CAT: implement get hw info flow.
  r  patch 8  - x86: refactor psr: L3 CAT: implement get value flow.
  a  patch 22 - tools: L2 CAT: support get HW info for L2 CAT.
  a  patch 23 - tools: L2 CAT: support show cbm for L2 CAT.
  a  patch 24 - tools: L2 CAT: support set cbm for L2 CAT.
  a  patch 25 - docs: add L2 CAT description in docs.

---
V9 change list:

Patch 1:
    - add 'CMT' to the list of acronyms.
      (suggested by Wei Liu)
    - change feature list to feature array.
    - modify data structure descriptions according to latest codes.
    - modify revision.
Patch 3:
    - replace feature list to a feature pointer array.
      (suggested by Roger Pau)
    - add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
      account.
      (suggested by Roger Pau)
    - move 'feat_ops' declaration into 'feat_node' structure.
      (suggested by Roger Pau)
    - directly use uninon for feature HW info and move its declaration into
      'feat_node' structure.
      (suggested by Roger Pau)
    - remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
      not useful after using feature pointer array.
      (suggested by Roger Pau)
    - rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
    - remove 'nr_feat' which is only for a record.
      (suggested by Jan Beulich)
    - add 'cos_num' to record how many COS registers are used by a feature in
      one time access.
      (suggested by Jan Beulich)
    - replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
      max 32 bits for it.
      (suggested by Jan Beulich)
Patch 4:
    - create this patch alone to move 'cpuid_count_leaf'.
      (suggested by Wei Liu)
Patch 5:
    - add commit message to explain the flows.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - create a new patch about moving 'cpuid_count_leaf'.
      (suggested by Wei Liu)
    - modify comment to explain why not free some resource in 'free_feature'.
      (suggested by Wei Liu)
    - implement 'psr_alloc_feat_enabled' to check if allocation feature is
      enabled in cmdline and some initialization work done.
      (suggested by Wei Liu)
    - implement 'cat_default_val' to set default value for CAT features.
      (suggested by Wei Liu)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
      init functions.
      (suggested by Roger Pau)
    - modify comments for global feature node.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - remove unnecessary 'else'.
      (suggested by Jan Beulich)
    - remove 'nr_feat'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - check global flag with boot cpu operations.
      (suggested by Jan Beulich)
    - remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
      (suggested by Jan Beulich)
    - assign value for 'cos_num'.
      (suggested by Jan Beulich)
    - change about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 6:
    - rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement 'psr_alloc_cos' to match 'psr_free_cos'.
      (suggested by Wei Liu)
    - use 'psr_alloc_feat_enabled'.
      (suggested by Wei Liu)
    - fix coding style issue.
      (suggested by Wei Liu)
    - remove 'inline'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - remove 'psr_cos_ids' check in 'psr_free_cos'.
      (suggested by Jan Beulich)
Patch 7:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - define 'PSR_INFO_SIZE'.
      (suggested by Roger Pau)
    - fix coding style issue.
      (suggested by Roger Pau and Jan Beulich)
    - use 'ARRAY_SIZE'.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
      function for both L3/L2 CAT.
      (suggested by Roger Pau)
    - move constant to the right of comparison.
      (suggested by Wei Liu)
    - remove wrong comment.
      (suggested by Jan Beulich)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
    - remove assignment for 'PSR_SOCKET_UNKNOWN'.
      (suggested by Jan Beulich)
    - retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move common data check into common function.
      (suggested by Jan Beulich)
Patch 8:
    - add commit message to explain there is an user space interface.
    - rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
      domain. Also declare it to 'const'.
      (suggested by Jan Beulich)
    - change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
      more reasonable.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move cos check into common function because this check is required by all
      features.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'..
      (suggested by Jan Beulich)
Patch 9:
    - use goto style error handling in 'psr_set_val'.
      (suggested by Wei Liu)
    - use ASSERT for checking old_cos.
      (suggested by Wei Liu and Jan Beulich)
    - fix coding style issue.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - use 'spin_is_locked' to check ref_lock.
      (suggested by Roger Pau)
    - add an input parameter 'array_len' for 'write_psr_msr'.
    - check 'socket_info' and 'psr_cos_ids' in this patch.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - fix commit message words.
      (suggested by Jan Beulich)
    - change 'assemble_val_array' to 'gather_val_array'.
      (suggested by Jan Beulich)
    - change 'set_new_val_to_array' to 'insert_new_val_to_array'.
      (suggested by Jan Beulich)
    - change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' to 'write_psr_msrs'.
      (suggested by Jan Beulich)
    - correct comments.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - adjust conditions after 'find_cos' to save a level of indentation.
      (suggested by Jan Beulich)
    - add 'ASSERT(!old_cos || ref[old_cos])'.
      (suggested by Jan Beulich)
    - move ASSERT() check into locked region.
      (suggested by Jan Beulich)
    - replace parameter '*val' to 'val[]' in some functions.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' parameters to prepare to only set one new value
      for one feature.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
    - add explanation about context switch.
      (suggested by Jan Beulich)
Patch 10:
    - add comments about boundary checking.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
      features and reuse cat_get_val in it.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - replace 'm' to 'new_val'.
      (suggested by Jan Beulich)
    - move cos check outside callback function.
      (suggested by Jan Beulich)
    - remove 'get_cos_num' callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 11:
    - modify comments of 'compare_val' to be same as current implementation.
      (suggested by Wei Liu)
    - fix indentation issue.
      (suggested by Wei Liu)
    - rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 12:
    - modify return value of 'pick_avail_cos' to make it more accurate.
    - rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - fix comment.
      (suggested by Wei Liu)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 13:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - add 'array_len' in 'struct cos_write_info' and check if val array
      exceeds it.
    - modify 'write_psr_msr' flow only to set one value a time. No need to
      set whole feature array values.
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 14:
    - modify commit message to describe flow clearer.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
      (suggested by Roger Pau and Jan Beulich)
    - encapsulate CDP operations into 'cat_init_feature' to reduce redundant
      codes.
      (suggested by Roger Pau)
    - reuse 'cat_get_cos_max' for CDP.
      (suggested by Roger Pau)
    - handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
      there is only 'psr=cdp' in cmdline.
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - add codes to handle CDP's 'cos_num'.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - do not free resources when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 15:
    - modify commit message to explain flow more clearly.
    - reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
      (suggested by Roger Pau)
    - fix coding style issues.
      (suggested by Wei Liu and Roger Pau)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
Patch 16:
    - modify the type of 'l3_cdp_get_val' to 'void'.
    - cos checking has been done in common function so remove related codes
      in CDP callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 17:
    - add comment to explain why CDP uses 2 COSs.
      (suggested by Wei Liu)
    - use 'cat_default_val'.
      (suggested by Wei Liu)
    - remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
      feat_node now.
      (suggested by Jan Beulich)
    - remove cos checking because it has been moved to common function.
      (suggested by Jan Beulich)
    - l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
      (suggested by Jan Beulich)
    - directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
      default value.
      (suggested by Jan Beulich)
    - modify 'l3_cdp_write_msr' flow to write value into register according
      to input type.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 18:
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
      redundant codes.
      (suggested by Roger Pau)
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - do not free resource when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
Patch 19:
    - reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - modify sysctl implementation of L2 CAT to input data[3] to use
      'cat_get_feat_info'.
      (suggested by Roger Pau)
    - modify macros names to newly defined ones.
      (suggested by Jan Beulich)
    - remove 'l2_info' to reuse 'l3_info'.
      (suggested by Jan Beulich)
    - modify macro name according to previous patch change.
      (suggested by Jan Beulich)
Patch 20:
    - reuse 'cat_get_val' for L2 CAT to reduce redundant codes
      (suggested by Roger Pau)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 21:
    - reuse some CAT common functions for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not. These things are
      done in patch "x86: refactor psr: set value: implement cos finding flow."
      (suggested by Roger Pau and Dario Faggioli)
    - remove 'get_cos_num' related codes.
      (suggested by Jan Beulich)
    - modify 'l2_cat_write_msr' according to previous patch change.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 22:
    - add some cases to handle return error no.
    - move xl_cmdimpl.c codes into xl/xl_psr.c.
    - change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
Patch 23:
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.
Patch 24:
    - handle the case to set both CODE and DATA for CDP at same time.
      For such case, user does not input '-c' or '-d' to set CDP cbm.
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.

Yi Sun (25):
  docs: create Cache Allocation Technology (CAT) and Code and Data
    Prioritization (CDP) feature document
  x86: refactor psr: remove L3 CAT/CDP codes.
  x86: refactor psr: implement main data structures.
  x86: move cpuid_count_leaf from cpuid.c to processor.h.
  x86: refactor psr: L3 CAT: implement CPU init and free flow.
  x86: refactor psr: L3 CAT: implement Domain init/free and schedule
    flows.
  x86: refactor psr: L3 CAT: implement get hw info flow.
  x86: refactor psr: L3 CAT: implement get value flow.
  x86: refactor psr: L3 CAT: set value: implement framework.
  x86: refactor psr: L3 CAT: set value: assemble features value array.
  x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  x86: refactor psr: L3 CAT: set value: implement write msr flow.
  x86: refactor psr: CDP: implement CPU init and free flow.
  x86: refactor psr: CDP: implement get hw info flow.
  x86: refactor psr: CDP: implement get value flow.
  x86: refactor psr: CDP: implement set value callback functions.
  x86: L2 CAT: implement CPU init and free flow.
  x86: L2 CAT: implement get hw info flow.
  x86: L2 CAT: implement get value flow.
  x86: L2 CAT: implement set value flow.
  tools: L2 CAT: support get HW info for L2 CAT.
  tools: L2 CAT: support show cbm for L2 CAT.
  tools: L2 CAT: support set cbm for L2 CAT.
  docs: add L2 CAT description in docs.

 docs/features/intel_psr_cat_cdp.pandoc |  469 ++++++++++
 docs/man/xl.pod.1.in                   |   25 +-
 docs/misc/xl-psr.markdown              |   18 +-
 tools/libxc/include/xenctrl.h          |    7 +-
 tools/libxc/xc_psr.c                   |   45 +-
 tools/libxl/libxl.h                    |    9 +
 tools/libxl/libxl_psr.c                |   70 +-
 tools/libxl/libxl_types.idl            |    1 +
 tools/xl/xl_cmdtable.c                 |    6 +-
 tools/xl/xl_psr.c                      |  168 ++--
 xen/arch/x86/cpuid.c                   |    6 -
 xen/arch/x86/domctl.c                  |   53 +-
 xen/arch/x86/psr.c                     | 1497 +++++++++++++++++++++++++-------
 xen/arch/x86/sysctl.c                  |   58 +-
 xen/include/asm-x86/msr-index.h        |    1 +
 xen/include/asm-x86/processor.h        |    7 +
 xen/include/asm-x86/psr.h              |   27 +-
 xen/include/public/domctl.h            |    2 +
 xen/include/public/sysctl.h            |    3 +-
 19 files changed, 2025 insertions(+), 447 deletions(-)
 create mode 100644 docs/features/intel_psr_cat_cdp.pandoc

-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel