|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
On Thu, Jan 19, 2017 at 02:01:03PM +0800, Yi Sun wrote:
> This patch creates L2 CAT feature document in doc/features/.
> It describes details of L2 CAT.
Perhaps also mention what is the title in the Intel SDM
to look into as well?
Perhaps:
"See Intel Resource Director Technology Monitoring Features"
in the Intel SDM."
>
> Signed-off-by: Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>
> ---
> docs/features/intel_psr_l2_cat.pandoc | 347
> ++++++++++++++++++++++++++++++++++
> 1 file changed, 347 insertions(+)
> create mode 100644 docs/features/intel_psr_l2_cat.pandoc
>
> diff --git a/docs/features/intel_psr_l2_cat.pandoc
> b/docs/features/intel_psr_l2_cat.pandoc
> new file mode 100644
> index 0000000..77bd61f
> --- /dev/null
> +++ b/docs/features/intel_psr_l2_cat.pandoc
> @@ -0,0 +1,347 @@
> +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> +% Revision 1.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> + Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> + Component(s): Hypervisor, toolstack
> +
> + Hardware: Atom codename Goldmont and beyond CPUs
> +---------------- ----------------------------------------------------
> +
> +# Overview
> +
> +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> +CPU's shared L2 cache based on application priority or Class of Service
> +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> +represent cache capacity and indicate the degree of overlap and
indicates
> +isolation between classes. Once L2 CAT is configured, the processor
> +allows access to portions of L2 cache according to the established
> +class of service.
> +
> +## Terminology
> +
> +* CAT Cache Allocation Technology
> +* CBM Capacity BitMasks
> +* CDP Code and Data Prioritization
> +* COS/CLOS Class of Service
> +* MSRs Machine Specific Registers
> +* PSR Intel Platform Shared Resource
> +* VMM Virtual Machine Monitor
> +
> +# User details
> +
> +* Feature Enabling:
> +
> + Add "psr=cat" to boot line parameter to enable all supported level CAT
> + features.
> +
> +* xl interfaces:
> +
> + 1. `psr-cat-show [OPTIONS] domain-id`:
> +
> + Show domain L2 or L3 CAT CBM.
> +
> + New option `-l` is added.
> + `-l2`: Show cbm for L2 cache.
> + `-l3`: Show cbm for L3 cache.
> +
> + If neither `-l2` nor `-l3` is given, show both of them. If any one
> + is not supported, will print error info.
> +
> + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> +
> + Set domain L2 or L3 CBM.
> +
> + New option `-l` is added.
> + `-l2`: Specify cbm for L2 cache.
> + `-l3`: Specify cbm for L3 cache.
> +
> + If neither `-l2` nor `-l3` is given, level 3 is the default option.
> +
> + 3. `psr-hwinfo [OPTIONS]`:
> +
> + Show L2 & L3 CAT HW informations on every socket.
> +
> +# Technical details
> +
> +L2 CAT is a member of Intel PSR features and part of CAT, it shares
> +some base PSR infrastructure in Xen.
> +
> +## Hardware perspective
> +
> +L2 CAT defines a new range MSRs to assign different L2 cache access
^- 'of'
> +patterns which are known as CBMs, each CBM is associated with a COS.
> +
> +```
> +
> + +----------------------------+----------------+
> + IA32_PQR_ASSOC | MSR (per socket) | Address |
> + +----+---+-------+ +----------------------------+----------------+
> + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
> + +----+---+-------+ +----------------------------+----------------+
> + └-------------> | ... | ... |
> + +----------------------------+----------------+
> + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
> + +----------------------------+----------------+
> +```
> +
> +When context switch happens, the COS of VCPU is written to per-thread
> +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> +according to the corresponding CBM.
> +
> +## The relationship between L2 CAT and L3 CAT/CDP
> +
> +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
s/all/both/
> +
> +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
s/by//
> +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
> +L2 CAT must be continuous too.
> +
> +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> +associate register `IA32_PQR_ASSOC`, which means one COS associates to a
s/associates to a/is associated with a/
> +pair of L2 CBM and L3 CBM.
> +
> +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> +other PSR features in future). In some cases, a VM is permitted to have a
> +COS that is beyond one (or more) of PSR features but within the others.
> +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> +limit) since COS 9 is beyond the max COS (8) of L2 CAT.
Thank you for mentioning the above!
> +
> +## Design Overview
> +
> +* Core COS/CBM association
> +
> + When enforcing L2 CAT, all cores of domains have the same default
> + COS (COS0) which associated to the fully open CBM (all ones bitmask)
^-is
> + to access all L2 cache. The default COS is used only in hypervisor
> + and is transparent to tool stack and user.
> +
> + System administrator can change PSR allocation policy at runtime by
> + tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
> + to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
> + is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
> + L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
> + things would be easier, one COS corresponds to one L2 CBM.
> +
> +* VCPU schedule
> +
> + This part reuses L3 CAT COS infrastructure.
> +
> +* Multi-sockets
> +
> + Different sockets may have different L2 CAT capability (e.g. max COS)
> + although it is consistent on the same socket. So the capability of
> + per-socket L2 CAT is specified.
> +
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> + 1. Boot line parameter "psr=cat" now will enable L2 CAT and L3
> + CAT if hardware supported.
> +
> + 2. SYSCTL:
> + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> +
> + 3. DOMCTL:
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
> +
> +* xl interfaces:
> +
> + 1. psr-cat-show -l2 domain-id
> + Show L2 cbm for a domain.
> + => XEN_SYSCTL_PSR_CAT_get_l2_info /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> +
> + 2. psr-mba-set -l2 domain-id cbm
> + Set L2 cbm for a domain.
> + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> +
> + 3. psr-hwinfo
> + Show PSR HW information, including L2 CAT
> + => XEN_SYSCTL_PSR_CAT_get_l2_info
> +
> +* Key data structure:
> +
> + 1. Feature HW info
> +
> + ```
> + struct psr_cat_hw_info {
> + unsigned int cbm_len;
> + unsigned int cos_max;
> + };
> + ```
> +
> + - Member `cbm_len`
> +
> + `cbm_len` is one of the hardware info of CAT.
Could you expand? Is it the max number of bits you can set?
> +
> + - Member `cos_max`
> +
> + `cos_max` is one of the hardware info of CAT.
> +
> + 2. Feature list node
> +
> + ```
> + struct feat_node {
> + enum psr_feat_type feature;
> + struct feat_ops ops;
> + struct psr_cat_hw_info info;
> + uint64_t cos_reg_val[MAX_COS_REG_NUM];
> + struct list_head list;
> + };
> + ```
> +
> + When a PSR enforcement feature is enabled, it will be added into a
> + feature list. The head of the list is created in psr initialization.
> +
> + - Member `feature`
> +
> + `feature` is an integer number, to indicate which feature the list
> entry
> + corresponds to.
> +
> + - Member `ops`
> +
> + `ops` maintains a callback function list of the feature. It will be
> introduced
> + in details later.
??? Like by each patch to this document?
> +
> + - Member `info`
> +
> + `info` maintains the feature HW information which can be got through
s/which an be got through/which are provided to/
Perhaps?
> + psr_hwinfo command.
> +
> + - Member `cos_reg_val`
> +
> + `cos_reg_val` is an array to maintain the value set in all COS
> registers of
> + the feature.
> +
> + 3. Per-socket PSR features information structure
> +
> + ```
> + struct psr_cat_socket_info {
> + unsigned int feat_mask;
> + unsigned int nr_feat;
> + struct list_head feat_list;
> + unsigned int cos_ref[MAX_COS_REG_NUM];
> + spinlock_t ref_lock;
> + };
> + ```
> +
> + We collect all PSR allocation features information of a socket in
> + this `struct psr_cat_socket_info`.
> +
> + - Member `feat_mask`
> +
> + `feat_mask` is a bitmap, to indicate which feature is enabled on
> + current socket. We define `feat_mask` bitmap as:
> +
> + bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
> + stands for L3 CDP is enalbed.
enabled
> +
> + bit 2: L2 CAT status.
And the 'nr_feat' is 3 ?
> +
> + - Member `cos_ref`
> +
> + `cos_ref` is an array which maintains the reference of one COS.
Is it safe to assume that this maps to 'cos_reg_val' ? If so you may
want to mention that.
> + If the COS is used by one domain, the reference will increase one.
s/one/by one/
> + If a domain releases the COS, the reference will decrease one. The
s/decrease one/decrease by one/
> + array is indexed by COS.
> +
> + 4. Feature operation functions structure
> +
> + ```
> + struct feat_ops {
> + void (*init_feature)(unsigned int eax, unsigned int ebx,
> + unsigned int ecx, unsigned int edx,
> + struct feat_node *feat,
> + struct psr_cat_socket_info *info);
> + int (*get_feat_info)(const struct feat_node *feat, enum cbm_type
> type,
> + uint32_t dat[], uint32_t array_len);
> + int (*get_val)(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val);
> + unsigned int (*get_max_cos_max)(const struct feat_node *feat);
> + unsigned int (*get_cos_num)(const struct feat_node *feat);
> + int (*get_old_val)(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos);
> + int (*set_new_val)(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos,
> + enum cbm_type type,
> + uint64_t m);
> + int (*compare_val)(const uint64_t val[], const struct feat_node
> *feat,
> + unsigned int cos, bool *found);
> + unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
> + enum cbm_type type);
> + unsigned int (*exceeds_cos_max)(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos);
> + int (*write_msr)(unsigned int cos, const uint64_t val[],
> + struct feat_node *feat);
> + };
> + ```
> +
> + We abstract above callback functions to encapsulate the feature
> specific
> + behaviors into them. Then, it is easy to add a new feature. We just
> need:
> + 1) Implement such ops and callback functions for every feature.
> + 2) Register the ops into `struct feat_node`.
> + 3) Add the feature into feature list during CPU initialization.
> +
> +# Limitations
> +
> +L2 CAT can only work on HW which enables it(check by CPUID). So far, there
> +is no HW enables both L2 CAT and L3 CAT/CDP. But SW implementation has
> considered
s/no HW/no HW which/
> +such scenario to enable both L2 CAT and L3 CAT/CDP.
> +
> +# Testing
> +
> +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
> +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
> +
> +For example:
> + root@:~$ xl psr-hwinfo --cat
> + Cache Allocation Technology (CAT): L2
> + Socket ID : 0
> + Maximum COS : 3
> + CBM length : 8
> + Default CBM : 0xff
> +
> + root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> +
> + root@:~$ xl psr-cat-show -l2 1
> + Socket ID : 0
> + Default CBM : 0xff
> + ID NAME CBM
> + 1 ubuntu14 0x7f
> +
> +# Areas for improvement
> +
> +N/A
> +
> +# Known issues
> +
> +N/A
> +
> +# References
> +
> +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES"
> [Intel® 64 and IA-32 Architectures Software Developer Manuals,
> vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date Revision Version Notes
> +---------- -------- -------- -------------------------------------------
> +2016-08-12 1.0 Xen 4.9 Design document written
> +---------- -------- -------- -------------------------------------------
> --
> 1.9.1
>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |