|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [PATCH v5 3/3] tools & docs: add tools and docs support for Intel CDP
This is the xl/xc changes to support Intel Code/Data Prioritization.
CAT xl commands to set/get CBMs are extended to support CDP.
Add new CDP options with CAT commands in xl interface man page.
Add description of CDP in xl-psr.markdown.
Signed-off-by: He Chen <he.chen@xxxxxxxxxxxxxxx>
---
Changes in v5:
* merge tools and docs patches
* replace EINVAL with ENXIO in libxl__psr_cat_log_err_msg
* revise options parsing in psr-cat-cbm-set and invalidate passing -c
and -d at the same time
* refine CDP status output codes in psr_cat_hwinfo
* adjust CBM output format in command xl psr-cat-show
* docs revision
Example of new output format for command xl psr-cat-show:
CAT-only:
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM
0 Domain-0 0xfffff
1 centos.hvm 0xfffff
CDP enabled:
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM
0 Domain-0 code: 0xfffff data: 0xfffff
1 centos.hvm code: 0xfffff data: 0xfffff
---
docs/man/xl.pod.1 | 15 ++++++++++++
docs/misc/xl-psr.markdown | 47 +++++++++++++++++++++++++++++++-----
tools/libxc/include/xenctrl.h | 7 ++++--
tools/libxc/xc_psr.c | 17 ++++++++++++-
tools/libxl/libxl.h | 7 ++++++
tools/libxl/libxl_psr.c | 5 +++-
tools/libxl/libxl_types.idl | 3 +++
tools/libxl/xl_cmdimpl.c | 56 +++++++++++++++++++++++++++++++++----------
tools/libxl/xl_cmdtable.c | 2 ++
9 files changed, 137 insertions(+), 22 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index f22c3f3..6388351 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1530,6 +1530,13 @@ applications. In the Xen implementation, CAT is used to
control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
+Intel Broadwell and later server platforms also offer Code/Data Prioritization
+(CDP) for cache allocations, which support specifying code or data cache for
+applications. CDP is used on a per VM basis in the Xen implementation. To
+specify code or data CBM for the domain, CDP feature must be enabled and CBM
+type options need to be specified when setting CBM, and the type options (code
+and data) are mutually exclusive.
+
=over 4
=item B<psr-cat-cbm-set> [I<OPTIONS>] I<domain-id> I<cbm>
@@ -1545,6 +1552,14 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-c>, B<--code>
+
+Set code CBM when CDP is enabled.
+
+=item B<-d>, B<--data>
+
+Set data CBM when CDP is enabled.
+
=back
=item B<psr-cat-show> [I<domain-id>]
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index 3545912..0527211 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -14,7 +14,7 @@ tracks cache utilization of memory accesses according to the
RMID and reports
monitored data via a counter register.
For more detailed information please refer to Intel SDM chapter
-"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology".
+"Platform Shared Resource Monitoring: Cache Monitoring Technology".
In Xen's implementation, each domain in the system can be assigned a RMID
independently, while RMID=0 is reserved for monitoring domains that don't
@@ -52,7 +52,7 @@ event type to monitor system total/local memory bandwidth.
The same RMID can
be used to monitor both cache usage and memory bandwidth at the same time.
For more detailed information please refer to Intel SDM chapter
-"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology".
+"Platform Shared Resource Monitoring: Cache Monitoring Technology".
In Xen's implementation, MBM shares the same set of underlying monitoring
service with CMT and can be used to monitor memory bandwidth on a per domain
@@ -91,17 +91,42 @@ For example, assuming a system with 8 portions and 3
domains:
first domain exclusive access to half the cache, and the other two exclusive
access to one quarter each.
-For more detailed information please refer to Intel SDM chapter
-"17.15 - Platform Shared Resource Control: Cache Allocation Technology".
-
In Xen's implementation, CBM can be configured with libxl/xl interfaces but
COS is maintained in hypervisor only. The cache partition granularity is per
domain, each domain has COS=0 assigned by default, the corresponding CBM is
all-ones, which means all the cache resource can be used by default.
+Code/Data Prioritization (CDP) Technology is an extension of CAT, which is
+available on Intel Broadwell and later server platforms. CDP enables isolation
+and separate prioritization of code and data fetches to the L3 cache in a
+software configurable manner, which can enable workload prioritization and
+tuning of cache capacity to the characteristics of the workload. CDP extends
+Cache Allocation Technology (CAT) by providing separate code and data masks
+per Class of Service (COS).
+
+CDP can be enabled by adding `psr=cdp` to Xen command line.
+
+When CDP is enabled,
+
+ * the CAT masks are re-mapped into interleaved pairs of masks for data or code
+ fetches.
+
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+CDP allows the OS or Hypervisor to partition cache allocation in a more
+fine-grained. Code cache and data cache can be specified independently. With
CDP
+enabled, one COS corresponds to two CBMs (code CBM & data CBM), since the sum
of
+CBMs is fixed, that means the number of available COSes will reduce by half
when
+CDP is on.
+
+For more detailed information please refer to Intel SDM chapter
+"Platform Shared Resource Control: Cache Allocation Technology".
+
### xl interfaces
-System CAT information such as maximum COS and CBM length can be obtained by:
+System CAT information such as maximum COS, CBM length and CDP status can be
+obtained by:
`xl psr-hwinfo --cat`
@@ -116,6 +141,16 @@ A cbm is valid only when:
obtained with `xl psr-hwinfo --cat`.
* All the set bits are contiguous.
+When CDP is enabled, `-c` or `--code` option is available to set code CBM for
+the domain.
+
+When CDP is enabled, `-d` or `--data` option is available to set data CBM for
+the domain.
+
+If neither `-c` nor `-d` option is specified when CDP is on, the same code CBM
+and data CBM will be set for the domain. Passing both `-c` and `-d` options is
+invalid.
+
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 2000f12..8f4acec 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2794,7 +2794,9 @@ enum xc_psr_cmt_type {
typedef enum xc_psr_cmt_type xc_psr_cmt_type;
enum xc_psr_cat_type {
- XC_PSR_CAT_L3_CBM = 1,
+ XC_PSR_CAT_L3_CBM = 1,
+ XC_PSR_CAT_L3_CODE = 2,
+ XC_PSR_CAT_L3_DATA = 3,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
@@ -2820,7 +2822,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch,
uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len);
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
#endif
#endif /* XENCTRL_H */
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index d8b3a51..5bbe950 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -260,6 +260,12 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t
domid,
case XC_PSR_CAT_L3_CBM:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM;
break;
+ case XC_PSR_CAT_L3_CODE:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE;
+ break;
+ case XC_PSR_CAT_L3_DATA:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
+ break;
default:
errno = EINVAL;
return -1;
@@ -287,6 +293,12 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t
domid,
case XC_PSR_CAT_L3_CBM:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM;
break;
+ case XC_PSR_CAT_L3_CODE:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE;
+ break;
+ case XC_PSR_CAT_L3_DATA:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
+ break;
default:
errno = EINVAL;
return -1;
@@ -306,7 +318,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t
domid,
}
int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len)
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled)
{
int rc;
DECLARE_SYSCTL;
@@ -320,6 +333,8 @@ int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t
socket,
{
*cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
*cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 5f9047c..611e98d 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -796,6 +796,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst,
libxl_mac *src);
* If this is defined, the Cache Allocation Technology feature is supported.
*/
#define LIBXL_HAVE_PSR_CAT 1
+
+/*
+ * LIBXL_HAVE_PSR_CDP
+ *
+ * If this is defined, the Code and Data Prioritization feature is supported.
+ */
+#define LIBXL_HAVE_PSR_CDP 1
#endif
/*
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index 3378239..4ad0fc8 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -87,6 +87,9 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
case EEXIST:
msg = "The same CBM is already set to this domain";
break;
+ case ENXIO:
+ msg = "Unable to set code or data CBM when CDP is disabled";
+ break;
default:
libxl__psr_log_err_msg(gc, err);
@@ -352,7 +355,7 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx,
libxl_psr_cat_info **info,
for (i = 0; i < nr_sockets; i++) {
if (xc_psr_cat_get_l3_info(ctx->xch, i, &ptr[i].cos_max,
- &ptr[i].cbm_len)) {
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
libxl__psr_cat_log_err_msg(gc, errno);
rc = ERROR_FAIL;
free(ptr);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index ef346e7..fa017ad 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -787,9 +787,12 @@ libxl_psr_cmt_type = Enumeration("psr_cmt_type", [
libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(0, "UNKNOWN"),
(1, "L3_CBM"),
+ (2, "L3_CODE"),
+ (3, "L3_DATA"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
("cos_max", uint32),
("cbm_len", uint32),
+ ("cdp_enabled", bool),
])
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index ebbb9a5..dd16414 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -8390,6 +8390,8 @@ static int psr_cat_hwinfo(void)
}
printf("%-16s: %u\n", "Socket ID", socketid);
printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
+ printf("%-16s: %s\n", "CDP Status",
+ info->cdp_enabled ? "Enabled" : "Disabled");
printf("%-16s: %u\n", "Maximum COS", info->cos_max);
printf("%-16s: %u\n", "CBM length", info->cbm_len);
printf("%-16s: %#llx\n", "Default CBM",
@@ -8401,7 +8403,8 @@ out:
return rc;
}
-static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid)
+static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
+ bool cdp_enabled)
{
char *domain_name;
uint64_t cbm;
@@ -8410,20 +8413,29 @@ static void psr_cat_print_one_domain_cbm(uint32_t
domid, uint32_t socketid)
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_CBM,
- socketid, &cbm))
- printf("%#16"PRIx64, cbm);
-
+ if (!cdp_enabled) {
+ if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_CBM,
+ socketid, &cbm))
+ printf("0x%22"PRIx64, cbm);
+ } else {
+ if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_CODE,
+ socketid, &cbm))
+ printf("%10s%#8"PRIx64, "code:", cbm);
+ if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_DATA,
+ socketid, &cbm))
+ printf("%10s%#8"PRIx64, "data:", cbm);
+ }
printf("\n");
}
-static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid)
+static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
+ bool cdp_enabled)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
return 0;
}
@@ -8433,7 +8445,7 @@ static int psr_cat_print_domain_cbm(uint32_t domid,
uint32_t socketid)
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
libxl_dominfo_list_free(list, nr_domains);
return 0;
@@ -8455,9 +8467,9 @@ static int psr_cat_print_socket(uint32_t domid, uint32_t
socketid,
printf("%-16s: %u\n", "Socket ID", socketid);
printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
- printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
+ printf("%5s%25s%22s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, socketid);
+ return psr_cat_print_domain_cbm(domid, socketid, info->cdp_enabled);
}
static int psr_cat_show(uint32_t domid)
@@ -8486,9 +8498,10 @@ out:
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
- libxl_psr_cbm_type type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ libxl_psr_cbm_type type;
uint64_t cbm;
int ret, opt = 0;
+ int opt_data = 0, opt_code = 0;
libxl_bitmap target_map;
char *value;
libxl_string_list socket_list;
@@ -8497,13 +8510,15 @@ int main_psr_cat_cbm_set(int argc, char **argv)
static struct option opts[] = {
{"socket", 1, 0, 's'},
+ {"data", 0, 0, 'd'},
+ {"code", 0, 0, 'c'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -8517,6 +8532,23 @@ int main_psr_cat_cbm_set(int argc, char **argv)
libxl_string_list_dispose(&socket_list);
free(value);
break;
+ case 'd':
+ opt_data = 1;
+ break;
+ case 'c':
+ opt_code = 1;
+ break;
+ }
+
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Invalid to pass both -c and -d at the same time\n");
+ return -1;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
}
if (libxl_bitmap_is_empty(&target_map))
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 0071f12..fdc1ac6 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -543,6 +543,8 @@ struct cmd_spec cmd_table[] = {
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets
are processed\n"
+ "-c Set code CBM if CDP is supported\n"
+ "-d Set data CBM if CDP is supported\n"
},
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |