|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [PATCH v10 0/6] enable Cache QoS Monitoring (CQM) feature
Changes from v9:
- Revise the readonly mapping mechanism to share data between Xen and
userspace. We create L3C buffer for each socket, share both buffer
address MFNs and buffer MFNs to userspace.
- Split the pqos.c into pqos.c and cqm.c for better code structure.
- Show the total L3 cache size when issueing xl pqos-list cqm command.
- Abstract a libxl_getcqminfo() function to fetch cqm data from Xen.
- Several coding style fixes.
Changes from v8:
- Address comments from Ian Campbell, including:
* Modify the return handling for xc_sysctl();
* Add man page items for platform QoS related commands.
* Fix typo in commit message.
Changes from v7:
- Address comments from Andrew Cooper, including:
* Check CQM capability before allocating cpumask memory.
* Move one function declaration into the correct patch.
Changes from v6:
- Address comments from Jan Beulich, including:
* Remove the unnecessary CPUID feature check.
* Remove the unnecessary socket_cpu_map.
* Spin_lock related changes, avoid spin_lock_irqsave().
* Use readonly mapping to pass cqm data between Xen/Userspace,
to avoid data copying.
* Optimize RDMSR/WRMSR logic to avoid unnecessary calls.
* Misc fixes including __read_mostly prefix, return value, etc.
Changes from v5:
- Address comments from Dario Faggioli, including:
* Define a new libxl_cqminfo structure to avoid reference of xc
structure in libxl functions.
* Use LOGE() instead of the LIBXL__LOG() functions.
Changes from v4:
- When comparing xl cqm parameter, use strcmp instead of strncmp,
otherwise, "xl pqos-attach cqmabcd domid" will be considered as
a valid command line.
- Address comments from Andrew Cooper, including:
* Adjust the pqos parameter parsing function.
* Modify the pqos related documentation.
* Add a check for opt_cqm_max_rmid in initialization code.
* Do not IPI CPU that is in same socket with current CPU.
- Address comments from Dario Faggioli, including:
* Fix an typo in export symbols.
* Return correct libxl error code for qos related functions.
* Abstract the error printing logic into a function.
- Address comment from Daniel De Graaf, including:
* Add return value in for pqos related check.
- Address comments from Konrad Rzeszutek Wilk, including:
* Modify the GPLv2 related file header, remove the address.
Changes from v3:
- Use structure to better organize CQM related global variables.
- Address comments from Andrew Cooper, including:
* Remove the domain creation flag for CQM RMID allocation.
* Adjust the boot parameter format, use custom_param().
* Add documentation for the new added boot parameter.
* Change QoS type flag to be uint64_t.
* Initialize the per socket cpu bitmap in system boot time.
* Remove get_cqm_avail() function.
* Misc of format changes.
- Address comment from Daniel De Graaf, including:
* Use avc_current_has_perm() for XEN2__PQOS_OP that belongs to SECCLASS_XEN2.
Changes from v2:
- Address comments from Andrew Cooper, including:
* Merging tools stack changes into one patch.
* Reduce the IPI number to one per socket.
* Change structures for CQM data exchange between tools and Xen.
* Misc of format/variable/function name changes.
- Address comments from Konrad Rzeszutek Wilk, including:
* Simplify the error printing logic.
* Add xsm check for the new added hypercalls.
Changes from v1:
- Address comments from Andrew Cooper, including:
* Change function names, e.g., alloc_cqm_rmid(), system_supports_cqm(), etc.
* Change some structure element order to save packing cost.
* Correct some function's return value.
* Some programming styles change.
* ...
Future generations of Intel Xeon processor may offer monitoring capability in
each logical processor to measure specific quality-of-service metric,
for example, the Cache QoS Monitoring to get L3 cache occupancy.
Detailed information please refer to Intel SDM chapter 17.14.
Cache QoS Monitoring provides a layer of abstraction between applications and
logical processors through the use of Resource Monitoring IDs (RMIDs).
In Xen design, each guest in the system can be assigned an RMID independently,
while RMID=0 is reserved for monitoring domains that doesn't enable CQM service.
When any of the domain's vcpu is scheduled on a logical processor, the domain's
RMID will be activated by programming the value into one specific MSR, and when
the vcpu is scheduled out, a RMID=0 will be programmed into that MSR.
The Cache QoS Hardware tracks cache utilization of memory accesses according to
the RMIDs and reports monitored data via a counter register. With this solution,
we can get the knowledge how much L3 cache is used by a certain guest.
To attach CQM service to a certain guest, two approaches are provided:
1) Create the guest with "pqos_cqm=1" set in configuration file.
2) Use "xl pqos-attach cqm domid" for a running guest.
To detached CQM service from a guest, users can:
1) Use "xl pqos-detach cqm domid" for a running guest.
2) Also destroying a guest will detach the CQM service.
To get the L3 cache usage, users can use the command of:
$ xl pqos-list cqm
The below data is just an example showing how the CQM related data is exposed to
end user.
[root@localhost]# xl pqos-list cqm
Name ID SocketID L3C_Usage
SocketID L3C_Usage
Domain-0 0 0 19096 KB
1 13608 KB
ExampleHVMDomain1 1 0 0 KB
1 5992 KB
ExampleHVMDomain2 2 0 5600 KB
1 0 KB
ExampleHVMDomain3 3 0 0 KB
1 6048 KB
ExampleHVMDomain4 4 0 4928 KB
1 0 KB
ExampleHVMDomain5 5 0 0 KB
1 5488 KB
ExampleHVMDomain6 6 0 4704 KB
1 1512 KB
Socket L3 Cache Size: 35840 KB
RMID count 56 RMID available 48
Dongxiao Xu (6):
x86: detect and initialize Cache QoS Monitoring feature
x86: dynamically attach/detach CQM service for a guest
x86: collect CQM information from all sockets
x86: enable CQM monitoring for each domain RMID
xsm: add platform QoS related xsm policies
tools: enable Cache QoS Monitoring feature for libxl/libxc
docs/man/xl.pod.1 | 23 ++
docs/misc/xen-command-line.markdown | 7 +
tools/flask/policy/policy/modules/xen/xen.if | 2 +-
tools/flask/policy/policy/modules/xen/xen.te | 5 +-
tools/libxc/xc_domain.c | 41 ++++
tools/libxc/xenctrl.h | 13 ++
tools/libxl/Makefile | 3 +-
tools/libxl/libxl.h | 12 ++
tools/libxl/libxl_pqos.c | 190 +++++++++++++++++
tools/libxl/libxl_types.idl | 9 +
tools/libxl/xl.h | 3 +
tools/libxl/xl_cmdimpl.c | 109 ++++++++++
tools/libxl/xl_cmdtable.c | 15 ++
xen/arch/x86/Makefile | 1 +
xen/arch/x86/domain.c | 8 +
xen/arch/x86/domctl.c | 24 +++
xen/arch/x86/pqos/Makefile | 2 +
xen/arch/x86/pqos/cqm.c | 292 ++++++++++++++++++++++++++
xen/arch/x86/pqos/pqos.c | 94 +++++++++
xen/arch/x86/setup.c | 3 +
xen/arch/x86/sysctl.c | 70 ++++++
xen/include/asm-x86/cpufeature.h | 1 +
xen/include/asm-x86/domain.h | 2 +
xen/include/asm-x86/msr-index.h | 5 +
xen/include/asm-x86/pqos.h | 68 ++++++
xen/include/public/domctl.h | 11 +
xen/include/public/sysctl.h | 12 ++
xen/xsm/flask/hooks.c | 8 +
xen/xsm/flask/policy/access_vectors | 17 +-
29 files changed, 1044 insertions(+), 6 deletions(-)
create mode 100644 tools/libxl/libxl_pqos.c
create mode 100644 xen/arch/x86/pqos/Makefile
create mode 100644 xen/arch/x86/pqos/cqm.c
create mode 100644 xen/arch/x86/pqos/pqos.c
create mode 100644 xen/include/asm-x86/pqos.h
--
1.7.9.5
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |