[Xen-devel] patch for restricted vPMU modes

[Brendan Gregg <bgregg@xxxxxxxxxxx>]

G'Day,

The vpmu feature of Xen is incredibly useful for performance analysis, however, it's currently all counters or nothing. In secure environments, there can be hesitation to enable access to all PMCs (there are hundreds of them). I've included a prototype patch that introduces two new restricted vpmu modes:

vpmu=ipc: As the most restricted minimum set. This enables cycles, reference cycles, and instructions only. This is enough to calculate instructions per cycle (IPC).

vpm=arch: This enables the 7 pre-defined architectural events as listed in cpuid, and in Table 18-1 of the Intel software developer's manual, vol 3B.

There can be a third mode added later on, with a larger set (including micro-ops PMCs).

I've included the short patch below for Xen 4.6.0, which provides these modes (it also fixes a minor copy-and-paste error with core2_get_fixed_pmc_count(), which I believe was accessing the wrong register). I am not a veteran Xen programmer, so please feel free to edit or rewrite this patch. In case this email messes it up, it's also on:Âhttps://github.com/brendangregg/Misc/blob/master/xen/xen-4.6.0-vpmu-filter.diff

I've shown testing of four modes (off, on, ipc, arch) here:Âhttps://gist.github.com/brendangregg/b7318c0f49bf906dc8df

For example, here is Linux perf running in a PVHVM guest with the new vpmu=ipc mode:

root@vm0hvm:~# perf stat -d ./noploop

ÂPerformance counter stats for './noploop':

   Â1511.326375 task-clock (msec)     #  Â0.999 CPUs utilized     Â

        24 context-switches     Â#  Â0.016 K/sec         Â

        Â0 cpu-migrations      Â#  Â0.000 K/sec         Â

       Â113 page-faults        #  Â0.075 K/sec         Â

  Â5,028,638,883 cycles          Â#  Â3.327 GHz          Â

        Â0 stalled-cycles-frontend  #  Â0.00% frontend cycles idle ÂÂ

        Â0 stalled-cycles-backend  Â#  Â0.00% backend Âcycles idle ÂÂ

  20,043,427,933 instructions       Â#  Â3.99 Âinsns per cycle    Â

        Â0 branches         Â#  Â0.000 K/sec         Â

        Â0 branch-misses       #  Â0.00% of all branches    Â

        Â0 L1-dcache-loads      #  Â0.000 K/sec         Â

        Â0 L1-dcache-load-misses   #  Â0.00% of all L1-dcache hits Â

        Â0 LLC-loads         #  Â0.000 K/sec         Â

 Â<not supported> LLC-load-misses:HG  Â

Note that IPC is shown ("insns per cycle"), but other counters are not.

---patch---

diff -ur xen-4.6.0-clean/docs/misc/xen-command-line.markdown xen-4.6.0-brendan/docs/misc/xen-command-line.markdown

--- xen-4.6.0-clean/docs/misc/xen-command-line.markdown 2015-10-05 07:33:39.000000000 -0700

+++ xen-4.6.0-brendan/docs/misc/xen-command-line.markdown 2015-11-20 15:29:05.663781176 -0800

@@ -1444,7 +1444,7 @@

Âflushes on VM entry and exit, increasing performance.

Â

Â### vpmu

-> `= ( bts )`

+> `= ( <boolean> | bts | ipc | arch )`

Â

Â> Default: `off`

Â

@@ -1460,6 +1460,15 @@

ÂIf 'vpmu=bts' is specified the virtualisation of the Branch Trace Store (BTS)

Âfeature is switched on on Intel processors supporting this feature.

Â

+vpmu=ipc enables performance monitoring, but restricts the counters to the

+most minimum set possible: instructions, cycles, and reference cycles. These

+can be used to calculate instructions per cycle (IPC).

+

+vpmu=arch enables performance monitoring, but restricts the counters to the

+pre-defined architectural events only. These are exposed by cpuid, and listed

+in Table 18-1 from the Intel 64 and IA-32 Architectures Software Developer's

+Manual, Volume 3B, System Programming Guide, Part 2.

+

ÂNote that if **watchdog** option is also specified vpmu will be turned off.

Â

Â*Warning:*

diff -ur xen-4.6.0-clean/xen/arch/x86/cpu/vpmu.c xen-4.6.0-brendan/xen/arch/x86/cpu/vpmu.c

--- xen-4.6.0-clean/xen/arch/x86/cpu/vpmu.c 2015-10-05 07:33:39.000000000 -0700

+++ xen-4.6.0-brendan/xen/arch/x86/cpu/vpmu.c 2015-11-20 15:29:50.847781176 -0800

@@ -43,9 +43,11 @@

ÂCHECK_pmu_params;

Â

Â/*

- * "vpmu" : Â Â vpmu generally enabled

- * "vpmu=off" : vpmu generally disabled

- * "vpmu=bts" : vpmu enabled and Intel BTS feature switched on.

+ * "vpmu" : Â Â vpmu generally enabled (all counters)

+ * "vpmu=off" Â: vpmu generally disabled

+ * "vpmu=bts" Â: vpmu enabled and Intel BTS feature switched on.

+ * "vpmu=ipc" Â: vpmu enabled for IPC counters only (most restrictive)

+ * "vpmu=arch" : vpmu enabled for predef arch counters only (restrictive)

 */

Âstatic unsigned int __read_mostly opt_vpmu_enabled;

Âunsigned int __read_mostly vpmu_mode = XENPMU_MODE_OFF;

@@ -67,6 +69,10 @@

  Âdefault:

    Âif ( !strcmp(s, "bts") )

      Âvpmu_features |= XENPMU_FEATURE_INTEL_BTS;

+ Â Â Â Âelse if ( !strcmp(s, "ipc") )

+ Â Â Â Â Â Âvpmu_features |= XENPMU_FEATURE_IPC_ONLY;

+ Â Â Â Âelse if ( !strcmp(s, "arch") )

+ Â Â Â Â Â Âvpmu_features |= XENPMU_FEATURE_ARCH_ONLY;

    Âelse if ( *s )

    Â{

      Âprintk("VPMU: unknown flag: %s - vpmu disabled!\n", s);

diff -ur xen-4.6.0-clean/xen/arch/x86/cpu/vpmu_intel.c xen-4.6.0-brendan/xen/arch/x86/cpu/vpmu_intel.c

--- xen-4.6.0-clean/xen/arch/x86/cpu/vpmu_intel.c 2015-10-05 07:33:39.000000000 -0700

+++ xen-4.6.0-brendan/xen/arch/x86/cpu/vpmu_intel.c 2015-11-20 15:29:42.571781176 -0800

@@ -166,10 +166,10 @@

 */

Âstatic int core2_get_fixed_pmc_count(void)

Â{

- Â Âu32 eax;

+ Â Âu32 edx;

Â

- Â Âeax = cpuid_eax(0xa);

- Â Âreturn MASK_EXTR(eax, PMU_FIXED_NR_MASK);

+ Â Âedx = cpuid_edx(0xa);

+ Â Âreturn MASK_EXTR(edx, PMU_FIXED_NR_MASK);

Â}

Â

Â/* edx bits 5-12: Bit width of fixed-function performance counters Â*/

@@ -652,12 +652,52 @@

    Âtmp = msr - MSR_P6_EVNTSEL(0);

    Âif ( tmp >= 0 && tmp < arch_pmc_cnt )

    Â{

+ Â Â Â Â Â Âint umaskevent, blocked = 0;

      Âstruct xen_pmu_cntr_pair *xen_pmu_cntr_pair =

        Âvpmu_reg_pointer(core2_vpmu_cxt, arch_counters);

Â

      Âif ( msr_content & ARCH_CTRL_MASK )

        Âreturn -EINVAL;

Â

+ Â Â Â Â Â Â/* PMC filters */

+ Â Â Â Â Â Âumaskevent = msr_content & MSR_IA32_CMT_EVTSEL_UE_MASK;

+ Â Â Â Â Â Âif ( vpmu_features & XENPMU_FEATURE_IPC_ONLY ||

+ Â Â Â Â Â Â Â Â vpmu_features & XENPMU_FEATURE_ARCH_ONLY )

+ Â Â Â Â Â Â{

+ Â Â Â Â Â Â Â Âblocked = 1;

+ Â Â Â Â Â Â Â Âswitch ( umaskevent )

+ Â Â Â Â Â Â Â Â{

+ Â Â Â Â Â Â Â Â/*

+ Â Â Â Â Â Â Â Â * See Table 18-1 from the Intel 64 and IA-32 Architectures Software

+ Â Â Â Â Â Â Â Â * Developer's Manual, Volume 3B, System Programming Guide, Part 2.

+ Â Â Â Â Â Â Â Â */

+ Â Â Â Â Â Â Â Âcase 0x003c: /* unhalted core cycles */

+ Â Â Â Â Â Â Â Âcase 0x013c: /* unhalted ref cycles */

+ Â Â Â Â Â Â Â Âcase 0x00c0: /* instruction retired */

+ Â Â Â Â Â Â Â Â Â Âblocked = 0;

+ Â Â Â Â Â Â Â Âdefault:

+ Â Â Â Â Â Â Â Â Â Âbreak;

+ Â Â Â Â Â Â Â Â}

+ Â Â Â Â Â Â}

+

+ Â Â Â Â Â Âif ( vpmu_features & XENPMU_FEATURE_ARCH_ONLY )

+ Â Â Â Â Â Â{

+ Â Â Â Â Â Â Â Â/* additional counters beyond IPC only; blocked already set */

+ Â Â Â Â Â Â Â Âswitch ( umaskevent )

+ Â Â Â Â Â Â Â Â{

+ Â Â Â Â Â Â Â Âcase 0x4f2e: /* LLC reference */

+ Â Â Â Â Â Â Â Âcase 0x412e: /* LLC misses */

+ Â Â Â Â Â Â Â Âcase 0x00c4: /* branch instruction retired */

+ Â Â Â Â Â Â Â Âcase 0x00c5: /* branch */

+ Â Â Â Â Â Â Â Â Â Âblocked = 0;

+ Â Â Â Â Â Â Â Âdefault:

+ Â Â Â Â Â Â Â Â Â Âbreak;

+ Â Â Â Â Â Â Â }

+ Â Â Â Â Â Â}

+

+ Â Â Â Â Â Âif ( blocked )

+ Â Â Â Â Â Â Â Âreturn -EINVAL;

+

      Âif ( has_hvm_container_vcpu(v) )

        Âvmx_read_guest_msr(MSR_CORE_PERF_GLOBAL_CTRL,

                  &core2_vpmu_cxt->global_ctrl);

diff -ur xen-4.6.0-clean/xen/include/public/pmu.h xen-4.6.0-brendan/xen/include/public/pmu.h

--- xen-4.6.0-clean/xen/include/public/pmu.h 2015-10-05 07:33:39.000000000 -0700

+++ xen-4.6.0-brendan/xen/include/public/pmu.h 2015-11-20 15:30:08.887781176 -0800

@@ -84,9 +84,17 @@

Â

Â/*

 * PMU features:

- * - XENPMU_FEATURE_INTEL_BTS: Intel BTS support (ignored on AMD)

+ * - XENPMU_FEATURE_INTEL_BTS: ÂIntel BTS support (ignored on AMD)

+ * - XENPMU_FEATURE_IPC_ONLY: Â Restrict PMC to the most minimum set possible.

+ * Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂInstructions, cycles, and ref cycles. Can be

+ * Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âused to calculate instructions-per-cycle (IPC).

+ * - XENPMU_FEATURE_ARCH_ONLY: ÂRestrict PMCs to the Intel pre-definedÂ

+ * Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âarchitecteral events exposed by cpuid and

+ * Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âlisted in Table 18-1 of the developer's manual.

 */

-#define XENPMU_FEATURE_INTEL_BTS Â1

+#define XENPMU_FEATURE_INTEL_BTS Â(1<<0)

+#define XENPMU_FEATURE_IPC_ONLY Â (1<<1)

+#define XENPMU_FEATURE_ARCH_ONLY Â(1<<2)

Â

Â/*

 * Shared PMU data between hypervisor and PV(H) domains.

---patch---

Brendan

--

Brendan Gregg, Senior Performance Architect, Netflix

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel