Re: [RFC] Avoid dom0/HVM performance penalty from MSR access tightening

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Feb 23, 2022 at 09:38:56AM -0600, Alex Olson wrote:
> I appreciate your interest, apologies for not replying right away. I've been
> digging deeper to have a more meaningful resposne.
> 
> I had attempted to instrument the MSR reads, but only saw a small number reads
> being blocked by the code change. They appear to be the list below and the
> others seem fairly harmless:
> 
> 0x00000034    MSR_SMI_COUNT
> 0x0000019a    IA32_CLOCK_MODULATION/MSR_IA32_THERM_CONTROL MSR
> 0x000003f8    MSR_PKG_C3_RESIDENCY
> 0x000003f9    MSR_PKG_C6_RESIDENCY
> 0x000003fa    MSR_PKG_C7_RESIDENCY
> 0x00000606    MSR_RAPL_POWER_UNIT
> 0x0000060d    MSR_PKG_C2_RESIDENCY
> 0x00000611    MSR_PKG_ENERGY_STATUS
> 0x00000619    MSR_DRAM_ENERGY_STATUS
> 0x00000630    MSR_PKG_C8_RESIDENCY
> 0x00000631    MSR_PKG_C9_RESIDENCY
> 0x00000632    MSR_PKG_C10_RESIDENCY
> 0x00000639    MSR_PP0_ENERGY_STATUS
> 0x00000641    MSR_PP1_ENERGY_STATUS
> 
> As for my test program, it is just a crude loop compiled with "gcc -O3",
> normally takes about 10 seconds to execute:
> int main()
> {
>     for (volatile int i=1; i!=0; ++i){}
>     return 0;
> }
> 
> The relative changes in execution time of the test program and also that  HVM
> guest startup time (associated with the "qemu" process being busy) completely
> agreed.  I also observed the same changes under a PVH guest for the test
> program.
> 
> Thus, it seemed like the CPU was somehow operating a different frequency than
> expected, rather than faults consuming execution time.
> 
> -- (after a lot more investigation) --
> 
> Further instrumentation showed that the
> IA32_CLOCK_MODULATION/MSR_IA32_THERM_CONTROL MSR initially had value
> "0x10"  which appears to be invalid both in the Intel Software Developer's
> manual and what I think I'm seeing in the ACPI tables.
> 
> In dom0 Linux 5.2.38,  this value seems to have caused the
> acpi_processor_get_throttling_ptc() function to see an invalid result from
> acpi_get_throttling_state() and thus execute __acpi_processor_set_throttling()
> which wrote the MSR with a value of zero and had the side effect of disabling
> throttling (restoring normal performance).  (This all happened as the CPUs 
> were
> detected).
> 
> When the unknown MSR reads are blocked, the call to
> __acpi_processor_set_throttling() did not occur since the MSR read did not
> result in the invalid value -- thus the CPU remained in a throttling state.
> 
> So far, this seems to explain the dom0 performance issues I saw.
> 
> The domU observation was related... In some of my testing, dom0 was limited 
> (via
> Xen command-line) to a small number of cores so that the others could be
> dedicated to other domains.  When a domU VM was launched on the others (not 
> used
> by dom0), its MSR remained at the original value resulting in low performance
> since dom0 hadn't a chance to rewrite it...   Thus, I saw different domU
> behavior based on the number of cores allocated to dom0.
> 
> 
> -- summary --
> 
> In desparation, I ended up resetting BIOS settings to defaults and 
> mysteriously
> this issue doesn't occur anymore.  Not sure what could have gone wrong before 
> as
> the original settings were not far from defaults.  It seems my issues stemmed
> from the server's BIOS setting the throttling MSR to an invalid value but it 
> had
> illuminated some unusual behaviors under Xen...
> 
> It seems to me there are a few findings useful to the Xen developers from
> venturing down this rabbithole:
> 
> 1) For conditions in which MSR registers are writeable from PV guests (such as
> dom0),  they should probably be readable well, looks like 
> MSR_IA32_THERM_CONTROL
> is currently one of a small number of "unreadable" but writeable
> MSRs.  Otherwise seemingly valid read-(check/modify)-write operations will
> behave incorrectly under Xen.

So it's one of those MSRs that's only writable when dom0 has it's
vCPUs pinned. We could allow dom0 to read from it in that case (that's
an oversight of the MSR handling rework), but it would seem better to
just remove access to it altogether: it's bogus to allow dom0 to play
with the MSR in the first place IMO, and it won't really solve issues
like the one reported here unless dom0 vCPUs == pCPUs and all are
pinned, so that dom0 can fix the MSR in all CPUs.

Thanks, Roger.