Re: [RFC] Avoid dom0/HVM performance penalty from MSR access tightening

[Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>]

On 10/02/2022 17:27, Alex Olson wrote:
> I'm seeing strange performance issues under Xen on a Supermicro server with a 
> Xeon D-1541 CPU caused by an MSR-related commit.
>
> Commit 322ec7c89f6640ee2a99d1040b6f786cf04872cf 'x86/pv: disallow access to 
> unknown MSRs'
> surprisingly introduces a severe performance penality where dom0 has about 
> 1/8th
> the normal CPU performance. Even even when 'xenpm' is used to select the
> performance governor and operate the CPU at maximum frequency, actual CPU
> performance is still 1/2 of normal (as well as using 
> "cpufreq=xen,performance").
>
> The patch below fixes it but I don't fully understand why.
>
> Basically, when *reads* of MSR_IA32_THERM_CONTROL are blocked, dom0 and
> guests (pinned to other CPUs) see the performance issues.
>
> For benchmarking purposes, I built a small C program that runs a "for
> loop" 
> 4Billion iterations and timed its execution. In dom0, the
> performance issues
> also cause HVM guest startup time to go from 9-10
> seconds to almost 80 seconds.
>
> I assumed Xen was managing CPU frequency and thus blocking related MSR
> access by dom0 (or any other domain). However,  clearly something else
> is happening and I don't understand why.
>
> I initially attempted to copy the same logic as the write MSR case. This
> was effective at fixing the dom0 performance issue, but still left other
> domains running at 1/2 speed. Hence, the change below has no access control.
>
>
> If anyone has any insight as to what is really happening, I would be all ears
> as I am unsure if the change below is a proper solution.

Well that's especially entertaining...

So your patch edits pv/emul-priv-op.c#read_msr(), so is only changing
the behaviour for PV dom0.

What exactly is your small C program doing?


The change that that patch made was to turn a read which previously
succeeded into a #GP fault.

The read has already been bogus, even if they appeared to work before. 
When dom0 is scheduled around, it no longer knows which MSR it is
actually reading, so at the best, the data being read is racy as to
which CPU you're instantaneously scheduled on.


At a guess, something in Linux is doing something especially dumb when
given #GP and is falling into a tight loop of trying to read the MSR. 
Do you happen to know which of those two is the more dominating factor?

~Andrew