Re: [Xen-devel] MSR load lists on Harpertown

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 18.12.18 at 10:56, <andrew.cooper3@xxxxxxxxxx> wrote:
> On 18/12/2018 02:17, Tian, Kevin wrote:
>>> From: Andrew Cooper [mailto:andrew.cooper3@xxxxxxxxxx]
>>> Sent: Monday, December 17, 2018 10:21 PM
>>>
>>> On 17/12/2018 13:09, Andrew Cooper wrote:
>>>> On 17/12/2018 02:39, Tian, Kevin wrote:
>>>>>>>> After some investigation, it turns out that after vmentry, while the
>>>>>>>> load list has the value 0xd01 (NXE, LMA, LME, SCE), the value loaded
>>>>>>>> into hardware is 0xd00 (NXE, LMA, LME).
>>>>>>>>
>>>>>>>> I.e. when an MSR load list is used for EFER, we resume the guest with
>>>>>>>> SCE cleared.  This is rather terminal for 64bit guests, as
>>>>>>>> syscall/sysret instructions take a #UD fault.
>>>>>>>>
>>>>>>>> I can't see anything relevant in the Specification Update for this
>>>>>>>> processor.
>>>>>>>>
>>>>>>>> I've confirmed that by not using a load list, the current value in EFER
>>>>>>>> is preserved once the vmentry is complete, and by disabling the EFER
>>>>>>>> intercept, I can re-set SCE in non-root context and have syscall/sysret
>>>>>>>> work correctly.
>>>>>>>>
>>>>>>>> However, given this behaviour, I can't think of any way to context
>>>>>>>> switch NX properly, and leave 64bit guests in a working state.
>>>>>>>>
>>>>>>>> Do you have any suggestions?
>>>>>>>>
>>>>>> I'm checking internally whether it's a known issue.
>>>>> from feedbacks that I collected so far, no one is aware of this issue.
>>>>>
>>>>>> btw did you try upgrading to a newer microcode?
>>>>>>
>>>>> while I'm approaching more channels, does it work by directly
>>>>> WRMSR to EFER just before VMENTRY for above special case (
>>>>> thus remove EFER from MSR load/save list), if ucode update
>>>>> also fails? there is just a small window where NX might be wrong
>>>>> setting for Xen, but it might be OK for that carefully-baked code
>>>>> snippet?
>>>> Apologies for the delay.  I was travelling last week.
>>>>
>>>> We cannot load the full guest's EFER value in Xen context.  If the guest
>>>> has NX disabled, the next stack access in Xen will fault because the NX
>>>> bit becomes reserved when EFER.NXE is clear.
>>>>
>>>> As for the more general case of loading the guests EFER value (ignoring
>>>> NXE), we already know that works, because it is how Xen functioned for a
>>>> decade.
>>>>
>>>> I see that the latest production microcode on otcshare is slightly newer
>>>> than exists in the microcode_ctl package.  I'll give it a spin.
>>> The behaviour is still the same, even with the latest microcode:
>>>
>>> (XEN) microcode: CPU0 updated from revision 0x60f to 0x612, date =
>>> 2015-08-02
>>>
>> Just confirmed that there is no erratum which could explain above 
>> behavior.
>>
>> I see several options here:
>>
>> a) load guest EFER in Xen context, just before vmentry. make sure no stack 
>> access between load and VM enter. but it sounds messy to add such sku
>> specific workaround in that common path (especially just for an old one);
>>
>> b) leave guest following Xen NXE setting on Harpertown. doing so bears
>> with limitation as what fd32dcfe tries to fix, but it's better than syscall
>> error in 64bit guest. also not clean since some ad-hoc logic that is removed
>> by fd32dcfe may have to be added back;
>>
>> c) leave above issue unfixed. Harpertown is pretty old...
>>
>> d) more debug whether above is caused by other software bug
> 
> Option a) is not possible.  While we could in principle turn the line of
> pop gpr's into movs and avoid the stack accesses,

How would this avoid stack accesses?

> we will triple fault if an NMI/#MC hits.
> 
> As for option d), I'm quite certain at this point that it is a
> hardware/microcode issue, rather than a software issue.
> 
> I'd prefer to leave Harpertown cores working than simply dismissing them
> at this point, because they are still useful for compatibility testing. 
> TBH,  I was expecting option b) to be the outcome.

So did I, with ...

> I don't suppose there is any chance of this being published as an
> erratum which I can reference in the source code?

... such a reference.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel