Re: [Xen-devel] [XTF PATCH] XSA-186: Work around suspected Broadwell TLB erratum

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 28.10.16 at 15:02, <andrew.cooper3@xxxxxxxxxx> wrote:
> On 28/10/16 13:49, Jan Beulich wrote:
>>>>> On 28.10.16 at 14:39, <andrew.cooper3@xxxxxxxxxx> wrote:
>>> On 28/10/16 13:03, Jan Beulich wrote:
>>>>>>> On 28.10.16 at 12:36, <andrew.cooper3@xxxxxxxxxx> wrote:
>>>>> Signed-off-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>>>> Reviewed-by: Jan Beulich <jbeulich@xxxxxxxx>
>>>> (Maybe you want to drop the ...
>>>>
>>>>> --- a/tests/xsa-186/main.c
>>>>> +++ b/tests/xsa-186/main.c
>>>>> @@ -144,6 +144,29 @@ void test_main(void)
>>>>>      memcpy(stub, insn_buf_start, insn_buf_end - insn_buf_start);
>>>>>  
>>>>>      /*
>>>>> +     * Work around suspected Broadwell TLB Erratum
>>>>> +     *
>>>>> +     * Occasionally, this test failes with:
>>>>> +     *
>>>>> +     *   --- Xen Test Framework ---
>>>>> +     *   Environment: HVM 64bit (Long mode 4 levels)
>>>>> +     *   XSA-186 PoC
>>>>> +     *   ******************************
>>>>> +     *   PANIC: Unhandled exception at 0008:fffffffffffffffa
>>>>> +     *   Vec 14 #PF[-I-sr-] %cr2 fffffffffffffffa
>>>>> +     *   ******************************
>>>>> +     *
>>>>> +     * on Broadwell hardware.  The mapping is definitely present as the
>>>>> +     * memcpy() has already succeeded.  Inserting an invlpg resolves the
>>>>> +     * issue, sugguesting that there is a race conditon between dTLB/iTLB
>>>> ... stray u which slipped into "suggesting".)
>>>>
>>>> Btw - would you mind trying something else: Instead of the INVLPG,
>>>> put a CPUID or some other serializing instruction in here. ISTR that
>>>> for self modifying code this is required, i.e. the CPU could have been
>>>> fetching instructions ahead of the memcpy(), and nothing would be
>>>> there to force it to drop what it has already executed speculatively,
>>>> including the exception token.
>>> That is an interesting point, but still doesn't explain the symptoms. 
>>> If the icache wasn't flushed, we might get junk instructions and a #UD/#GP.
>> No. As the processor speculates the call, it won't be able to fetch
>> the target instruction and hence would insert an exception token
>> into the queue. There would be junk instruction bytes only if there
>> was a prior mapping for that page, but aiui a mapping for that
>> address gets established exactly once.
> 
> Re-reading Intel Vol 3 11.6 "Self-Modifying Code".
> 
> * A write to a memory location in a code segment that is currently
> cached in the processor causes the associated cache line (or lines) to
> be invalidated. This check is based on the physical address of the
> instruction.  If the write affects a prefetched instruction, the
> prefetch queue is invalidated. This latter check is based on the linear
> address of the instruction.
> 
> * Systems software, such as a debugger, that might possibly modify an
> instruction using a different linear address than that used to fetch the
> instruction, will execute a serializing operation, such as a CPUID
> instruction, before the modified instruction is executed, which will
> automatically resynchronize the instruction cache and prefetch queue.
> 
> As this is a single vcpu using a single flat address space, the memcpy()
> should invalidate any speculative execution which has already happened.

And still you describe only the case where there would need to be
a prior mapping - without one there simply is no physical address to
compare against. What if speculative execution ends up performing
the call to stub before you finish populating page tables? That would
also explain the error code. But aiui this might still be an erratum, as
it might be a memory ordering violation (depending on whether insn
fetches count as reads here, which would then have to observe
earlier writes, albeit the addresses of the two are different).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel