Re: [Xen-devel] Question about VPID during MOV-TO-CR3

[Tamas K Lengyel <tamas.lengyel@xxxxxxxxxxxx>]

On Wed, Sep 21, 2016 at 9:30 AM, Tamas K Lengyel
<tamas.lengyel@xxxxxxxxxxxx> wrote:
> On Wed, Sep 21, 2016 at 9:23 AM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>>>>> On 21.09.16 at 17:16, <tamas.lengyel@xxxxxxxxxxxx> wrote:
>>> On Wed, Sep 21, 2016 at 9:09 AM, Tamas K Lengyel
>>> <tamas.lengyel@xxxxxxxxxxxx> wrote:
>>>> On Wed, Sep 21, 2016 at 8:44 AM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>>>>>>>> On 21.09.16 at 16:18, <tamas.lengyel@xxxxxxxxxxxx> wrote:
>>>>>> On Wed, Sep 21, 2016 at 4:23 AM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>>>>>>>>>> On 20.09.16 at 19:29, <tamas.lengyel@xxxxxxxxxxxx> wrote:
>>>>>>>> I'm trying to figure out the design decision regarding the handling of
>>>>>>>> guest MOV-TO-CR3 operations and TLB flushes. AFAICT since support for
>>>>>>>> VPID has been added to Xen, every guest MOV-TO-CR3 flushes the TLB
>>>>>>>> (vmx_cr_access -> hvm_mov_to_cr -> hvm_set_cr3 -> paging_update_cr3 ->
>>>>>>>> hap_update_cr3 -> vmx_update_guest_cr -> hvm_asid_flush_vcpu). From a
>>>>>>>> TLB utilization point-of-view this seems to be rather wasteful.
>>>>>>>> Furthermore, it even breaks the guests' ability to take advantage of
>>>>>>>> PCID, as the TLB just guts flushed when a new process is scheduled.
>>>>>>>> Does anyone have an insight into what was the rationale behind this?
>>>>>>>
>>>>>>> Since you don't quote the specific commit(s), I would guess that
>>>>>>> this was mainly an attempt by the author(s) to keep things simple
>>>>>>> for themselves, i.e. not having to properly think through under
>>>>>>> which conditions less than a full TLB flush would suffice.
>>>>>>
>>>>>> The commit that added VPID and the TLB flush is
>>>>>> e2cf9bd6e055ea678da129b776f4521f6a0b50fe x86, vmx: Enable VPID
>>>>>> (Virtual Processor Identification). So this has been there as long as
>>>>>> Xen supported VPID. The only case where flushing the TLB on a guest
>>>>>> MOV-TO-CR3 that possibly would make sense to me is if we are running a
>>>>>> PV guest. But this is hvm/vmx, so why would we care about what the
>>>>>> guest does to its cr3 from a TLB standpoint?
>>>>>
>>>>> Are you forgetting that a move to CR3 needs to flush all non-global
>>>>> TLB entries? Or else, why do you think no flushing needs to happen
>>>>> at all?
>>>>>
>>>>
>>>> The guest can mark entries as global or non-global but it will have no
>>>> affect on VPID, every translation is still going to be tagged by VPID
>>>> when the translation was triggered in guest-context. So why does Xen
>>>> need to jump in flush the TLB when the guest OS likely already done
>>>> so?
>>
>> Likely? We can't base anything on likelihood (the more that no matter
>> what flushing may have been done before the CR3 write, further
>> flushing may be necessary and mustn't be skipped). We need to
>> provide architecturally correct behavior, and that includes the flushing
>> of non-global entries. This doesn't mean we need to flush anything
>> ourselves, but we have to make previously created non-global TLB
>> entries unavailable.
>
> What I'm saying is that the guest OS should be in charge of managing
> its own TLB when VPID is in use. Whether it does flush the TLB or not
> is not of our concern. If it's a sane OS it will likely flush when it
> needs to, but we should not be jumping in and doing it as we do right
> now. We are actually breaking the architectural behavior by forcing a
> flush, MOV-TO-CR3 doesn't by itself flush the TLB on real hardware.
> Also, there are no non-global TLB entries we need to flush as long as
> we are using VPID. Any translation used by Xen or by any other domain
> will have a different VPID, so there is no chance of stale TLB entries
> being an issue.
>

So reading through the Intel SDM the following bits are relevant here:

28.3.3.1
Operations that Invalidate Cached Mappings
The following operations invalidate cached mappings as indicated:
• Operations that architecturally invalidate entries in the TLBs or
paging-structure caches independent of VMX
operation (e.g., the INVLPG and INVPCID instructions) invalidate
linear mappings and combined mappings. 1
They are required to do so only for the current VPID (but, for
combined mappings, all EP4TAs). Linear
mappings for the current VPID are invalidated even if EPT is in use. 2
Combined mappings for the current
VPID are invalidated even if EPT is not in use.

To me this reads that the CPU will automatically handle the TLB
flushing for all operations that would normally do so when running
without a hypervisor, but only within the context of the VPID. While
it doesn't list MOV-TO-CR3 specifically, I'm sure it falls into this
same category regarding non-global TLB entries that would be flushed
by it. Thus, there is no need for the VMM to step in do anything in
this regard if my interpretation is correct.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel