Re: [Xen-devel] [PATCH v3 7/7] xen/x86: use PCID feature

[Juergen Gross <jgross@xxxxxxxx>]

On 26/03/18 12:13, Jan Beulich wrote:
>>>> On 26.03.18 at 10:55, <jgross@xxxxxxxx> wrote:
>> On 26/03/18 10:28, Jan Beulich wrote:
>>>>>> On 26.03.18 at 08:49, <jgross@xxxxxxxx> wrote:
>>>> On 23/03/18 16:58, Jan Beulich wrote:
>>>>>>>> On 23.03.18 at 15:11, <jgross@xxxxxxxx> wrote:
>>>>>> On 23/03/18 14:46, Jan Beulich wrote:
>>>>>>> So in the end the question is: Why not use just two PCIDs, and
>>>>>>> allow global pages just like we do now, with the added benefit
>>>>>>> that we no longer need to flush Xen's global TLB entries just
>>>>>>> because we want to get rid of PV guest user ones.
>>>>>>
>>>>>> I can't see how that would work without either needing some more TLB
>>>>>> flushes in order to prevent stale TLB entries or loosing the Meltdown
>>>>>> mitigation.
>>>>>>
>>>>>> Which %cr3/PCID combination should be used in hypervisor, guest kernel
>>>>>> and guest user mode?
>>>>>
>>>>> Xen would run with PCID 0 (and full Xen mappings) at all times
>>>>> (except early entry and late exit code of course). The guest would
>>>>> run with PCID 1 (and minimal Xen mappings) at all times. The switch
>>>>> of PCID eliminates the need for flushes on the way out and back in.
>>>>
>>>> You still need the kernel page tables flushed when switching to user
>>>> mode, right?
>>>
>>> Of course.
>>>
>>>>>> Which pages would be global?
>>>>>
>>>>> Use of global pages would continue to be as today: Xen has some,
>>>>> and guest user mode has some. Of course it is quite possible that
>>>>> the use of global pages with a single guest PCID is still worse than
>>>>> no global pages with two guest PCIDs, but that's a separate step
>>>>> to take (and measure) imo.
>>>>
>>>> But global pages of Xen would either make it vulnerable with regard to
>>>> Meltdown or you need a TLB flush again when switching between Xen and
>>>> guest making all the PCID stuff moot.
>>>
>>> No - the guest would run with PCID 1 active, and global Xen TLB
>>> entries would exist for PCID 0 only.
>>
>> Uuh, global pages are accessible via all PCIDs. That's why they are
>> called global...
> 
> Okay, in that case all of what I've said in this regard was rubbish.
> (I don't, btw, think that this is the only sensible interpretation of
> "global" - it could as well mean protected from ordinary flushes
> within the given PCID realm.)

That's the reason I gave the reference to the SDM. It clearly states
that TLB entries with the global bit set don't have to match the current
PCID for being regarded to match.

> 
>>>> - 2 PCIDs
>>>> - no TLB flushes needed when switching between Xen and guest
>>>> - when switching from guest kernel to guest user the kernel pages must
>>>>   be flushed from TLB
>>>> - flushing of single guest user pages needs 2 changes of %cr3 and 2
>>>>   INVLPGs, switch code must be mapped to guest page tables
>>>> - flushing of complete TLB via 1 INVPCID
>>>>
>>>> So the advantage of the 2 PCID solution are the single TLB entries for
>>>> guest user pages compared to 2 entries for guest user pages accessed by
>>>> the guest kernel or Xen.
>>>>
>>>> The disadvantage are the flushed guest kernel pages when executing user
>>>> code, the more complicated single user page flushing and the dynamical
>>>> Xen global bit handling.
>>>
>>> Right. In order to make forward progress here I think we should
>>> shelve the discussion on the 2-PCID alternative for now. What I'd
>>> like to ask for as a change to your current approach is to use
>>> PCID 0 for Xen rather than running Xen with PCIDs 2 or 3 when
>>> PCIDs are enabled, and (implicitly) with PCID 0 when they're
>>> disabled. Or alternatively don't use PCID 0 at all when PCIDs are
>>> enabled. I'm simply worried of us overlooking a case where PCID
>>> 0 TLB entries may be left in place (when switching between PCIDs
>>> enabled and PCIDs disabled) when they should have been flushed,
>>> opening back up a Meltdown-like attack window.
>> The reason I didn't use PCID 0 for running Xen was to use a few
>> INVPCID calls as possible for single page invalidation and still
>> covering the cases for PCID on while XPTI off and including PCID 0.
> 
> How would the number of INVPCIDs needed differ depending on
> the actual PCID values used?

See answer below.

>> I can change the scheme to use different values for guest PCIDs
>> with XPTI on, of course. Are you fine with:
>>
>> - XPTI off: PCID 0 = kernel, PCID 1 = user
>> - XPTI on:  PCID 0 = kernel/Xen, PCID 1 = user/Xen,
>>             PCID 2 = kernel/guest, PCID 3 = user/guest
> 
> Yes, that would fit the first variant I've described. I take it you
> prefer not to avoid PCID 0 altogether when PCIDs are enabled -
> is there a particular reason?

Yes. As written in the comment in flush_area_local() I can't be sure
whether the current address space is that of a domain with XPTI
enabled (the idle domain could be "current"). So I need to always
flush with PCID 0 and with the possible PCID values for a XPTI domain.
When using PCID 0 for XPTI as well I'll need 4 INVPCIDs, while when
avoiding it I'd need 5 (at least when current == idle).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel