Re: [Xen-devel] [PATCH v3 7/7] xen/x86: use PCID feature

[Juergen Gross <jgross@xxxxxxxx>]

On 26/03/18 10:28, Jan Beulich wrote:
>>>> On 26.03.18 at 08:49, <jgross@xxxxxxxx> wrote:
>> On 23/03/18 16:58, Jan Beulich wrote:
>>>>>> On 23.03.18 at 15:11, <jgross@xxxxxxxx> wrote:
>>>> On 23/03/18 14:46, Jan Beulich wrote:
>>>>> So in the end the question is: Why not use just two PCIDs, and
>>>>> allow global pages just like we do now, with the added benefit
>>>>> that we no longer need to flush Xen's global TLB entries just
>>>>> because we want to get rid of PV guest user ones.
>>>>
>>>> I can't see how that would work without either needing some more TLB
>>>> flushes in order to prevent stale TLB entries or loosing the Meltdown
>>>> mitigation.
>>>>
>>>> Which %cr3/PCID combination should be used in hypervisor, guest kernel
>>>> and guest user mode?
>>>
>>> Xen would run with PCID 0 (and full Xen mappings) at all times
>>> (except early entry and late exit code of course). The guest would
>>> run with PCID 1 (and minimal Xen mappings) at all times. The switch
>>> of PCID eliminates the need for flushes on the way out and back in.
>>
>> You still need the kernel page tables flushed when switching to user
>> mode, right?
> 
> Of course.
> 
>>>> Which pages would be global?
>>>
>>> Use of global pages would continue to be as today: Xen has some,
>>> and guest user mode has some. Of course it is quite possible that
>>> the use of global pages with a single guest PCID is still worse than
>>> no global pages with two guest PCIDs, but that's a separate step
>>> to take (and measure) imo.
>>
>> But global pages of Xen would either make it vulnerable with regard to
>> Meltdown or you need a TLB flush again when switching between Xen and
>> guest making all the PCID stuff moot.
> 
> No - the guest would run with PCID 1 active, and global Xen TLB
> entries would exist for PCID 0 only.

Uuh, global pages are accessible via all PCIDs. That's why they are
called global...

> 
>> So lets compare the possibilities:
>>
>> My approach:
>> - no global pages
>> - 4 different PCIDs
>> - no TLB flushes needed when switching between Xen and guest
>> - no TLB flushes needed when switching between guest user and kernel
>> - flushing of single pages (guest or Xen) rather simple (4 INVPCIDs)
>> - flushing of complete TLB via 1 INVPCID
>>
>> 2 PCIDs (Xen and guest), keeping guest user pages as global pages
>> - Xen can't use global pages - global bit must be handled dynamically
>>   for Xen pages (or do we want to drop global pages e.g. for AMD, too?
> 
> As per above - I don't see why Xen couldn't use global pages.
> The option of using them is part of why I'm wondering whether
> this might be worth looking into.

See chapter 4.10.2.4 SDM Vol. 3

> 
>> - 2 PCIDs
>> - no TLB flushes needed when switching between Xen and guest
>> - when switching from guest kernel to guest user the kernel pages must
>>   be flushed from TLB
>> - flushing of single guest user pages needs 2 changes of %cr3 and 2
>>   INVLPGs, switch code must be mapped to guest page tables
>> - flushing of complete TLB via 1 INVPCID
>>
>> So the advantage of the 2 PCID solution are the single TLB entries for
>> guest user pages compared to 2 entries for guest user pages accessed by
>> the guest kernel or Xen.
>>
>> The disadvantage are the flushed guest kernel pages when executing user
>> code, the more complicated single user page flushing and the dynamical
>> Xen global bit handling.
> 
> Right. In order to make forward progress here I think we should
> shelve the discussion on the 2-PCID alternative for now. What I'd
> like to ask for as a change to your current approach is to use
> PCID 0 for Xen rather than running Xen with PCIDs 2 or 3 when
> PCIDs are enabled, and (implicitly) with PCID 0 when they're
> disabled. Or alternatively don't use PCID 0 at all when PCIDs are
> enabled. I'm simply worried of us overlooking a case where PCID
> 0 TLB entries may be left in place (when switching between PCIDs
> enabled and PCIDs disabled) when they should have been flushed,
> opening back up a Meltdown-like attack window.
The reason I didn't use PCID 0 for running Xen was to use a few
INVPCID calls as possible for single page invalidation and still
covering the cases for PCID on while XPTI off and including PCID 0.

I can change the scheme to use different values for guest PCIDs
with XPTI on, of course. Are you fine with:

- XPTI off: PCID 0 = kernel, PCID 1 = user
- XPTI on:  PCID 0 = kernel/Xen, PCID 1 = user/Xen,
            PCID 2 = kernel/guest, PCID 3 = user/guest

Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel