Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

[Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>]

On 01/25/2018 12:31 AM, Bitweasil . wrote:
> I've recently discovered that if you attempt to use introspection to
> capture CR3 changes with the new KPTI enabled kernels, the guest dies
> shortly after the start of introspection with failed VM entry due to
> invalid guest state.
> 
> I believe the invalid state here is the high bit being set in CR3 -
> while this is how one indicates that PCID should not invalidate the
> various page table caches, introspection leads to this being set in the
> VMCS, which appears to be wrong.
> 
> With the XenServer 4.7.1 code base (which is my working code base at the
> moment), I have not found a way around this, as the
> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
> the CR3 value, and vm_event_register_write_resume only allows inhibiting
> the write, not writing a modified value.
> 
> I've attempted several ways to work around this with a livepatch, and
> have not (yet) been successful.
> 
> Masking at the top of hvm_set_cr3 allows the guest to continue, but
> appears to do the wrong thing with regards to the guest (tasks begin
> dying quickly from invalid opcode errors).
> 
> In any case, Andrew mentions that this appears to still be an issue in
> staging, so this likely needs addressing.  At this point in time, I
> believe guests with KPTI enabled cannot be introspected if that
> introspection involves capturing CR3 changes.
> 
> Please let me know if you need any more details on this issue!

I've managed to reproduce the problem and this patch has fixed it for me
with Xen staging:

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index db282b5..7be962e 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -2320,6 +2320,9 @@ int hvm_set_cr3(unsigned long value, bool_t may_defer)
         }
     }

+    if ( hvm_pcid_enabled(v) )
+        value &= ((1ull << 63) - 1);
+
     if ( hvm_paging_enabled(v) && !paging_mode_hap(v->domain) &&
          (value != v->arch.hvm_vcpu.guest_cr[3]) )
     {

Would you mind giving this a spin and confirm it also solves your problems?


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel