Xen project Mailing List

I've recently discovered that if you attempt to use introspection to capture CR3 changes with the new KPTI enabled kernels, the guest dies shortly after the start of introspection with failed VM entry due to invalid guest state.

I believe the invalid state here is the high bit being set in CR3 - while this is how one indicates that PCID should not invalidate the various page table caches, introspection leads to this being set in the VMCS, which appears to be wrong.

With the XenServer 4.7.1 code base (which is my working code base at the moment), I have not found a way around this, as the vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set the CR3 value, and vm_event_register_write_resume only allows inhibiting the write, not writing a modified value.

I've attempted several ways to work around this with a livepatch, and have not (yet) been successful.

Masking at the top of hvm_set_cr3 allows the guest to continue, but appears to do the wrong thing with regards to the guest (tasks begin dying quickly from invalid opcode errors).

In any case, Andrew mentions that this appears to still be an issue in staging, so this likely needs addressing. At this point in time, I believe guests with KPTI enabled cannot be introspected if that introspection involves capturing CR3 changes.

Please let me know if you need any more details on this issue!

-Bit

(XEN) [19458.318035] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0).
(XEN) [19458.318042] ************* VMCS Area **************
(XEN) [19458.318050] *** Guest State ***
(XEN) [19458.318056] CR0: actual=0x000000008005003b, shadow=0x0000000080050033, gh_mask=ffffffffffffffff
(XEN) [19458.318062] CR4: actual=0x0000000000362670, shadow=0x0000000000360670, gh_mask=ffffffffffffffff
(XEN) [19458.318069] CR3 = 0x800000001ded7080
(XEN) [19458.318076] PDPTE0 = 0x0000000000020000 PDPTE1 = 0x000006f800150018
(XEN) [19458.318082] PDPTE2 = 0x0000000000000000 PDPTE3 = 0x000006f800150018
(XEN) [19458.318089] RSP = 0xffff880015b87f50 (0xffff880015b87f50) RIP = 0xffffffff81845857 (0xffffffff81845857)
(XEN) [19458.318095] RFLAGS=0x00000082 (0x00000082) DR7 = 0x0000000000000400
(XEN) [19458.318101] Sysenter RSP=0000000000000000 CS:RIP=0010:ffffffff8184a220
(XEN) [19458.318105]        sel attr limit   base
(XEN) [19458.318112]   CS: 0010 0a09b ffffffff 0000000000000000
(XEN) [19458.318119]   DS: 0000 1c000 ffffffff 0000000000000000
(XEN) [19458.318126]   SS: 0018 0c093 ffffffff 0000000000000000
(XEN) [19458.318133]   ES: 0000 1c000 ffffffff 0000000000000000
(XEN) [19458.318140]   FS: 0000 1c000 ffffffff 00007fde038ba700
(XEN) [19458.318147]   GS: 0000 1c000 ffffffff ffff88001ba00000
(XEN) [19458.318152] GDTR:            0000007f ffff88001ba0c000
(XEN) [19458.318158] LDTR: 0000 1c000 ffffffff 0000000000000000
(XEN) [19458.318164] IDTR:            00000fff ffffffffff574000
(XEN) [19458.318169]   TR: 0040 0008b 00002087 ffff88001ba048c0
(XEN) [19458.318175] EFER = 0x0000000000000000 PAT = 0x0407010600070106
(XEN) [19458.318179] PreemptionTimer = 0x00000000 SM Base = 0x00000000
(XEN) [19458.318185] DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000
(XEN) [19458.318233] PerfGlobCtl = 0x0000000000000000 BndCfgS = 0x0000000000000000
(XEN) [19458.318297] Interruptibility = 00000000 ActivityState = 00000000
(XEN) [19458.318324] *** Host State ***
(XEN) [19458.318329] RIP = 0xffff82d0801ee100 (vmx_asm_vmexit_handler) RSP = 0xffff8300bfcfff90
(XEN) [19458.318333] CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
(XEN) [19458.318335] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff82d08035e780
(XEN) [19458.318337] GDTBase=ffff82d0802d9000 IDTBase=ffff82d080357ce0
(XEN) [19458.318339] CR0=000000008005003b CR3=000000010f001000 CR4=00000000003526e0
(XEN) [19458.318341] Sysenter RSP=ffff8300bfcfffc0 CS:RIP=e008:ffff82d08022bb30
(XEN) [19458.318343] EFER = 0x0000000000000000 PAT = 0x0000050100070406
(XEN) [19458.318344] *** Control State ***
(XEN) [19458.318347] PinBased=0000003f CPUBased=b6a0e5fa SecondaryExec=001014ea
(XEN) [19458.318348] EntryControls=000153ff ExitControls=008fefff
(XEN) [19458.318350] ExceptionBitmap=00060082 PFECmask=00000000 PFECmatch=00000000
(XEN) [19458.318352] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) [19458.318353] VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
(XEN) [19458.318355]         reason=80000021 qualification=0000000000000000
(XEN) [19458.318357] IDTVectoring: info=00000000 errcode=00000000
(XEN) [19458.318359] TSC Offset = 0xffffd23bbd8772ac TSC Multiplier = 0x0000000000000000
(XEN) [19458.318361] TPR Threshold = 0x00 PostedIntrVec = 0x00
(XEN) [19458.318365] EPT pointer = 0x000000010ee9501e EPTP index = 0x0000
(XEN) [19458.318396] PLE Gap=00000080 Window=00001000
(XEN) [19458.318402] Virtual processor ID = 0xccd3 VMfunc controls = 0000000000000000
(XEN) [19458.318406] **************************************
(XEN) [19458.318412] domain_crash called from vmx_vmexit_handler+0x4ab/0x19f5
(XEN) [19458.318417] Domain 15 (vcpu#0) crashed on cpu#0:
(XEN) [19458.318443] ----[ Xen-4.7.1-1.0 x86_64 debug=n Not tainted ]----
(XEN) [19458.318448] CPU:    0
(XEN) [19458.318453] RIP:    0010:[<ffffffff81845857>]
(XEN) [19458.318458] RFLAGS: 0000000000000082   CONTEXT: hvm guest (d15v0)
(XEN) [19458.318466] rax: 800000001ded7080   rbx: 0000000000000000   rcx: 00007fde033ce730
(XEN) [19458.318470] rdx: 00000000000000fa   rsi: 0000000000000002   rdi: 00007ffd8ee85250
(XEN) [19458.318484] rbp: 00007ffd8ee85410   rsp: ffff880015b87f50   r8: 0000000000000000
(XEN) [19458.318498] r9: 0000000000000017   r10: 0000000000000000   r11: 0000000000000246
(XEN) [19458.318502] r12: 00007ffd8ee85250   r13: 0000000000000000   r14: 0000000000000004
(XEN) [19458.318525] r15: 000055a8503b3828   cr0: 0000000080050033   cr4: 0000000000360670
(XEN) [19458.318538] cr3: 800000001ded7080   cr2: 00007ffef290a090
(XEN) [19458.318552] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0018   cs: 0010

[Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection