Re: [Xen-devel] XenDom0/FreeBSD: guest crash when nested virtualization is used

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 24/07/2019 19:02, Oleg Ginzburg wrote:

Hello maillist,

I use XEN on the FreeBSD platform. Everything worked fine until I
needed to use nested virtualization (for testing purposes).

After some communication with Roger Pau Monné, maintainer of XEN port
in FreeBSD ( https://www.freshports.org/emulators/xen-kernel )  it was
suggested that this might be a common Xen problem and not related to
FreeBSD.

So nested virt under Xen is in an "almost completely broken and unusable" state.

Your mileage won't vary very much.  It will tend to be 0.

(d2) Booting from Hard Disk...
(d2) Booting from 0000:7c00
(XEN) d2v0 VMLAUNCH error: 0x7
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000080050033, shadow=0x0000000060000010, gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=fffffffffffff871
(XEN) CR3 = 0x00000000feffc000
(XEN) RSP = 0x0000000000000000 (0x0000000000000000)  RIP = 0x000000000000fff0 (0x000000000000fff0)
(XEN) RFLAGS=0x00010002 (0x00010002)  DR7 = 0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN)        sel  attr  limit   base
(XEN)   CS: f000 0009b 0000ffff 00000000ffff0000
(XEN)   DS: 0000 00093 0000ffff 0000000000000000
(XEN)   SS: 0000 00093 0000ffff 0000000000000000
(XEN)   ES: 0000 00093 0000ffff 0000000000000000
(XEN)   FS: 0000 00093 0000ffff 0000000000000000
(XEN)   GS: 0000 00093 0000ffff 0000000000000000
(XEN) GDTR:            0000ffff 0000000000000000
(XEN) LDTR: 0000 00082 0000ffff 0000000000000000
(XEN) IDTR:            0000ffff 0000000000000000
(XEN)   TR: 0000 0008b 0000ffff 0000000000000000
(XEN) EFER(VMCS) = 0x0000000000000000  PAT = 0x0000050100070406
(XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000
(XEN) DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000
(XEN) Interruptibility = 00000000  ActivityState = 00000000
(XEN) InterruptStatus = 0000
(XEN) *** Host State ***
(XEN) RIP = 0xffff82d08030f8b0 (vmac.c#vmx_asm_vmexit_handler)  RSP = 0xffff8320259bff70

Something is definitely strange in your build of Xen.  vmac.c doesn't contain the vmexit handler.

(XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff8320259c2c00
(XEN) GDTBase=ffff8320259b2000 IDTBase=ffff8320259b6000
(XEN) CR0=0000000080050033 CR3=000000201bc45000 CR4=00000000003526e0
(XEN) Sysenter RSP=ffff8320259bffa0 CS:RIP=e008:ffff82d080354420
(XEN) EFER = 0x0000000000000d01  PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6a1edfa SecondaryExec=000214eb
(XEN) EntryControls=000011ff ExitControls=002fefff
(XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000
(XEN) VMEntry: intr_info=8000030d errcode=00000000 ilen=00000000
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000005
(XEN)         reason=00000030 qualification=0000000000000181
(XEN) IDTVectoring: info=80000b0d errcode=0000f000

The IDTVectoring and VMEntry fields look like we intercepted a page fault, but are trying to re-inject it without an error code, which is possibly what hardware is complaining about.

Are you able to instrument the virtual vmentry/exit code to see if this is the case?

~Andrew

(XEN) TSC Offset = 0xfff9d10bc60f4ad6  TSC Multiplier = 0x0000000000000000
(XEN) TPR Threshold = 0x00  PostedIntrVec = 0xf4
(XEN) EPT pointer = 0x000000207dd3b01e  EPTP index = 0x0000
(XEN) PLE Gap=00000080 Window=00001000
(XEN) Virtual processor ID = 0x3540 VMfunc controls = 0000000000000000
(XEN) domain_crash called from vmcs.c:1777
(XEN) Domain 2 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.12.0  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    f000:[<000000000000fff0>]
(XEN) RFLAGS: 0000000000010002   CONTEXT: hvm guest (d2v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) rdx: 00000000000206c1   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000002050
(XEN) cr3: 00000000feffc000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: ffff90e74ec00000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: f000

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel