Xen project Mailing List

Re: [Xen-users] Domu windows 2012 crash.

To: Hans van Kranenburg <hans@xxxxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxxx

From: Johnny Ström <johnny.strom@xxxxxxxxxxxxxxxxx>

Date: Wed, 18 Mar 2020 17:32:31 +0200

Cc: kim Ståhl <kim.stahl@xxxxxxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

Delivery-date: Wed, 18 Mar 2020 15:52:02 +0000

List-id: Xen user discussion <xen-users.lists.xenproject.org>

On 3/14/20 2:54 PM, Hans van Kranenburg wrote:

Hi Johnny,

On 2/20/20 10:10 AM, johnny Strom wrote:

After the latest updates/security updates for debian 9 and debian 10,
so will windows 2012 crash if more than 4096 memory is available to domu.

I am able to reproduce this with a Windows server essentials 2019 HVM domU.

An example from xl dmesg:

(XEN) Failed to shatter gfn 105245: -12
(XEN) d75v1 EPT violation 0x19c (--x/rw-) gpa 0x00000105245760 mfn
0x285245 type 0
(XEN) d75v1 Walking EPT tables for GFN 105245:
(XEN) d75v1  epte 9c000004105f9007
(XEN) d75v1  epte 9c000002800000f3
(XEN) d75v1  --- GLA 0x7ff98b40d760
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 75 (vcpu#1) crashed on cpu#4:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    0033:[<00007ff98b40d760>]
(XEN) RFLAGS: 0000000000010216   CONTEXT: hvm guest (d75v1)
(XEN) rax: 0000000000001212   rbx: 000000c714d9da58   rcx: 0000023500001590
(XEN) rdx: 000000c700000001   rsi: 000000c714d9da18   rdi: 000000c714d9db20
(XEN) rbp: 000000c714d9d950   rsp: 000000c714d9d918   r8:  0000023500001470
(XEN) r9:  00000235000014f0   r10: 00007ff99c5c0923   r11: 000000c714d9d970
(XEN) r12: 0000000000000000   r13: 000000c714d9d9d0   r14: 000000c714d9da58
(XEN) r15: 0000000000000006   cr0: 0000000080050031   cr4: 0000000000060678
(XEN) cr3: 00000001d9458002   cr2: 00007ff98b6fa048
(XEN) fsb: 0000000000000000   gsb: 000000c714e9e000   gss: ffffcd015dc40000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033

So it's in debian 9 update: xen (4.8.5.final+shim4.10.4-1+deb9u12)
and in debian 10 update:  xen (4.11.3+24-g14b62ab3e5-1) that is causing
the crashes.

For 4.11, can you please try the workaround to enable "executable EPT
superpages" again?

It's the ept=exec-sp hypervisor setting, and can also be set without
rebooting using:

    xl set-parameters ept=exec-sp

Hello

I have been running test with "xl set-parameters ept=exec-sp" and

windows 2012 R2.

And it dose not crash anymore. I used memory = 5120 that did make it crash within 2 minutes before. So xl set-parameters ept=exec-sp seems to be an workaround for now. Best regards Johnny


(and no-exec-sp to go back to the default again)

This seems to be related to XSA-304, whose patches were included in the
latest security uploads in Debian.

https://xenbits.xen.org/xsa/advisory-304.html

I do not know if it's Windows or Xen at fault here, but at least this
workaround stops the domU from crashing all the time.

Hans

memory = 5120 = Domu crashes see below

memory = 4096 = Domu works ok.


I am reporting it here since perhaps other distributions might
carry the same patches and have the same problems.





Debian 10 Xen hypervisor running windows 2012 as domu.


(XEN) Failed to shatter gfn e660e: -12
(XEN) d3v1 EPT violation 0x19c (--x/rw-) gpa 0x000000e660e075 mfn
0xceae0e type 0
(XEN) d3v1 Walking EPT tables for GFN e660e:
(XEN) d3v1  epte 9c00000ec17dd007
(XEN) d3v1  epte 9c0000078a576007
(XEN) d3v1  epte 9c00000ceae000f3
(XEN) d3v1  --- GLA 0x7fca357d075
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 3 (vcpu#1) crashed on cpu#14:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    14
(XEN) RIP:    0033:[<000007fca357d075>]
(XEN) RFLAGS: 0000000000010283   CONTEXT: hvm guest (d3v1)
(XEN) rax: 000007fca3e7feb0   rbx: 0000000023c34600   rcx: 000000de005e44f0
(XEN) rdx: 000000de6cd67210   rsi: 000000de005e44f0   rdi: 000000de005e0358
(XEN) rbp: 000000de005e4028   rsp: 000000de6f68cec8   r8: 0000000000000004
(XEN) r9:  0000000000000001   r10: 0000000000000002   r11: 000007fca3571ef8
(XEN) r12: 0000000000000008   r13: 0000000000000058   r14: 000000de6f68d7e0
(XEN) r15: 0000000000000002   cr0: 0000000080050031   cr4: 00000000000006f8
(XEN) cr3: 00000000b5640000   cr2: 000000de6f7a049c
(XEN) fsb: 00000000ff7e6000   gsb: 000007f5ff7e4000   gss: 000007f7bb927000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) Failed to shatter gfn 15f9b: -12
(XEN) d9v0 EPT violation 0x19c (--x/rw-) gpa 0x00000015f9b000 mfn
0xcb719b type 0
(XEN) d9v0 Walking EPT tables for GFN 15f9b:
(XEN) d9v0  epte 9c00000ee8476007
(XEN) d9v0  epte 9c0000078ccc2007
(XEN) d9v0  epte 9c00000cb70000f3
(XEN) d9v0  --- GLA 0x7f63b039000
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 9 (vcpu#0) crashed on cpu#12:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    12
(XEN) RIP:    0033:[<000007f63b038ffd>]
(XEN) RFLAGS: 0000000000010246   CONTEXT: hvm guest (d9v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 000000000000000e
(XEN) rdx: 000007040ab01ff0   rsi: 000000f5a749a3e0   rdi: 0000000000000000
(XEN) rbp: 000000f5a808f570   rsp: 000000f5a808f470   r8: 0000000000000010
(XEN) r9:  0000000000000002   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000004   r13: 000007f63b0fe3a8   r14: 0000000000000000
(XEN) r15: 0000000000000001   cr0: 0000000080050031   cr4: 00000000000006f8
(XEN) cr3: 0000000052280000   cr2: 000007f63b039000
(XEN) fsb: 0000000039f8a000   gsb: 000007f639f88000   gss: 000007f705f8a000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) Failed to shatter gfn cd7c9: -12
(XEN) d15v0 EPT violation 0x19c (--x/rw-) gpa 0x000000cd7c9060 mfn
0xdc37c9 type 0
(XEN) d15v0 Walking EPT tables for GFN cd7c9:
(XEN) d15v0  epte 9c00000f1e239007
(XEN) d15v0  epte 9c0000078cc13007
(XEN) d15v0  epte 9c00000dc36000f3
(XEN) d15v0  --- GLA 0x7ff7a5b2060
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 15 (vcpu#0) crashed on cpu#14:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    14
(XEN) RIP:    0033:[<000007ff7a5b2060>]
(XEN) RFLAGS: 0000000000010206   CONTEXT: hvm guest (d15v0)
(XEN) rax: 0000000000000002   rbx: 0000000000000002   rcx: 000007ff7a5d3c60
(XEN) rdx: 000007ff7a5cc0d0   rsi: 0000000080070002   rdi: 00000042424bb320
(XEN) rbp: 0000004241cff869   rsp: 0000004241cff7c8   r8: 000000000000066a
(XEN) r9:  000007ff7a5b8e08   r10: 0000000000000000   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000004240e5a478   r14: 000000000000000e
(XEN) r15: 00000042424d3370   cr0: 0000000080050031   cr4: 00000000000006f8
(XEN) cr3: 0000000112a40000   cr2: 000007ff7a5b2060
(XEN) fsb: 0000000096c7e000   gsb: 000007f796c7c000   gss: 000007f796c7c000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) Failed to shatter gfn ecddb: -12
(XEN) d16v0 EPT violation 0x19c (--x/rw-) gpa 0x000000ecddb3b6 mfn
0xc841db type 0
(XEN) d16v0 Walking EPT tables for GFN ecddb:
(XEN) d16v0  epte 9c00000ed02af007
(XEN) d16v0  epte 9c0000078daf2007
(XEN) d16v0  epte 9c00000c840000f3
(XEN) d16v0  --- GLA 0x7ff962313b6
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 16 (vcpu#0) crashed on cpu#17:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    17
(XEN) RIP:    0033:[<000007ff962313b6>]
(XEN) RFLAGS: 0000000000010246   CONTEXT: hvm guest (d16v0)
(XEN) rax: 0000000000000003   rbx: 0000000004895fd0   rcx: 0000000004895fd0
(XEN) rdx: 0000000004895fd0   rsi: 0000000007464de0   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: 00000000096df660   r8: 0000000004aa7630
(XEN) r9:  0000000000000000   r10: 000007ff95f7c660   r11: 0000000000000000
(XEN) r12: 000000007ffe03c0   r13: 0000000000000000   r14: 000000007ffe03b0
(XEN) r15: 000000007ffe0008   cr0: 0000000080050031   cr4: 00000000000006f8
(XEN) cr3: 000000007eac0000   cr2: 000000ed1d62a000
(XEN) fsb: 00000000dc9be000   gsb: 000007f7dc9bc000   gss: fffff802f933d000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033





Debian 9 XEN hypervisor running windows 2012 as domu.


(XEN) Failed to shatter gfn daf71: -12
(XEN) d19v0 EPT violation 0x39c (--x/rw-) gpa 0x000000daf710d0 mfn
0x38a5971 type 0
(XEN) d19v0 Walking EPT tables for GFN daf71:
(XEN) d19v0  epte 9c00004019911107
(XEN) d19v0  epte 9c0000201a67f107
(XEN) Failed to shatter gfn ca3dc: -12
(XEN) d19v0  epte 9c000038a5800bf3
(XEN) d19v0  --- GLA 0x7f8fc0130d0
(XEN) domain_crash called from vmx.c:3385
(XEN) d19v1 EPT violation 0x39c (--x/rw-) gpa 0x000000ca3dc430 mfn
0x38b65dc type 0
(XEN) Domain 19 (vcpu#0) crashed on cpu#29:
(XEN) d19v1 Walking EPT tables for GFN ca3dc:
(XEN) ----[ Xen-4.8.5  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    29
(XEN) RIP:    0033:[<000007f8fc0130d0>]
(XEN) RFLAGS: 0000000000010246   CONTEXT: hvm guest (d19v0)
(XEN) rax: 000007f8fc0130d0   rbx: 0000000000000001   rcx: 00000040673d4430
(XEN) rdx: 0000004002a8e3b0   rsi: 0000000000000001   rdi: 00000040673d42d0
(XEN) rbp: 0000004002a8e930   rsp: 0000004002a8e348   r8: 0000000000000001
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 000007f8fb5d3020
(XEN) r12: 0000004002a8f510   r13: 0000004002a8eed0   r14: 00000040676c0e20
(XEN) r15: 0000004067450768   cr0: 0000000080050031   cr4: 00000000001506f8
(XEN) cr3: 0000000122000000   cr2: 000007f8fc0130d0
(XEN) fsb: 00000000ff3d2000   gsb: 000007f5ff3d0000   gss: 000007f7eb928000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) d19v1  epte 9c00004019911107
(XEN) d19v1  epte 9c0000201a67f107
(XEN) domain_crash called from vmx.c:3385
(XEN) Failed to shatter gfn 3e8f3: -12
(XEN) printk: 2 messages suppressed.
(XEN) d20v0 EPT violation 0x39c (--x/rw-) gpa 0x0000003e8f3010 mfn
0x3901ef3 type 0
(XEN) d20v0 Walking EPT tables for GFN 3e8f3:
(XEN) d20v0  epte 9c000040128ff107
(XEN) d20v0  epte 9c00002059ee9107
(XEN) d20v0  epte 9c00003901e00bf3
(XEN) d20v0  --- GLA 0x7f8780b5010
(XEN) domain_crash called from vmx.c:3385
(XEN) Domain 20 (vcpu#0) crashed on cpu#35:
(XEN) ----[ Xen-4.8.5  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    35
(XEN) RIP:    0033:[<000007f8780b5010>]
(XEN) RFLAGS: 0000000000010283   CONTEXT: hvm guest (d20v0)
(XEN) rax: 000007f8780b5010   rbx: 000007f8780b6e00   rcx: 0000000004b50de0
(XEN) rdx: 000000000992d6f0   rsi: 0000000000000000   rdi: 000007f878097d80
(XEN) rbp: 0000000004b50de0   rsp: 000000000992d6c8   r8: 0000000000000010
(XEN) r9:  0000000000000002   r10: 000007f8780b6e00   r11: 0000000000000001
(XEN) r12: 000007f89bbb3ea8   r13: 000007f89babeb28   r14: 000007f8780b6bb0
(XEN) r15: 000000000992d930   cr0: 0000000080050031   cr4: 00000000001506f8
(XEN) cr3: 00000000e8400000   cr2: 0000005730908000
(XEN) fsb: 000000004a060000   gsb: 000007f64a05e000   gss: 000007f79c1c2000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) Failed to shatter gfn db554: -12
(XEN) d22v1 EPT violation 0x39c (--x/rw-) gpa 0x000000db554f70 mfn
0x3c65354 type 0
(XEN) d22v1 Walking EPT tables for GFN db554:
(XEN) d22v1  epte 9c00004006b1d107
(XEN) d22v1  epte 9c0000201d16e107
(XEN) d22v1  epte 9c00003c65200bf3
(XEN) d22v1  --- GLA 0x7fd2cb99f70
(XEN) domain_crash called from vmx.c:3385
(XEN) Domain 22 (vcpu#1) crashed on cpu#31:
(XEN) ----[ Xen-4.8.5  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    31
(XEN) RIP:    0033:[<000007fd2cb99f70>]
(XEN) RFLAGS: 0000000000010206   CONTEXT: hvm guest (d22v1)
(XEN) rax: 000007fd2cb99f70   rbx: 000000473498eec8   rcx: 00000047349935b0
(XEN) rdx: 000000473498cbe0   rsi: 000000000000000d   rdi: 00000047349935b0
(XEN) rbp: 00000046338cc040   rsp: 00000046338cbfc8   r8: 00000046338cbd98
(XEN) r9:  fffffffeff1edb14   r10: 0000000000000000   r11: 0000000000000001
(XEN) r12: 000007fd376385b8   r13: 0000004633c3d5e0   r14: 000000473498cdb0
(XEN) r15: 0000004859c65478   cr0: 0000000080050031   cr4: 00000000001506f8
(XEN) cr3: 00000000cc000000   cr2: 000007fd2cb99f70
(XEN) fsb: 0000000002065000   gsb: 000007f702063000   gss: 000007f702063000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033
(XEN) Failed to shatter gfn c662c: -12
(XEN) d23v0 EPT violation 0x39c (--x/rw-) gpa 0x000000c662c8b8 mfn
0x3dba02c type 0
(XEN) d23v0 Walking EPT tables for GFN c662c:
(XEN) d23v0  epte 9c000040159e4107
(XEN) d23v0  epte 9c0000201cfc6107
(XEN) d23v0  epte 9c00003dba000bf3
(XEN) d23v0  --- GLA 0x7fa57bd58b8
(XEN) domain_crash called from vmx.c:3385
(XEN) Domain 23 (vcpu#0) crashed on cpu#34:
(XEN) ----[ Xen-4.8.5  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    34
(XEN) RIP:    0033:[<000007fa57bd58b8>]
(XEN) RFLAGS: 0000000000010202   CONTEXT: hvm guest (d23v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 000000a99e82eaa0
(XEN) rdx: 0000000000000000   rsi: 000000000000000c   rdi: 000000a99e82f1c0
(XEN) rbp: 000000000002b02a   rsp: 000000a99e82ea58   r8: 0000000000000003
(XEN) r9:  000007fa57b2e2f0   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 000000a99e82f1c0   r14: 000000a99e82eaf0
(XEN) r15: 000000000000000c   cr0: 0000000080050031   cr4: 00000000001506f8
(XEN) cr3: 0000000109700000   cr2: 000007fa6c62d8bb
(XEN) fsb: 00000000e80d2000   gsb: 000007f6e80d0000   gss: 000007f6e80d0000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users

_______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-users

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.