Re: [Xen-devel] Xen 4.7.0 boot PANIC on kernel 4.7.0-4 + UEFI ?

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 27/07/16 00:32, lists@xxxxxxxxxxxx wrote:
> I'm running Xen-4.7.0_08-452 + linux kernel 4.7.0-4.g89a2ada-default on 
> X86_64 UEFI hardware.
>
> If I boot without Xen hypervisor enabled it boots fine.
>
> If I boot with Xen enabled it PANICs:
>       
>       (XEN) [2016-07-26 22:05:33] Hardware Dom0 crashed: rebooting machine in 
> 5 seconds.

The Dom0 kernel crashed.  As a result, Xen tried to reboot.

>       (XEN) [2016-07-26 22:05:33] APIC error on CPU0: 40(00)
>       (XEN) [2016-07-26 22:05:38] ----[ Xen-4.7.0_08-452  x86_64  debug=n  
> Tainted:    C ]----
>       (XEN) [2016-07-26 22:05:38] CPU:    0
>       (XEN) [2016-07-26 22:05:38] RIP:    e008:[<000000009e7463c6>] 
> 000000009e7463c6
>       (XEN) [2016-07-26 22:05:38] RFLAGS: 0000000000010202   CONTEXT: 
> hypervisor (d0v0)
>       (XEN) [2016-07-26 22:05:38] rax: 0000000000000003   rbx: 
> 0000000000000000   rcx: 0000000000000000
>       (XEN) [2016-07-26 22:05:38] rdx: 000000009e7467a0   rsi: 
> 0000000000000000   rdi: 0000000000000000
>       (XEN) [2016-07-26 22:05:38] rbp: 0000000000000000   rsp: 
> ffff83008ce27d78   r8:  ffff83008ce27db8
>       (XEN) [2016-07-26 22:05:38] r9:  ffff83008ce27da8   r10: 
> 0000000000000000   r11: 0000000000000000
>       (XEN) [2016-07-26 22:05:38] r12: 0000000000000000   r13: 
> 0000000000000cf9   r14: 0000000000000065
>       (XEN) [2016-07-26 22:05:38] r15: ffff830000000000   cr0: 
> 0000000080050033   cr4: 00000000001526e0
>       (XEN) [2016-07-26 22:05:38] cr3: 000000084b4f6000   cr2: 
> 0000000000000018
>       (XEN) [2016-07-26 22:05:38] ds: 0000   es: 0000   fs: 0000   gs: 0000   
> ss: 0000   cs: e008
>       (XEN) [2016-07-26 22:05:38] Xen code around <000000009e7463c6> 
> (000000009e7463c6):
>       (XEN) [2016-07-26 22:05:38]  0f 48 8b 44 24 40 8b ce <ff> 50 08 3b d8 
> 0f 4c d8 48 ff c7 48 3b 7c 24 30

This disassembles to

callq  *0x8(%rax)

and %rax looks like an implausible value for a function pointer.  This
particular issue is definitely an EFI firmware issue.

>       (XEN) [2016-07-26 22:05:38] Xen stack trace from rsp=ffff83008ce27d78:
>       (XEN) [2016-07-26 22:05:38]    ffff83084b4b51c0 0000000000000000 
> 0000000000000000 ffff82d0801670f6
>       (XEN) [2016-07-26 22:05:38]    ffff83008ce27db0 0000000000000000 
> 00000000001526e0 0000000000000206
>       (XEN) [2016-07-26 22:05:38]    0000000000000003 0000000841e06000 
> 0000000000000000 000000009efe42f6
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000000 
> efff000000000000 ffff82d0807fe000
>       (XEN) [2016-07-26 22:05:38]    000000084b4f6000 ffff82d08022f94a 
> 0000000841e06000 0000000000000000
>       (XEN) [2016-07-26 22:05:38]    0000000000000007 000000000000e008 
> 0000000000000296 0000000000000000
>       (XEN) [2016-07-26 22:05:38]    00000000fffffffe ffff82d08018bcc8 
> 0000000000000000 0000138800000008
>       (XEN) [2016-07-26 22:05:38]    000083008ce27ea8 0000000000000000 
> ffff83008ce27eb8 0000000000000003
>       (XEN) [2016-07-26 22:05:38]    0000000000000003 ffff83084b4b5000 
> ffff83084b4b51c0 0000000000000000
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 ffff82d08012bf0d 
> 0000000000000003 ffff82d08012bfaf
>       (XEN) [2016-07-26 22:05:38]    ffffffff81e03f28 ffff82d080105871 
> 0000000000000000 ffff83008ce27fff
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000000 
> ffffffff81e03f28 ffff82d080105978
>       (XEN) [2016-07-26 22:05:38]    ffff830092826000 ffff82d080197f8e 
> 0000000000000001 ffff82d08022cc55
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000000 
> ffffffff81e03f28 0000000000000000
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000000 
> 0000000000000067 0000000000007ff0
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000200 
> 0000000000000000 0000000100000000
>       (XEN) [2016-07-26 22:05:38]    000000016f141000 ffffffff81efb2c0 
> 000000016f141000 0000010e00000004
>       (XEN) [2016-07-26 22:05:38]    ffffffff81f6374c 000000000000e033 
> 0000000000000246 ffffffff81e03e50
>       (XEN) [2016-07-26 22:05:38]    000000000000e02b 0000000000000000 
> 0000000000000000 0000000000000000
>       (XEN) [2016-07-26 22:05:38]    0000000000000000 0000000000000000 
> ffff830092826000 0000000000000000
>       (XEN) [2016-07-26 22:05:38] Xen call trace:
>       (XEN) [2016-07-26 22:05:38]    [<000000009e7463c6>] 000000009e7463c6
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d0801670f6>] 
> i387.c#_vcpu_save_fpu+0x86/0x190
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d08022f94a>] 
> efi_reset_system+0x3a/0x60
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d08018bcc8>] 
> machine_restart+0x208/0x2d0
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d08012bf0d>] 
> shutdown.c#maybe_reboot+0x3d/0x40
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d08012bfaf>] 
> hwdom_shutdown+0x9f/0xf0
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d080105871>] 
> domain_shutdown+0xf1/0x100
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d080105978>] 
> __domain_crash_synchronous+0x18/0x30
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d080197f8e>] 
> asm_domain_crash_synchronous+0x3e/0x40
>       (XEN) [2016-07-26 22:05:38]    [<ffff82d08022cc55>] 
> entry.o#handle_exception_saved+0x9b/0xa4
>       (XEN) [2016-07-26 22:05:38] 
>       (XEN) [2016-07-26 22:05:38] 
>       (XEN) [2016-07-26 22:05:38] ****************************************
>       (XEN) [2016-07-26 22:05:38] Panic on CPU 0:
>       (XEN) [2016-07-26 22:05:38] GENERAL PROTECTION FAULT
>       (XEN) [2016-07-26 22:05:38] [error_code=0000]
>       (XEN) [2016-07-26 22:05:38] ****************************************
>       (XEN) [2016-07-26 22:05:38] 
>       (XEN) [2016-07-26 22:05:38] Reboot in five seconds...
>
> This server used to be running OK as recently as last week with this config.  
> Some recent upgrade - kernel, qemu, xen, etc - may have broken something.  
> Most notable was the the upgrade to kernel 4.6.3 -> 4.7.0.

I presume you mean an upgrade of the dom0 Linux kernel from 4.6.3 to 4.7.0?

> What other debug info can help figure out this specific problem?

This is first a Linux crash, followed by bad knock-on behaviour.

For the knockon behaviour, does Linux 4.6.3 encounter the same reboot
crash with Xen 4.7.0?

For the Linux crash, can you boot Linux with "earlyprintk=xen" and see
if that provides more help as to what went wrong?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel