Re: [Xen-devel] Xen host crash

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 29/08/13 17:38, Rushikesh Jadhav wrote:

This is a spinlock deadlock, resulting in the NMI watchdog timing out and killing the host.  Do you have Stack and register dump for PCPU6 ?

~Andrew

Hi Andrew, here is the stack trace and register dump for PCPU6 & PCPU7

PCPU6 host state:

RIP:    e008:[<ffff828c8010e310>]

RFLAGS: 0000000000000002

rax: 0000000000000004   rbx: 0000000000000001   rcx: ffff828c803629cc

rdx: ffff828c8036286c   rsi: ffff828c803628dc   rdi: 00000000ffffffff

rbp: 0000000000000082   rsp: ffff83247fd88e10   r8:  0000000000000001

r9:  0000000000000001   r10: 00000000fffffffc   r11: 0000000000000001

r12: 0000000000000001   r13: ffff832270af39a0   r14: 0000000000000002

r15: 0000000000000009

cr0: 0000000080050033   cr4: 00000000000026f0

cr3: 000000205a12e000   cr2: fffff880005c5000

ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008

current: DOM185 VCPU6 (ffff830047ee8000)

stack context: DOM185 VCPU6 (ffff830047ee8000)

idle VCPU: ffff83007ea5e000

Stack at 0xffff83247fd88e10: 

 ffff83247fd88e00:                                     8010eeb7 ffff828c 801fb1d0 ffff828c

 ffff83247fd88e20: 80127b36 ffff828c 00000028 00000030 7fd88f18 ffff8324 7fd88e48 ffff8324

 ffff83247fd88e40: 00000001 00000000 00000002 00000000 00000002 00000000 801f423d ffff828c

 ffff83247fd88e60: 00000000 00000000 801f45b3 ffff828c 00000000 00000000 00000096 00000000

 ffff83247fd88e80: 8011b7da ffff828c 000000e5 00000000 00000000 00000000 8019b4ab ffff828c

 ffff83247fd88ea0: 7fd8ff20 ffff8324 80100000 ffff828c 8015eb75 ffff828c 7fd88f58 ffff8324

 ffff83247fd88ec0: 7fd8ff28 ffff8324 7fd88f58 ffff8324 00000002 00000000 7fd8ff28 ffff8324

 ffff83247fd88ee0: 7fd88f58 ffff8324 00000002 00000000 8015eeba ffff828c 7fd8ff28 ffff8324

 ffff83247fd88f00: 7fd8ff28 ffff8324 7fd88f58 ffff8324 801567a1 ffff828c 00000000 00000000

 ffff83247fd88f20: 00000006 00000000 7fd88f58 ffff8324 8015f37f ffff828c 00000000 00000000

 ffff83247fd88f40: 339e8000 ffff8322 7fd8ff28 ffff8324 801d6877 ffff828c 00000009 00000000

 ffff83247fd88f60: 00000002 00000000 70af39a0 ffff8322 00000001 00000000 7fd8ff28 ffff8324

 ffff83247fd88f80: 339e8000 ffff8322 00ff00ff 00ff00ff 0000ffff 0000ffff 339e8018 ffff8322

 ffff83247fd88fa0: 00000002 00000000 00000000 00000000 0000000f 00000000 47ee8000 ffff8300

 ffff83247fd88fc0: 00366807 00000000 60e72e90 ffff8319 00000000 00000002 8011ab02 ffff828c

 ffff83247fd88fe0: 0000e008 00000000 00000246 00000000 7fd8fd70 ffff8324 00000000 00000000

Code:

 da e8 df 91 01 00 e9 19 fd ff ff 90 90 90 90 90 90 90 90 90 90 <4c> 89 3f 4c 89 77 08 4c 89 6f 10 

Call Trace:

 [ffff828c8010e310] dump_domains+0x4d0

  ffff828c8010eeb7  kexec_crash+0x57

  ffff828c80127b36  panic+0x136

  ffff828c8011b7da  __print_symbol+0x8a

  ffff828c8019b4ab  vmx_asm_vmexit_handler+0x6b

  ffff828c80100000  __per_cpu_shift+0x800ffff4

  ffff828c8015eb75  show_stack+0x155

  ffff828c8015eeba  fatal_trap+0x6a

  ffff828c801567a1  nmi_watchdog_tick+0x131

  ffff828c8015f37f  do_nmi+0xbf

  ffff828c801d6877  handle_ist_exception+0x52

  ffff828c8011ab02  _spin_lock+0x12

  PCPU6 guest state:

DOMAIN185 VCPU3

RIP:    0000:[<fffff800016caee0>]

RFLAGS: 0000000000010206

rax: 0000000000000000   rbx: ffffffffffffffff   rcx: fffffa6002998000

rdx: 0000000000000100   rsi: fffff6fd30014d00   rdi: 0000000000000010

rbp: 0000000000000010   rsp: fffffa6001bc6c48   r8:  0000000000000000

r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000

r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000

r15: 0000000000000000

cr0: 0000000080050033   cr4: 00000000000026f0

cr3: 000000205a12e000   cr2: fffff880005c5000

ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

VCPU pause flags: 0 arch flags 0x1

current on PCPU6

struct vcpu at ffff830047ee8000

Stack unavailable.

Ok.

The problematic spinlock is at address 0xffff831960e72e90, which is sadly a dynamically allocated one so cant be traced back to a symbol using the symbol table.

Having said that, you are using Xen 3.4 which is ages out of date, and in fact, probably using XenServer 5.6SP2 (so shouldn't be using xen-devel anyway).

I suggest you upgrade to something less ancient.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel