Re: [Xen-devel] cpuidle and un-eoid interrupts at the local apic

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 09/08/13 22:40, Andrew Cooper wrote:

On 09/08/13 22:27, Thimo E. wrote:

Next crash occured, debugging output included.

One Remark: Over the last days (besides many linux PV guests) 1 Windows Guest (with PV drivers) was running, today I've started another Windows guest and during 3 hours two crashed occured, coincidence ?

Best regards
  Thimo

So according to my debugging, we really have just pushed the same irq which we have subsequently seen again unexpectedly.

This bug has only ever been seen on Haswell hardware, and appears linked to running HVM guests.

So either there is an erroneous ACK the LAPIC which is clearing the ISR before the PEOI stack is expecting (which I

"can't"

Apologies for the confusion.

~Andrew

obviously see, looking at the code), or something more funky is going on with the hardware.

CC'ing in the Intel maintainers:  Do you have any ideas?  Could this be related to APICv?

~Andrew

(XEN) **Pending EOI error
(XEN)   irq 29, vector 0x24
(XEN)   s[0] irq 29, vec 0x24, ready 0, ISR 00000001, TMR 00000000, IRR 00000000
(XEN) All LAPIC state:
(XEN) [vector]      ISR      TMR      IRR
(XEN) [1f:00] 00000000 00000000 00000000
(XEN) [3f:20] 00000010 76efa12e 00000000
(XEN) [5f:40] 00000000 e6f0f2fc 00000000
(XEN) [7f:60] 00000000 32d096ca 00000000
(XEN) [9f:80] 00000000 78fcf87a 00000000
(XEN) [bf:a0] 00000000 f9b9fe4e 00000000
(XEN) [df:c0] 00000000 ffdfe7ab 00000000
(XEN) [ff:e0] 00000000 00000000 00000000
(XEN) Peoi stack trace records:
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN)   Marked {sp 0, irq 29, vec 0x24} ready
(XEN)   Pushed {sp 0, irq 29, vec 0x24}
(XEN)   Poped entry {sp 1, irq 29, vec 0x24}
(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:1 vec:f0 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:1 vec:38 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  1(----),
(XEN)    IRQ:   2 affinity:f vec:00 type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:1 vec:40 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:1 vec:48 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   5 affinity:1 vec:50 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  5(----),
(XEN)    IRQ:   6 affinity:1 vec:58 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:1 vec:60 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   8 affinity:1 vec:68 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  8(----),
(XEN)    IRQ:   9 affinity:1 vec:70 type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0:  9(----),
(XEN)    IRQ:  10 affinity:1 vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:1 vec:88 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  12 affinity:1 vec:90 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  13 affinity:1 vec:98 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:1 vec:a0 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  15 affinity:1 vec:a8 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  16 affinity:1 vec:db type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 16(----),
(XEN)    IRQ:  18 affinity:1 vec:2c type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 18(----),
(XEN)    IRQ:  19 affinity:1 vec:51 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  20 affinity:1 vec:29 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  22 affinity:1 vec:bb type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 22(----),
(XEN)    IRQ:  23 affinity:8 vec:c2 type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 23(----),
(XEN)    IRQ:  24 affinity:1 vec:28 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  25 affinity:1 vec:30 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  26 affinity:f vec:c0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  27 affinity:f vec:c8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  28 affinity:f vec:d0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  29 affinity:2 vec:24 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:276(----),
(XEN)    IRQ:  30 affinity:4 vec:93 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:275(----),
(XEN)    IRQ:  31 affinity:2 vec:4a type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:274(----),
(XEN)    IRQ:  32 affinity:2 vec:73 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:273(----),
(XEN)    IRQ:  33 affinity:1 vec:49 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:272(----),
(XEN)    IRQ:  34 affinity:8 vec:5f type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:271(----),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  1 Vec 56:
(XEN)       Apic 0x00, Pin  1: vec=38 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  3 Vec 64:
(XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  4 Vec 72:
(XEN)       Apic 0x00, Pin  4: vec=48 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  5 Vec 80:
(XEN)       Apic 0x00, Pin  5: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  6 Vec 88:
(XEN)       Apic 0x00, Pin  6: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  7 Vec 96:
(XEN)       Apic 0x00, Pin  7: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  8 Vec104:
(XEN)       Apic 0x00, Pin  8: vec=68 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  9 Vec112:
(XEN)       Apic 0x00, Pin  9: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 10 Vec120:
(XEN)       Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 11 Vec136:
(XEN)       Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 12 Vec144:
(XEN)       Apic 0x00, Pin 12: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 13 Vec152:
(XEN)       Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 14 Vec160:
(XEN)       Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 15 Vec168:
(XEN)       Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 16 Vec219:
(XEN)       Apic 0x00, Pin 16: vec=db delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 18 Vec 44:
(XEN)       Apic 0x00, Pin 18: vec=2c delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 19 Vec 81:
(XEN)       Apic 0x00, Pin 19: vec=51 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 20 Vec 41:
(XEN)       Apic 0x00, Pin 20: vec=29 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 22 Vec187:
(XEN)       Apic 0x00, Pin 22: vec=bb delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 23 Vec194:
(XEN)       Apic 0x00, Pin 23: vec=c2 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #2 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #2......
(XEN) .... register #00: 02000000
(XEN) .......    : physical APIC id: 02
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    1    1    38
(XEN)  02 000 00  0    0    0   0   0    1    1    F0
(XEN)  03 000 00  0    0    0   0   0    1    1    40
(XEN)  04 000 00  0    0    0   0   0    1    1    48
(XEN)  05 000 00  0    0    0   0   0    1    1    50
(XEN)  06 000 00  0    0    0   0   0    1    1    58
(XEN)  07 000 00  0    0    0   0   0    1    1    60
(XEN)  08 000 00  0    0    0   0   0    1    1    68
(XEN)  09 000 00  0    1    0   0   0    1    1    70
(XEN)  0a 000 00  0    0    0   0   0    1    1    78
(XEN)  0b 000 00  0    0    0   0   0    1    1    88
(XEN)  0c 000 00  0    0    0   0   0    1    1    90
(XEN)  0d 000 00  0    0    0   0   0    1    1    98
(XEN)  0e 000 00  0    0    0   0   0    1    1    A0
(XEN)  0f 000 00  0    0    0   0   0    1    1    A8
(XEN)  10 000 00  0    1    0   1   0    1    1    DB
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  0    1    0   1   0    1    1    2C
(XEN)  13 000 00  1    1    0   1   0    1    1    51
(XEN)  14 000 00  1    1    0   1   0    1    1    29
(XEN)  15 07A 0A  1    0    0   0   0    0    2    B4
(XEN)  16 000 00  0    1    0   1   0    1    1    BB
(XEN)  17 000 00  0    1    0   1   0    1    1    C2
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ56 -> 0:1
(XEN) IRQ64 -> 0:3
(XEN) IRQ72 -> 0:4
(XEN) IRQ80 -> 0:5
(XEN) IRQ88 -> 0:6
(XEN) IRQ96 -> 0:7
(XEN) IRQ104 -> 0:8
(XEN) IRQ112 -> 0:9
(XEN) IRQ120 -> 0:10
(XEN) IRQ136 -> 0:11
(XEN) IRQ144 -> 0:12
(XEN) IRQ152 -> 0:13
(XEN) IRQ160 -> 0:14
(XEN) IRQ168 -> 0:15
(XEN) IRQ219 -> 0:16
(XEN) IRQ44 -> 0:18
(XEN) IRQ81 -> 0:19
(XEN) IRQ41 -> 0:20
(XEN) IRQ187 -> 0:22
(XEN) IRQ194 -> 0:23
(XEN) .................................... done.
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) CA-107844****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Executing crash image


Am 05.08.2013 16:51, schrieb Andrew Cooper:

All of these crashes are coming out of mwait_idle, so the cpu in
question has literally just been in an lower power state.

I am wondering whether there is some caching issue where an update to
the Pending EOI stack pointer got "lost", but this seems like a little
too specific to be reasonably explained as a caching issue.

A new debugging patch is on its way (Sorry - it has been a very busy few
days)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel