[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [xen-unstable test] 11946: regressions - FAIL



On Mon, May 7, 2012 at 1:34 PM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>
> >>> On 07.05.12 at 13:50, Andrew Cooper <andrew.cooper3@xxxxxxxxxx> wrote:
> > On 07/05/2012 09:10, Jan Beulich wrote:
> >>>>> On 05.05.12 at 02:21, AP <apxeng@xxxxxxxxx> wrote:
> >>> (XEN) *** IRQ BUG found ***
> >>> (XEN) CPU0 -Testing vector 236 from bitmap
> >> 236 = 0xec = FIRST_LEGACY_VECTOR + 0x0c, i.e. an IRQ12 coming
> >> in through the 8259A. Something fundamentally fishy must be going
> >> on here, and I would suppose the code in question shouldn't even be
> >> reached for legacy vectors.
> >>
> >> Furthermore, calling dump_irqs() from the debugging code with
> >> desc->lock still held makes it impossible to get full output, as that
> >> function wants to lock all initialized IRQ descriptors.
> >
> > Yes - it has been vector 236 on each of the 3 reported failures from AP,
> > and I believe it was also vector 236 in the one case I managed to
> > reproduce the issue.
> >
> > However, once we have set up the IO-APIC, the 8259A should not be used
> > any more.  The boot dmeg shows that io_ack_method is indeed "old" (which
> > was going to be my first suggestion), and that EOI Broadcast Suppression
> > is enabled, which I have already identified as a source of problems for
> > some customers.  As a 'fix', I provided the ability for
> > "io_ack_method=new" to prevent EOI Broadcast Suppression being enabled.
> > This was upstreamed in c/s 24870:9bf3ec036bef, but apparently has not
> > completely fixed the customer problems - just made it substantially more
> > rare.
> >
> > AP: Can you manually invoke the 'i' debug key and provide that - it will
> > help to see how Xen is setting up the IO-APIC(s) on your system.

(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:01 vec:f0 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:02 vec:85 type=IO-APIC-edge    status=00000030 in-flight=0 domain-list=0:  1(----),
(XEN)    IRQ:   2 affinity:ff vec:e2 type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:01 vec:40 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:01 vec:48 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   5 affinity:01 vec:50 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   6 affinity:01 vec:58 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:01 vec:60 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   8 affinity:08 vec:29 type=IO-APIC-edge    status=00000030 in-flight=0 domain-list=0:  8(----),
(XEN)    IRQ:   9 affinity:02 vec:7f type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(----),
(XEN)    IRQ:  10 affinity:01 vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:01 vec:88 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  12 affinity:08 vec:d4 type=IO-APIC-edge    status=00000030 in-flight=0 domain-list=0: 12(----),
(XEN)    IRQ:  13 affinity:0f vec:98 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:01 vec:a0 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  15 affinity:01 vec:a8 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  16 affinity:02 vec:a6 type=IO-APIC-level   status=00000030 in-flight=0 domain-list=0: 16(----),
(XEN)    IRQ:  17 affinity:0f vec:c0 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  18 affinity:0f vec:c8 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  19 affinity:0f vec:f1 type=IO-APIC-level   status=00000000 mapped, unbound
(XEN)    IRQ:  20 affinity:0f vec:61 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  22 affinity:0f vec:32 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  23 affinity:01 vec:ac type=IO-APIC-level   status=00000030 in-flight=0 domain-list=0: 23(----),
(XEN)    IRQ:  24 affinity:01 vec:28 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  25 affinity:01 vec:30 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  26 affinity:01 vec:31 type=PCI-MSI/-X      status=00000030 in-flight=0 domain-list=0:279(----),
(XEN)    IRQ:  27 affinity:01 vec:39 type=PCI-MSI/-X      status=00000030 in-flight=0 domain-list=0:278(----),
(XEN)    IRQ:  28 affinity:01 vec:41 type=PCI-MSI/-X      status=00000030 in-flight=0 domain-list=0:277(----),
(XEN)    IRQ:  29 affinity:01 vec:49 type=PCI-MSI/-X      status=00000030 in-flight=0 domain-list=0:276(----),
(XEN)    IRQ:  30 affinity:01 vec:51 type=PCI-MSI/-X      status=00000030 in-flight=0 domain-list=0:275(----),
(XEN)    IRQ:  31 affinity:04 vec:d7 type=PCI-MSI         status=00000030 in-flight=0 domain-list=0:274(----),
(XEN)    IRQ:  32 affinity:04 vec:df type=PCI-MSI         status=00000030 in-flight=0 domain-list=0:273(----),
(XEN)    IRQ:  33 affinity:02 vec:b0 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:272(----),
(XEN)    IRQ:  34 affinity:02 vec:a8 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:271(----),
(XEN)    IRQ:  35 affinity:04 vec:ad type=PCI-MSI         status=00000030 in-flight=0 domain-list=0:270(----),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  1 Vec133:
(XEN)       Apic 0x00, Pin  1: vec=85 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  3 Vec 64:
(XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  4 Vec 72:
(XEN)       Apic 0x00, Pin  4: vec=48 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  5 Vec 80:
(XEN)       Apic 0x00, Pin  5: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  6 Vec 88:
(XEN)       Apic 0x00, Pin  6: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  7 Vec 96:
(XEN)       Apic 0x00, Pin  7: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  8 Vec 41:
(XEN)       Apic 0x00, Pin  8: vec=29 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  9 Vec127:
(XEN)       Apic 0x00, Pin  9: vec=7f delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 10 Vec120:
(XEN)       Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 11 Vec136:
(XEN)       Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 12 Vec212:
(XEN)       Apic 0x00, Pin 12: vec=d4 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 13 Vec152:
(XEN)       Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=1 dest_id:0
(XEN)     IRQ 14 Vec160:
(XEN)       Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 15 Vec168:
(XEN)       Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 16 Vec166:
(XEN)       Apic 0x00, Pin 16: vec=a6 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 17 Vec192:
(XEN)       Apic 0x00, Pin 17: vec=c0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 18 Vec200:
(XEN)       Apic 0x00, Pin 18: vec=c8 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 19 Vec241:
(XEN)       Apic 0x00, Pin 19: vec=f1 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 20 Vec 97:
(XEN)       Apic 0x00, Pin 20: vec=61 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 22 Vec 50:
(XEN)       Apic 0x00, Pin 22: vec=32 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 23 Vec172:
(XEN)       Apic 0x00, Pin 23: vec=ac delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0

> Seeing the 'z' output might also be helpful, especially to see whether
> any of the IO-APICs' RTEs is an ExtINT one.

(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #2 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #2......
(XEN) .... register #00: 02000000
(XEN) .......    : physical APIC id: 02
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:  
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    1    1    85
(XEN)  02 000 00  0    0    0   0   0    1    1    F0
(XEN)  03 000 00  0    0    0   0   0    1    1    40
(XEN)  04 000 00  0    0    0   0   0    1    1    48
(XEN)  05 000 00  0    0    0   0   0    1    1    50
(XEN)  06 000 00  0    0    0   0   0    1    1    58
(XEN)  07 000 00  0    0    0   0   0    1    1    60
(XEN)  08 000 00  0    0    0   0   0    1    1    29
(XEN)  09 000 00  0    1    0   0   0    1    1    A7
(XEN)  0a 000 00  0    0    0   0   0    1    1    78
(XEN)  0b 000 00  0    0    0   0   0    1    1    88
(XEN)  0c 000 00  0    0    0   0   0    1    1    D4
(XEN)  0d 000 00  1    0    0   0   0    1    1    98
(XEN)  0e 000 00  0    0    0   0   0    1    1    A0
(XEN)  0f 000 00  0    0    0   0   0    1    1    A8
(XEN)  10 000 00  0    1    0   1   0    1    1    AE
(XEN)  11 000 00  1    1    0   1   0    1    1    C0
(XEN)  12 000 00  1    1    0   1   0    1    1    C8
(XEN)  13 000 00  0    1    0   1   0    1    1    F1
(XEN)  14 000 00  1    1    0   1   0    1    1    61
(XEN)  15 0CA 0A  1    0    0   0   0    1    2    71
(XEN)  16 000 00  1    1    0   1   0    1    1    32
(XEN)  17 000 00  0    1    0   1   0    1    1    AC
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ133 -> 0:1
(XEN) IRQ64 -> 0:3
(XEN) IRQ72 -> 0:4
(XEN) IRQ80 -> 0:5
(XEN) IRQ88 -> 0:6
(XEN) IRQ96 -> 0:7
(XEN) IRQ41 -> 0:8
(XEN) IRQ167 -> 0:9
(XEN) IRQ120 -> 0:10
(XEN) IRQ136 -> 0:11
(XEN) IRQ212 -> 0:12
(XEN) IRQ152 -> 0:13
(XEN) IRQ160 -> 0:14
(XEN) IRQ168 -> 0:15
(XEN) IRQ174 -> 0:16
(XEN) IRQ192 -> 0:17
(XEN) IRQ200 -> 0:18
(XEN) IRQ241 -> 0:19
(XEN) IRQ97 -> 0:20
(XEN) IRQ50 -> 0:22
(XEN) IRQ172 -> 0:23
(XEN) .................................... done.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.