[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x



On 26.03.2013 19:50, Andrew Cooper wrote:
> On 26/03/2013 18:21, Marek Marczykowski wrote:
>> On 26.03.2013 18:54, Andrew Cooper wrote:
>>>>> Can you replace the ASSERT() with code similar to that in
>>>>>
>>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668
>>>>>
>>>>> Which should call dump_irqs() in before dying because of the ASSERT. 
>>>>> You might need to also take the latest version of dump_irqs() from
>>>>> unstable, as I seem to remember there was another assertion failure due
>>>>> to xfree()'ing in IRQ context.
>>>> Full log here:
>>>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log
>>>> Interesting part:
>> (...)
>>> Even more curious.  vector e9 does not appear to be programmed in.  Can
>>> you extend the debugging to also call __print_IO_APIC().
>>>
>>> The i debug key and z debug key list IO-APIC entries from different
>>> sources of information.
>> As you wish, full log:
>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs2.log
>>
>> Final part:
>> (XEN) *** IRQ BUG found ***
>> (XEN) CPU0 -Testing vector 233 from bitmap
>> 43,49,64,72,80,87-88,95-96,103,112,119-121,127,135,143-144,151-152,159-160,168,192,197,200,211,216,218
>> (XEN) Guest interrupt information:
>> (XEN)    IRQ:   0 affinity:00000000,00000000,00000000,00000001 vec:f0
>> type=IO-APIC-edge    status=00000000 mapped, unbound
>> (XEN)    IRQ:   1 affinity:00000000,00000000,00000000,00000001 vec:7f
>> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  1(-S--),
>> (XEN)    IRQ:   2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2
>> type=XT-PIC          status=00000000 mapped, unbound
>> (XEN)    IRQ:   3 affinity:00000000,00000000,00000000,00000001 vec:40
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:   4 affinity:00000000,00000000,00000000,00000001 vec:f1
>> type=IO-APIC-edge    status=00000000 mapped, unbound
>> (XEN)    IRQ:   5 affinity:00000000,00000000,00000000,00000001 vec:48
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:   6 affinity:00000000,00000000,00000000,00000001 vec:50
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:   7 affinity:00000000,00000000,00000000,00000008 vec:da
>> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  7(-S--),
>> (XEN)    IRQ:   8 affinity:00000000,00000000,00000000,00000004 vec:d8
>> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  8(-S--),
>> (XEN)    IRQ:   9 affinity:00000000,00000000,00000000,00000001 vec:87
>> type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(-S--),
>> (XEN)    IRQ:  10 affinity:00000000,00000000,00000000,00000001 vec:70
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  11 affinity:00000000,00000000,00000000,00000001 vec:78
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  12 affinity:00000000,00000000,00000000,00000001 vec:8f
>> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0: 12(-S--),
>> (XEN)    IRQ:  13 affinity:00000000,00000000,00000000,0000000f vec:90
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  14 affinity:00000000,00000000,00000000,00000001 vec:98
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  15 affinity:00000000,00000000,00000000,00000001 vec:a0
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  16 affinity:00000000,00000000,00000000,00000001 vec:97
>> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 16(-S--),
>> (XEN)    IRQ:  17 affinity:00000000,00000000,00000000,00000001 vec:9f
>> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 17(-S--),
>> (XEN)    IRQ:  18 affinity:00000000,00000000,00000000,00000004 vec:79
>> type=IO-APIC-level   status=00000002 mapped, unbound
>> (XEN)    IRQ:  19 affinity:00000000,00000000,00000000,0000000f vec:c8
>> type=IO-APIC-level   status=00000002 mapped, unbound
>> (XEN)    IRQ:  20 affinity:00000000,00000000,00000000,00000002 vec:d3
>> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 20(-S--),
>> (XEN)    IRQ:  22 affinity:00000000,00000000,00000000,0000000f vec:2b
>> type=IO-APIC-level   status=00000002 mapped, unbound
>> (XEN)    IRQ:  23 affinity:00000000,00000000,00000000,0000000f vec:a8
>> type=IO-APIC-edge    status=00000002 mapped, unbound
>> (XEN)    IRQ:  24 affinity:00000000,00000000,00000000,00000001 vec:28
>> type=DMA_MSI         status=00000000 mapped, unbound
>> (XEN)    IRQ:  25 affinity:00000000,00000000,00000000,00000001 vec:30
>> type=DMA_MSI         status=00000000 mapped, unbound
>> (XEN)    IRQ:  26 affinity:00000000,00000000,00000000,00000001 vec:c7
>> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:279(-S--),
>> (XEN)    IRQ:  27 affinity:00000000,00000000,00000000,00000001 vec:cf
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:278(-S--),
>> (XEN)    IRQ:  28 affinity:00000000,00000000,00000000,00000001 vec:d7
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:277(-S--),
>> (XEN)    IRQ:  29 affinity:00000000,00000000,00000000,00000001 vec:df
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:276(-S--),
>> (XEN)    IRQ:  30 affinity:00000000,00000000,00000000,00000001 vec:38
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:275(-S--),
>> (XEN)    IRQ:  31 affinity:00000000,00000000,00000000,00000004 vec:47
>> type=PCI-MSI         status=00000002 mapped, unbound
>> (XEN)    IRQ:  32 affinity:00000000,00000000,00000000,00000001 vec:a7
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:273(-S--),
>> (XEN)    IRQ:  33 affinity:00000000,00000000,00000000,00000001 vec:b7
>> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:272(-S--),
>> (XEN)    IRQ:  34 affinity:00000000,00000000,00000000,00000004 vec:40
>> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:271(PS--),
>> (XEN)    IRQ:  35 affinity:00000000,00000000,00000000,00000001 vec:af
>> type=PCI-MSI         status=00000050 in-flight=0 domain-list=1: 55(-S--),
>> (XEN) IO-APIC interrupt information:
>> (XEN)     IRQ  0 Vec240:
>> (XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  1 Vec127:
>> (XEN)       Apic 0x00, Pin  1: vec=7f delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  3 Vec 64:
>> (XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  4 Vec241:
>> (XEN)       Apic 0x00, Pin  4: vec=f1 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  5 Vec 72:
>> (XEN)       Apic 0x00, Pin  5: vec=48 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  6 Vec 80:
>> (XEN)       Apic 0x00, Pin  6: vec=50 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  7 Vec218:
>> (XEN)       Apic 0x00, Pin  7: vec=da delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  8 Vec216:
>> (XEN)       Apic 0x00, Pin  8: vec=d8 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ  9 Vec135:
>> (XEN)       Apic 0x00, Pin  9: vec=87 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=L mask=0 dest_id:0
>> (XEN)     IRQ 10 Vec112:
>> (XEN)       Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ 11 Vec120:
>> (XEN)       Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ 12 Vec143:
>> (XEN)       Apic 0x00, Pin 12: vec=8f delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ 13 Vec144:
>> (XEN)       Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=1 dest_id:0
>> (XEN)     IRQ 14 Vec152:
>> (XEN)       Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ 15 Vec160:
>> (XEN)       Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN)     IRQ 16 Vec151:
>> (XEN)       Apic 0x00, Pin 16: vec=97 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN)     IRQ 17 Vec159:
>> (XEN)       Apic 0x00, Pin 17: vec=9f delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN)     IRQ 18 Vec121:
>> (XEN)       Apic 0x00, Pin 18: vec=79 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN)     IRQ 19 Vec200:
>> (XEN)       Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN)     IRQ 20 Vec211:
>> (XEN)       Apic 0x00, Pin 20: vec=d3 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN)     IRQ 22 Vec 43:
>> (XEN)       Apic 0x00, Pin 22: vec=2b delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN)     IRQ 23 Vec168:
>> (XEN)       Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=1 dest_id:0
>> (XEN) number of MP IRQ sources: 15.
>> (XEN) number of IO-APIC #2 registers: 24.
>> (XEN) testing the IO APIC.......................
>> (XEN) IO APIC #2......
>> (XEN) .... register #00: 02000000
>> (XEN) .......    : physical APIC id: 02
>> (XEN) .......    : Delivery Type: 0
>> (XEN) .......    : LTS          : 0
>> (XEN) .... register #01: 00170020
>> (XEN) .......     : max redirection entries: 0017
>> (XEN) .......     : PRQ implemented: 0
>> (XEN) .......     : IO APIC version: 0020
>> (XEN) .... IRQ redirection table:
>> (XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
>> (XEN)  00 0DC 0C  1    0    0   0   0    1    2    87
>> (XEN)  01 000 00  0    0    0   0   0    1    1    7F
>> (XEN)  02 000 00  0    0    0   0   0    1    1    F0
>> (XEN)  03 000 00  0    0    0   0   0    1    1    40
>> (XEN)  04 000 00  0    0    0   0   0    1    1    F1
>> (XEN)  05 000 00  0    0    0   0   0    1    1    48
>> (XEN)  06 000 00  0    0    0   0   0    1    1    50
>> (XEN)  07 000 00  0    0    0   0   0    1    1    DA
>> (XEN)  08 000 00  0    0    0   0   0    1    1    D8
>> (XEN)  09 000 00  0    1    0   0   0    1    1    87
>> (XEN)  0a 000 00  0    0    0   0   0    1    1    70
>> (XEN)  0b 000 00  0    0    0   0   0    1    1    78
>> (XEN)  0c 000 00  0    0    0   0   0    1    1    8F
>> (XEN)  0d 000 00  1    0    0   0   0    1    1    90
>> (XEN)  0e 000 00  0    0    0   0   0    1    1    98
>> (XEN)  0f 000 00  0    0    0   0   0    1    1    A0
>> (XEN)  10 000 00  0    1    0   1   0    1    1    97
>> (XEN)  11 000 00  0    1    0   1   0    1    1    9F
>> (XEN)  12 000 00  1    1    0   1   0    1    1    79
>> (XEN)  13 000 00  1    1    0   1   0    1    1    C8
>> (XEN)  14 000 00  0    1    0   1   0    1    1    D3
>> (XEN)  15 000 00  1    0    0   0   0    0    0    00
>> (XEN)  16 000 00  1    1    0   1   0    1    1    2B
>> (XEN)  17 000 00  1    0    0   0   0    1    1    A8
>> (XEN) Using vector-based indexing
>> (XEN) IRQ to pin mappings:
>> (XEN) IRQ240 -> 0:2
>> (XEN) IRQ127 -> 0:1
>> (XEN) IRQ64 -> 0:3
>> (XEN) IRQ241 -> 0:4
>> (XEN) IRQ72 -> 0:5
>> (XEN) IRQ80 -> 0:6
>> (XEN) IRQ218 -> 0:7
>> (XEN) IRQ216 -> 0:8
>> (XEN) IRQ135 -> 0:9
>> (XEN) IRQ112 -> 0:10
>> (XEN) IRQ120 -> 0:11
>> (XEN) IRQ143 -> 0:12
>> (XEN) IRQ144 -> 0:13
>> (XEN) IRQ152 -> 0:14
>> (XEN) IRQ160 -> 0:15
>> (XEN) IRQ151 -> 0:16
>> (XEN) IRQ159 -> 0:17
>> (XEN) IRQ121 -> 0:18
>> (XEN) IRQ200 -> 0:19
>> (XEN) IRQ211 -> 0:20
>> (XEN) IRQ43 -> 0:22
>> (XEN) IRQ168 -> 0:23
>> (XEN) .................................... done.
>> (XEN) Xen BUG at io_apic.c:556
>> (XEN) ----[ Xen-4.1.5-rc1  x86_64  debug=y  Tainted:    C ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e008:[<ffff82c48015e2db>] 
>> smp_irq_move_cleanup_interrupt+0x216/0x28e
>> (XEN) RFLAGS: 0000000000010092   CONTEXT: hypervisor
>> (XEN) rax: 0000000000000000   rbx: 00000000000000e9   rcx: 0000000000000000
>> (XEN) rdx: 0000000000000000   rsi: 000000000000000a   rdi: ffff82c4802592e0
>> (XEN) rbp: ffff82c48029ff08   rsp: ffff82c48029feb8   r8:  0000000000000004
>> (XEN) r9:  0000000000000004   r10: 0000000000000004   r11: 0000000000000002
>> (XEN) r12: ffff830421080250   r13: ffff830421060534   r14: ffff82c48029ff18
>> (XEN) r15: ffff82c4802dd9e0   cr0: 000000008005003b   cr4: 00000000000026f0
>> (XEN) cr3: 000000026582c000   cr2: ffff8804020701d8
>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>> (XEN) Xen stack trace from rsp=ffff82c48029feb8:
>> (XEN)    0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000
>> (XEN)    000000000000e02b 0000000000000000 000000004bf51982 00000000000060a9
>> (XEN)    0000000000000000 0000000000000000 00007d3b7fd600c7 ffff82c48014de60
>> (XEN)    0000000000000000 0000000000000000 00000000000060a9 000000004bf51982
>> (XEN)    ffff8802d2665b28 0000000000000000 0000000000000000 0000000000007ff0
>> (XEN)    0000000000000022 0000000000000000 000000024bf57322 0000000001307da0
>> (XEN)    00000000000059a0 0000000000000000 00000000000060a9 0000002000000000
>> (XEN)    ffffffff8123c51a 000000000000e033 0000000000000293 ffff8802d2665b08
>> (XEN)    000000000000e02b 0000000000000000 0000000000000000 0000000000000000
>> (XEN)    0000000000000000 0000000000000000 ffff8300ca9a0000 0000000000000000
>> (XEN)    0000000000000000
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82c48015e2db>] smp_irq_move_cleanup_interrupt+0x216/0x28e
>>
>>
>>
> 
> So vector e9 doesn't appear to be programmed in anywhere.
> 
> I am starting to get more into the realm of guessing here but, can you
> use apic_verbosity=debug on the command line and copy this extra
> debugging logic to send_cleanup_vector()
> 
> You should be able to conditionally trigger it on "desc->arch.vector ==
> 0xe9".  You will probably also want to change the BUG() to a WARN(), so
> we get the interrupt and ioapic information on both sides of the cleanup
> vector, as well as getting the stack trace of the codepath through Xen
> as a result of vector 0xe9.

send_cleanup_vector() doesn't seem to be called with cfg->vector == 0xe9...
Can dom0 mess something here around?

-- 
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.