Re: [Xen-devel] [BUG] Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at irq.c:1163

[HÃkon Alstadheim <hakon@xxxxxxxxxxxxxxxxxx>]

Den 17. jan. 2016 16:25, skrev Andrew Cooper:
> On 17/01/16 15:16, Andrew Cooper wrote:
>>>> This isn't the first time we have seen this on Haswell processors. Do
>>>> you have microcode loading set up?
>>>>
>>>> ~Andrew
>>>>
>>> Still happening with kernel-genkernel-x86_64-4.1.15-gentoo and updated
>>> cpu microcode, using microcode from 20151106.
...
>>> Actually, this will be more useful:
>>>
>>> diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
>>> index 1228568..4e75b03 100644
>>> --- a/xen/arch/x86/irq.c
>>> +++ b/xen/arch/x86/irq.c
>>> @@ -1165,6 +1165,15 @@ static void __do_IRQ_guest(int irq)
>>>      if ( action->ack_type == ACKTYPE_EOI )
>>>      {
>>>          sp = pending_eoi_sp(peoi);
>>> +        if ( unlikely(!((sp == 0) || (peoi[sp-1].vector < vector))) )
>>> +        {
>>> +            int p;
>>> +
>>> +            printk("** sp %d, irq %d, vec %#x\n", sp, irq, vector);
>>> +            for ( p = sp; p > 0; --p )
>>> +                printk("**peoi[%d] = {%d, %#x, %d}\n",
>>> +                       p-1, peoi[p-1].irq, peoi[p-1].vector,
>>> peoi[p-1].ready);
>>> +        }
>>>          ASSERT((sp == 0) || (peoi[sp-1].vector < vector));
>>>          ASSERT(sp < (NR_DYNAMIC_VECTORS-1));
>>>          peoi[sp].irq = irq;
>>>
>>>
>>>

Got one again. dom5 is my desktop, dom1 is my
mail-server/router/firewall. (planning to split that up ... ) . Is there
any additional info that would be useful?

Running now with gentoo xen 4.6.0-r8 and xen-tools 4.6.0-r7. dom0 kernel
is gentoo-sources-4.1.15-r1 , and the above patch.

I tried running with maxcpus=6 for a while, but I had to disable some
services to get that running. So, when nothing happened for a while I
re-enabled all my cores (two cpus, 12 cores, 24 threads). I was running
with two cpu-pools, one for each cpu. I have not re-enabled that.
-------- Here is the dump from the assert (full log attached): -------------
(XEN) [2016-01-22 06:19:15] grant_table.c:1491:d5v2 Expanding dom (5)
grant table from (9) to (10) frames.
(XEN) [2016-01-22 07:32:50] grant_table.c:1491:d1v3 Expanding dom (1)
grant table from (12) to (13) frames.
(XEN) [2016-01-22 07:33:36] ** sp 1, irq 107, vec 0x3b
(XEN) [2016-01-22 07:33:36] **peoi[0] = {107, 0x3b, 0}
(XEN) [2016-01-22 07:33:36] Assertion '(sp == 0) || (peoi[sp-1].vector <
vector)' failed at irq.c:1172
(XEN) [2016-01-22 07:33:36] ----[ Xen-4.6.0  x86_64  debug=y 
Tainted:    C ]----
(XEN) [2016-01-22 07:33:36] CPU:    21
(XEN) [2016-01-22 07:33:36] RIP:    e008:[<ffff82d080170205>]
do_IRQ+0x451/0x6ea
(XEN) [2016-01-22 07:33:36] RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) [2016-01-22 07:33:36] rax: 000000000000003b   rbx:
000000000000003b   rcx: 0000000000000000
(XEN) [2016-01-22 07:33:36] rdx: ffff83107e048000   rsi:
000000000000000a   rdi: ffff82d0802ab768
(XEN) [2016-01-22 07:33:36] rbp: ffff83107e04fdc8   rsp:
ffff83107e04fd58   r8:  ffff83083ff00000
(XEN) [2016-01-22 07:33:36] r9:  0000000000000002   r10:
0000000000000024   r11: 0000000000000002
(XEN) [2016-01-22 07:33:36] r12: ffff83072ea1ef50   r13:
ffff830839bc8480   r14: 0000000000000001
(XEN) [2016-01-22 07:33:36] r15: 000000000000006b   cr0:
000000008005003b   cr4: 00000000001526e0
(XEN) [2016-01-22 07:33:36] cr3: 00000004d337d000   cr2: 00007f0152c28000
(XEN) [2016-01-22 07:33:36] ds: 0000   es: 0000   fs: 0000   gs: 0000  
ss: 0000   cs: e008
(XEN) [2016-01-22 07:33:36] Xen stack trace from rsp=ffff83107e04fd58:
(XEN) [2016-01-22 07:33:36]    000000000000006b ffff830839bc8480
ffff831000000000 ffff830839c06b24
(XEN) [2016-01-22 07:33:36]    0000000000000000 0000006b00000001
00000015ffc00015 ffff830839dadac0
(XEN) [2016-01-22 07:33:36]    ffff82d08034a480 00013dd61b4fef1a
ffff830839bd0da0 0000000000000004
(XEN) [2016-01-22 07:33:36]    0000000000000008 ffff830839bd0e60
00007cef81fb0207 ffff82d08023b132
(XEN) [2016-01-22 07:33:36]    ffff830839bd0e60 0000000000000008
0000000000000004 ffff830839bd0da0
(XEN) [2016-01-22 07:33:36]    ffff83107e04fef0 00013dd61b4fef1a
00013dd61b7d5a72 0000000000000000
(XEN) [2016-01-22 07:33:36]    ffff83007dfe8060 0000000000000914
ffff830839bc8020 20c49ba5e353f7cf
(XEN) [2016-01-22 07:33:36]    ffff83107e048000 00013dd61b4f6ed5
ffff830839bd0dd0 0000003b00000000
(XEN) [2016-01-22 07:33:36]    ffff82d0801af1ea 000000000000e008
0000000000000202 ffff83107e04fe80
(XEN) [2016-01-22 07:33:36]    0000000000000000 0000002075ff8000
00013dd61b466d12 ffff82d0801ebb00
(XEN) [2016-01-22 07:33:36]    ffff83107e04ff08 ffff82d080321b00
0000000000000000 0000000000000000
(XEN) [2016-01-22 07:33:36]    000002e200000229 ffffffff81a04000
ffff83107e048000 ffff83007dfe8000
(XEN) [2016-01-22 07:33:36]    00000000ffffffff ffff830839d2f000
ffff8304ccbf0000 ffff83107e04ff10
(XEN) [2016-01-22 07:33:36]    ffff82d0801607bc ffff82d08012c574
ffff830075ff8000 ffff83107e04fdd8
(XEN) [2016-01-22 07:33:36]    ffffffff81a04000 ffffffff81a04000
ffffffff81a04000 0000000000000000
(XEN) [2016-01-22 07:33:36]    ffffffff81a03ed8 0000000000000000
0000000000000206 ffff88014111bbd0
(XEN) [2016-01-22 07:33:36]    0000000000000000 0000000000000000
0000000000000000 ffffffff81a2e700
(XEN) [2016-01-22 07:33:36]    ffff88023f20da70 0000000000000000
0000000000000000 0000beef0000beef
(XEN) [2016-01-22 07:33:36]    ffffffff81038162 000000bf0000beef
0000000000000286 ffffffff81a03ed8
(XEN) [2016-01-22 07:33:36]    000000000000beef 000000000000beef
000000000000beef 000000000000beef
(XEN) [2016-01-22 07:33:36] Xen call trace:
(XEN) [2016-01-22 07:33:36]    [<ffff82d080170205>] do_IRQ+0x451/0x6ea
(XEN) [2016-01-22 07:33:36]    [<ffff82d08023b132>]
common_interrupt+0x62/0x70
(XEN) [2016-01-22 07:33:36]    [<ffff82d0801af1ea>] mwait_idle+0x2cb/0x315
(XEN) [2016-01-22 07:33:36]    [<ffff82d0801607bc>] idle_loop+0x51/0x6b
(XEN) [2016-01-22 07:33:36]
(XEN) [2016-01-22 07:33:36]
(XEN) [2016-01-22 07:33:36] ****************************************
(XEN) [2016-01-22 07:33:36] Panic on CPU 21:
(XEN) [2016-01-22 07:33:36] Assertion '(sp == 0) || (peoi[sp-1].vector <
vector)' failed at irq.c:1172
(XEN) [2016-01-22 07:33:36] ****************************************
(XEN) [2016-01-22 07:33:36]
(XEN) [2016-01-22 07:33:36] Reboot in five seconds...
----

Attachment: xen-assert.log
Description: Text Data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel