[Xen-devel] xen/arm: Assertion 'timer->status >= TIMER_STATUS_inactive' failed at timer.c:279

[Julien Grall <julien.grall@xxxxxxx>]

Hi Dario,

A couple of people have been reported Xen crash on the ARM64
Foundation Model [1] with recent unstable.

The crash seems to happen when Xen fails to bring up secondary CPUs
(see stack trace below).

From my understanding, csched_free_pdata is trying to kill the
timer spc->ticker. However the status of this timer is
TIMER_STATUS_invalid.

This is because csched_init_pdata has set a deadline for the
timer (set_timer) and the softirq to schedule the timer has
not yet happen (indeed Xen is still in early boot).

I am not sure how to fix this issue. How will you recommend
to fix it?

For your information the bisector fingered the following
commit:

commit 64269d936584c29de951c6613bf618640832b9a6
Author: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
Date:   Fri Apr 8 00:03:03 2016 +0200

    sched: implement .init_pdata in Credit, Credit2 and RTDS
    
    In fact, if a scheduler needs per-pCPU information,
    that needs to be initialized appropriately. So, we take
    the code that is performing initializations from (right
    now) .alloc_pdata, and use it for .init_pdata, leaving
    only actualy allocations in the former, if any (which
    is the case in RTDS and Credit1).
    
    On the other hand, in Credit2, since we don't really
    need any per-pCPU data allocation, everything that was
    being done in .alloc_pdata, is now done in .init_pdata.
    And the fact that now .alloc_pdata can be left undefined,
    allows us to just get rid of it.
    
    Still for Credit2, the fact that .init_pdata is called
    during CPU_STARTING (rather than CPU_UP_PREPARE) kills
    the need for the scheduler to setup a similar callback
    itself, simplifying the code.
    
    And thanks to such simplification, it is now also ok to
    move some of the logic meant at double checking that a
    cpu was (or was not) initialized, into ASSERTS (rather
    than an if() and a BUG_ON).
    
    Signed-off-by: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
    Reviewed-by: Meng Xu <mengxu@xxxxxxxxxxxxx>
    Reviewed-by: George Dunlap <george.dunlap@xxxxxxxxxxxxx>

(XEN) Assertion 'timer->status >= TIMER_STATUS_inactive' failed at timer.c:279
(XEN) ----[ Xen-4.7-unstable  arm64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     000000000022e304 timer.c#active_timer+0x8/0x24
(XEN) LR:     000000000022f624
(XEN) SP:     00000000002bfcd0
(XEN) CPSR:   600002c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN)      X0: 0000000000000000  X1: 0000000000000003  X2: 0000000000000000
(XEN)      X3: 0000000000120012  X4: 0000000000000010  X5: 0000000000000080
(XEN)      X6: 0000000000000004  X7: 0000000000000001  X8: 00000000fffffffd
(XEN)      X9: 000000000000000a X10: 00000000002bfb08 X11: 0000000000000033
(XEN)     X12: 0000000000000001 X13: 0000000000263dc0 X14: 0000000000000020
(XEN)     X15: 0000000000000000 X16: 0000000000000000 X17: 0000000000000000
(XEN)     X18: 0000000000000000 X19: 000080017ffcad68 X20: 000000000030bb00
(XEN)     X21: 00000000000002c0 X22: 0000000000000000 X23: 000000000030bb00
(XEN)     X24: 0000000000306000 X25: 000080017ffcad90 X26: 000000000000ffff
(XEN)     X27: 0000000900000000 X28: 0000000000000002  FP: 00000000002bfcd0
(XEN) 
(XEN)   VTCR_EL2: 80000000
(XEN)  VTTBR_EL2: 0000000000000000
(XEN) 
(XEN)  SCTLR_EL2: 30cd183d
(XEN)    HCR_EL2: 000000000038643f
(XEN)  TTBR0_EL2: 00000000feef6000
(XEN) 
(XEN)    ESR_EL2: f2000001
(XEN)  HPFAR_EL2: 0000000000000000
(XEN)    FAR_EL2: 0000000000000000
(XEN) 
(XEN) Xen stack trace from sp=00000000002bfcd0:
(XEN)    00000000002bfd20 00000000002208c0 000080017ffca240 0000000000000003
(XEN)    000080017ffcad50 0000000000000240 000080017ffca2a8 0000000000000003
(XEN)    00000000002bfe18 0000000080000000 00000000002bfd60 0000000000227a18
(XEN)    0000000000000003 00000000002807b0 000080017ff9c0c0 0000000000008002
(XEN)    0000000000008000 0000000000000003 00000000002bfd90 0000000000219944
(XEN)    000000000027e318 000000000027e310 000000000027e008 0000000000008002
(XEN)    00000000002bfde0 00000000002015fc 00000000fffffffd 0000000000000000
(XEN)    0000000000000003 0000000000000003 000000000027adb8 0000000000000004
(XEN)    00000000002a80a8 0000000080000000 00000000002bfe20 000000000028f7c0
(XEN)    0000000000000003 000000000027ea00 0000000000306450 0000000000280ab0
(XEN)    0000000900000000 000000000027e000 0000000000000000 00000000810021d8
(XEN)    0000000081002000 0000000080e02000 0000000088000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000001 0000000000000000
(XEN)    0000000000000000 0000000080002174 0000000000000000 0000000088000000
(XEN)    0000000000002106 00000000ff000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<000000000022e304>] timer.c#active_timer+0x8/0x24 (PC)
(XEN)    [<000000000022f624>] kill_timer+0x108/0x2e0 (LR)
(XEN)    [<00000000002208c0>] sched_credit.c#csched_free_pdata+0xd8/0x114
(XEN)    [<0000000000227a18>] schedule.c#cpu_schedule_callback+0xc0/0x12c
(XEN)    [<0000000000219944>] notifier_call_chain+0x78/0x9c
(XEN)    [<00000000002015fc>] cpu_up+0x104/0x130
(XEN)    [<000000000028f7c0>] start_xen+0xaf8/0xce0
(XEN)    [<00000000810021d8>] 00000000810021d8
(XEN) 
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion 'timer->status >= TIMER_STATUS_inactive' failed at timer.c:279
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

Regards,

[1] http://www.arm.com/products/tools/models/fast-models/foundation-model.php

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel