CPU_DOWN_FAILED hits ASSERTs in scheduling logic

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

Hello,

When the stop_machine_run() call in cpu_down() fails and calls the CPU
notifier CPU_DOWN_FAILED hook the following assert triggers in the
scheduling code:

Assertion '!cpumask_test_cpu(cpu, &prv->initialized)' failed at 
common/sched/cred1
----[ Xen-4.19-unstable  x86_64  debug=y  Tainted:   C    ]----
CPU:    0
RIP:    e008:[<ffff82d040248299>] 
common/sched/credit2.c#csched2_free_pdata+0xc8/0x177
RFLAGS: 0000000000010093   CONTEXT: hypervisor
rax: 0000000000000000   rbx: ffff83202ecc2f80   rcx: ffff83202f3e64c0
rdx: 0000000000000001   rsi: 0000000000000002   rdi: ffff83202ecc2f88
rbp: ffff83203ffffd58   rsp: ffff83203ffffd30   r8:  0000000000000000
r9:  ffff83202f3e6e01   r10: 0000000000000000   r11: 0f0f0f0f0f0f0f0f
r12: ffff83202ecb80b0   r13: 0000000000000001   r14: 0000000000000282
r15: ffff83202ecbbf00   cr0: 000000008005003b   cr4: 00000000007526e0
cr3: 00000000574c2000   cr2: 0000000000000000
fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
Xen code around <ffff82d040248299> 
(common/sched/credit2.c#csched2_free_pdata+0xc8/0x177):
 fe ff eb 9a 0f 0b 0f 0b <0f> 0b 49 8d 4f 08 49 8b 47 08 48 3b 48 08 75 2e
Xen stack trace from rsp=ffff83203ffffd30:
   ffff83202d74d100 0000000000000001 ffff82d0404c4430 0000000000000006
   0000000000000000 ffff83203ffffd78 ffff82d040257454 0000000000000000
   0000000000000001 ffff83203ffffda8 ffff82d04021f303 ffff82d0404c4628
   ffff82d0404c4620 ffff82d0404c4430 0000000000000006 ffff83203ffffdf0
   ffff82d04022bc4c ffff83203ffffe18 0000000000000001 0000000000000001
   00000000fffffff0 0000000000000000 0000000000000000 ffff82d0405e6500
   ffff83203ffffe08 ffff82d040204fd5 0000000000000001 ffff83203ffffe30
   ffff82d0402054f0 ffff82d0404c5860 0000000000000001 ffff83202ec75000
   ffff83203ffffe48 ffff82d040348c25 ffff83202d74d0d0 ffff83203ffffe68
   ffff82d0402071aa ffff83202ec751d0 ffff82d0405ce210 ffff83203ffffe80
   ffff82d0402343c9 ffff82d0405ce200 ffff83203ffffeb0 ffff82d040234631
   0000000000000000 0000000000007fff ffff82d0405d5080 ffff82d0405ce210
   ffff83203ffffee8 ffff82d040321411 ffff82d040321399 ffff83202f3a9000
   0000000000000000 0000001d91a6fa2d ffff82d0405e6500 ffff83203ffffde0
   ffff82d040324391 0000000000000000 0000000000000000 0000000000000000
   0000000000000000 0000000000000000 0000000000000000 0000000000000000
   0000000000000000 0000000000000000 0000000000000000 0000000000000000
   0000000000000000 0000000000000000 0000000000000000 0000000000000000
   0000000000000000 0000000000000000 0000000000000000 0000000000000000
   0000000000000000 0000000000000000 0000000000000000 0000000000000000
Xen call trace:
   [<ffff82d040248299>] R common/sched/credit2.c#csched2_free_pdata+0xc8/0x177
   [<ffff82d040257454>] F free_cpu_rm_data+0x41/0x58
   [<ffff82d04021f303>] F common/sched/cpupool.c#cpu_callback+0xfb/0x466
   [<ffff82d04022bc4c>] F notifier_call_chain+0x6c/0x96
   [<ffff82d040204fd5>] F common/cpu.c#cpu_notifier_call_chain+0x1b/0x36
   [<ffff82d0402054f0>] F cpu_down+0xa7/0x143
   [<ffff82d040348c25>] F cpu_down_helper+0x11/0x27
   [<ffff82d0402071aa>] F 
common/domain.c#continue_hypercall_tasklet_handler+0x50/0xbd
   [<ffff82d0402343c9>] F common/tasklet.c#do_tasklet_work+0x76/0xaf
   [<ffff82d040234631>] F do_tasklet+0x5b/0x8d
   [<ffff82d040321411>] F arch/x86/domain.c#idle_loop+0x78/0xe6
   [<ffff82d040324391>] F continue_running+0x5b/0x5d


****************************************
Panic on CPU 0:
Assertion '!cpumask_test_cpu(cpu, &prv->initialized)' failed at 
common/sched/credit2.c:4111
****************************************

The issue seems to be that since the CPU hasn't been removed, it's
still part of prv->initialized and the assert in csched2_free_pdata()
called as part of free_cpu_rm_data() triggers.

It's easy to reproduce by substituting the stop_machine_run() call in
cpu_down() with an error.

Thanks, Roger.