Re: Xen crash in scheduler during cpu hotplug

[Jan Beulich <jbeulich@xxxxxxxx>]

On 15.05.2024 13:58, Andrew Cooper wrote:
> Just so it doesn't get lost.  In XenRT, we've found:
> 
>> (XEN) ----[ Xen-4.19.0-1-d  x86_64  debug=y  Tainted:     H  ]----
>> (XEN) CPU:    45
>> (XEN) RIP:    e008:[<ffff82d040244cbf>]
>> common/sched/credit.c#csched_load_balance+0x41/0x877
>> (XEN) RFLAGS: 0000000000010092   CONTEXT: hypervisor
>> (XEN) rax: ffff82d040981618   rbx: ffff82d040981618   rcx:
>> 0000000000000000
>> (XEN) rdx: 0000003ff68cd000   rsi: 000000000000002d   rdi:
>> ffff83103723d450
>> (XEN) rbp: ffff83207caa7d48   rsp: ffff83207caa7b98   r8: 
>> 0000000000000000
>> (XEN) r9:  ffff831037253cf0   r10: ffff83103767c3f0   r11:
>> 0000000000000009
>> (XEN) r12: ffff831037237990   r13: ffff831037237990   r14:
>> ffff831037253720
>> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4:
>> 0000000000f526e0
>> (XEN) cr3: 000000005bc2f000   cr2: 0000000000000010
>> (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss:
>> 0000000000000000
>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>> (XEN) Xen code around <ffff82d040244cbf>
>> (common/sched/credit.c#csched_load_balance+0x41/0x877):
>> (XEN)  48 8b 0c 10 48 8b 49 08 <48> 8b 79 10 48 89 bd b8 fe ff ff 49
>> 8b 4e 28 48
>> <snip>
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82d040244cbf>] R
>> common/sched/credit.c#csched_load_balance+0x41/0x877

While this is of course pretty little information, I've still tried to
decipher it, first noticing it's credit1 that's being used here. Once
forcing csched_load_balance() non-inline (no idea why it is a separate
function in your build), I can see a sufficiently matching pattern at
approximately the same offset into the function. That's

    const struct cpupool *c = get_sched_res(cpu)->cpupool;
    ...
    const cpumask_t *online = c->res_valid;
    ...
    BUG_ON(get_sched_res(cpu) != snext->unit->res);

overlapping, with the crash being on the middle of the quoted lines.
IOW the CPU pool is still NULL for this sched resource. Cc-ing
Jürgen for possible clues ...

Jan

>> (XEN)    [<ffff82d040245a18>] F
>> common/sched/credit.c#csched_schedule+0x36a/0x69f
>> (XEN)    [<ffff82d040252644>] F common/sched/core.c#do_schedule+0xe8/0x433
>> (XEN)    [<ffff82d0402572dd>] F common/sched/core.c#schedule+0x2e5/0x2f9
>> (XEN)    [<ffff82d040232f35>] F common/softirq.c#__do_softirq+0x94/0xbe
>> (XEN)    [<ffff82d040232fc8>] F do_softirq+0x13/0x15
>> (XEN)    [<ffff82d0403075ef>] F arch/x86/domain.c#idle_loop+0x92/0xe6
>> (XEN)
>> (XEN) Pagetable walk from 0000000000000010:
>> (XEN)  L4[0x000] = 000000103ffca063 ffffffffffffffff
>> (XEN)  L3[0x000] = 000000103ffc9063 ffffffffffffffff
>> (XEN)  L2[0x000] = 000000103ffc8063 ffffffffffffffff
>> (XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 45:
>> (XEN) FATAL PAGE FAULT
>> (XEN) [error_code=0000]
>> (XEN) Faulting linear address: 0000000000000010
>> (XEN) ****************************************
> 
> which we can reproduce reliably enough with a shell script doing
> xen-hptool cpu-{off,on}line.
> 
> ~Andrew
>