Re: HVM guest only bring up a single vCPU

[Julien Grall <julien@xxxxxxx>]

Hi Jan,

On 27/08/2021 11:52, Jan Beulich wrote:
> On 27.08.2021 12:35, Julien Grall wrote:
> > Hi Jan,
> >
> > On 27/08/2021 07:28, Jan Beulich wrote:
> > > On 27.08.2021 01:42, Andrew Cooper wrote:
> > > > On 26/08/2021 22:00, Julien Grall wrote:
> > > > > Hi Andrew,
> > > > >
> > > > > While doing more testing today, I noticed that only one vCPU would be
> > > > > brought up with HVM guest with Xen 4.16 on my setup (QEMU):
> > > > >
> > > > > [    1.122180]
> > > > > ================================================================================
> > > > > [    1.122180] UBSAN: shift-out-of-bounds in
> > > > > oss/linux/arch/x86/kernel/apic/apic.c:2362:13
> > > > > [    1.122180] shift exponent -1 is negative
> > > > > [    1.122180] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.14.0-rc7+ #304
> > > > > [    1.122180] Hardware name: Xen HVM domU, BIOS 4.16-unstable 06/07/2021
> > > > > [    1.122180] Call Trace:
> > > > > [    1.122180]  dump_stack_lvl+0x56/0x6c
> > > > > [    1.122180]  ubsan_epilogue+0x5/0x50
> > > > > [    1.122180]  __ubsan_handle_shift_out_of_bounds+0xfa/0x140
> > > > > [    1.122180]  ? cgroup_kill_write+0x4d/0x150
> > > > > [    1.122180]  ? cpu_up+0x6e/0x100
> > > > > [    1.122180]  ? _raw_spin_unlock_irqrestore+0x30/0x50
> > > > > [    1.122180]  ? rcu_read_lock_held_common+0xe/0x40
> > > > > [    1.122180]  ? irq_shutdown_and_deactivate+0x11/0x30
> > > > > [    1.122180]  ? lock_release+0xc7/0x2a0
> > > > > [    1.122180]  ? apic_id_is_primary_thread+0x56/0x60
> > > > > [    1.122180]  apic_id_is_primary_thread+0x56/0x60
> > > > > [    1.122180]  cpu_up+0xbd/0x100
> > > > > [    1.122180]  bringup_nonboot_cpus+0x4f/0x60
> > > > > [    1.122180]  smp_init+0x26/0x74
> > > > > [    1.122180]  kernel_init_freeable+0x183/0x32d
> > > > > [    1.122180]  ? _raw_spin_unlock_irq+0x24/0x40
> > > > > [    1.122180]  ? rest_init+0x330/0x330
> > > > > [    1.122180]  kernel_init+0x17/0x140
> > > > > [    1.122180]  ? rest_init+0x330/0x330
> > > > > [    1.122180]  ret_from_fork+0x22/0x30
> > > > > [    1.122244]
> > > > > ================================================================================
> > > > > [    1.123176] installing Xen timer for CPU 1
> > > > > [    1.123369] x86: Booting SMP configuration:
> > > > > [    1.123409] .... node  #0, CPUs:      #1
> > > > > [    1.154400] Callback from call_rcu_tasks_trace() invoked.
> > > > > [    1.154491] smp: Brought up 1 node, 1 CPU
> > > > > [    1.154526] smpboot: Max logical packages: 2
> > > > > [    1.154570] smpboot: Total of 1 processors activated (5999.99
> > > > > BogoMIPS)
> > > > >
> > > > > I have tried a PV guest (same setup) and the kernel could bring up all
> > > > > the vCPUs.
> > > > >
> > > > > Digging down, Linux will set smp_num_siblings to 0 (via
> > > > > detect_ht_early()) and as a result will skip all the CPUs. The value
> > > > > is retrieve from a CPUID leaf. So it sounds like we don't set the
> > > > > leaft correctly.
> > > > >
> > > > > FWIW, I have also tried on Xen 4.11 and could spot the same issue.
> > > > > Does this ring any bell to you?
> > > > The CPUID data we give to guests is generally nonsense when it comes to
> > > > topology.  By any chance does the hardware you're booting this on not
> > > > have hyperthreading enabled/active to begin with?
> > > Well, I'd put the question slightly differently: What CPUID data does
> > > qemu supply to Xen here? I could easily see us making an assumption
> > > somewhere that is met by all hardware but is theoretically wrong to
> > > make and not met by qemu, which then leads to further issues with what
> > > we expose to our guest.
> > I have pasted the output from cpuid on a baremetal Linux here:
> "baremetal" still meaning it was running on qemu, not itself baremetal?
Correct.

> > https://pastebin.com/WvaXiXuL
>     miscellaneous (1/ebx):
>        process local APIC physical ID = 0x0 (0)
>        maximum IDs for CPUs in pkg    = 0x0 (0)
>        CLFLUSH line size              = 0x8 (8)
>        brand index                    = 0x0 (0)
>
> As suspected the field is zero, and hence will remain zero after
> multiplying by 2. I suppose the patch sent earlier should then get you
> further.
I am about to try your patch. I will let you know the results.

Cheers,

--
Julien Grall