Re: [PATCH] xen/x86: Change stub page freeing to fix smt=0

[Jason Andryuk <jason.andryuk@xxxxxxx>]

On 2026-05-26 18:03, Andrew Cooper wrote:
> On 26/05/2026 9:31 pm, Jason Andryuk wrote:
> > A single stubs page is initialized with 0xcc and re-used, with multiple
> > CPUs each using a portion of the shared page.  In cpu_smpboot_free(),
> > each stubs area is checked against 0xcc.  When all are set to 0xcc, the
> > page is freed.
> >
> > Booting a system with smt=0, CPU0 is initially setup, allocating the
> > stubs page and initializing to 0xcc.  When more CPUs are brought up,
> > CPU1 is initialized and then immediately brough offline as it is the
> > sibling of CPU0.  Since the page was initially memset with 0xcc,
> > cpu_smpboot_free() finds all stubs as 0xcc and frees the page.
> > However, the page is still assigned to CPU0 and continues to be assigned
> > to other CPUs.
> It's more complicated than this.
>
> With CONFIG_PV (and !opt_fred in 4.22 which is perhaps newer than you're
> testing), the LSTAR and CSTAR stubs guarantee that the 0xcc's are
> overwritten with real instructions.
>
> In !CONFIG_PV, the 0xcc's only get overwritten by the exception recovery
> selftests (CPU0 only, and gated on CONFIG_SELF_TESTS), and "complicated"
> instructions in the emulator (which in your safety environment, you
> likely have compiled out).
>
> So, in your environment, I think you probably can exclude the stubs
> entirely and trim even more LoC.
Thanks.  Ok, my build was !CONFIG_PV, so 0xcc's were not overwritten. 
The fault happened before the self tests ran.
> > Meanwhile the page can be reallocated, which can lead to misbehavior.
> > The particular instance was the stubs page re-used as a page table which
> > later faulted when the entry was all 0xcc.
> >
> > Change to initializing the page as 0xd6/STUB_BUF_FREE, and initializing
> > individual stubs as 0xcc/STUB_BUF_USED.  0xd6 now indicates unused, and
> > 0xcc indicates used/assigned.  When freeing a CPU, the stub is set to
> > 0xd6, and the page is freed if all stubs are 0xd6.  Initializing with
> > STUB_BUF_FREE lets cpu_smpboot_free() a page that was only ever
> > partially used.
> >
> > 0xd6/UDB is a 1 byte invalid opcode, which is similar to the existing
> > use of 0xcc.  0xd6 is used to identify bug frames, but the stub addr
> > (e.g. 0xffff82d07fffe000) fails the is_active_kernel_text() check.  It
> > should be okay to use here.
> >
> > Fixes: 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
> > Signed-off-by: Jason Andryuk <jason.andryuk@xxxxxxx>
> > ---
> > It would be nice to use get_page()/put_page() to let count_info handle
> > reference counting, but they require an owning domain.
> >
> > The listed Fixes introduced the use of 0xcc, but the smt commit may have
> > made it more problematic.
> > Fixes: d8f974f1a646 ("x86: command line option to avoid use of secondary 
> > hyper-threads")
> Honestly, I dislike all of this "try to free data for going-offline
> CPUs".  It is both complex and a non-stop source of bugs for tantamount
> to 0 benefit.
>
> On x86, we must boot all CPUs we find in the MADT.  You're seeing this
> behaviour already.  This is because if an #MC hits any group of CPUs
> where any CR4.MCE=0, it's an instant reset.
>
> For this reason, firmware doesn't hand APs over to the OS in the
> Wait-for-SIPI state (which resets CR4 to 0); they're in MWAIT or IO-wait
> typically these days, using firmware provided stacks.  But firmware
> cannot handle an #MC intended for the OS, so the OS must set up stacks
> and at least an NMI and #MC handler even for those CPUs not wanting to run.
>
> This what park_offline_cpus is trying to do, and while it's set for
> Intel and clear for AMD, I'm pretty sure this is a bug on AMD because
> you can still get MCEs with core-scope groups.
>
>
> Beyond that,  smt=0 is an emergency bodge for speculation safety, which
> is always better done by changing SMT settings in the firmware.
I was asked to check something with SMT disabled, and I could not find 
SMT in my firmware.  I looked for a while without success, and then set 
smt=0 :/
> xen-hptool is useful for testing but it's not a thing anyone uses in a
> production system.
>
> ACPI CPU hot-add does exist in virtual environments, but hot-remove is
> theoretical at best.  I've not seen any evidence of ACPI hotplug
> actually working on Xen, and I think the chances that it does are slim;
> it requires AML execution, and is right in the middle of the split-brain
> problem with physical vs virtual details that dom0 suffers.
>
>
> So, lets just allocate the stubs and "leak" them in testing scenarios.
> It removes bugs and removes code, and has no effect on well-configured
> systems (where cpu offline is not used in practice).
Ok.

Thanks,
Jason