Re: domU suspend issue - freeze processes failed - Linux 6.16

[Jason Andryuk <jason.andryuk@xxxxxxx>]

On 2025-09-24 10:28, Yann Sionneau wrote:
> On 9/24/25 15:30, Marek Marczykowski-Górecki wrote:
> > On Wed, Sep 24, 2025 at 01:17:15PM +0300, Grygorii Strashko wrote:
> > > On 22.09.25 13:09, Marek Marczykowski-Górecki wrote:
> > > > On Fri, Aug 22, 2025 at 08:42:30PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > On Fri, Aug 22, 2025 at 05:27:20PM +0200, Jürgen Groß wrote:
> > > > > > On 22.08.25 16:42, Marek Marczykowski-Górecki wrote:
> > > > > > > On Fri, Aug 22, 2025 at 04:39:33PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > When suspending domU I get the following issue:
> > > > > > > >
> > > > > > > >         Freezing user space processes
> > > > > > > >         Freezing user space processes failed after 20.004 seconds (1 tasks 
> > > > > > > > refusing to freeze, wq_busy=0):
> > > > > > > >         task:xl              state:D stack:0     pid:466   tgid:466   ppid:1    
> > > > > > > >   task_flags:0x400040 flags:0x00004006
> > > > > > > >         Call Trace:
> > > > > > > >          <TASK>
> > > > > > > >          __schedule+0x2f3/0x780
> > > > > > > >          schedule+0x27/0x80
> > > > > > > >          schedule_preempt_disabled+0x15/0x30
> > > > > > > >          __mutex_lock.constprop.0+0x49f/0x880
> > > > > > > >          unregister_xenbus_watch+0x216/0x230
> > > > > > > >          xenbus_write_watch+0xb9/0x220
> > > > > > > >          xenbus_file_write+0x131/0x1b0
> > > > > > > >          vfs_writev+0x26c/0x3d0
> > > > > > > >          ? do_writev+0xeb/0x110
> > > > > > > >          do_writev+0xeb/0x110
> > > > > > > >          do_syscall_64+0x84/0x2c0
> > > > > > > >          ? do_syscall_64+0x200/0x2c0
> > > > > > > >          ? generic_handle_irq+0x3f/0x60
> > > > > > > >          ? syscall_exit_work+0x108/0x140
> > > > > > > >          ? do_syscall_64+0x200/0x2c0
> > > > > > > >          ? __irq_exit_rcu+0x4c/0xe0
> > > > > > > >          entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > > > > > >         RIP: 0033:0x79b618138642
> > > > > > > >         RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
> > > > > > > >         RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
> > > > > > > >         RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
> > > > > > > >         RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
> > > > > > > >         R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
> > > > > > > >         R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
> > > > > > > >          </TASK>
> > > > > > > >         OOM killer enabled.
> > > > > > > >         Restarting tasks: Starting
> > > > > > > >         Restarting tasks: Done
> > > > > > > >         xen:manage: do_suspend: freeze processes failed -16
> > > > > > > >
> > > > > > > > The process in question is `xl devd` daemon. It's a domU serving a
> > > > > > > > xenvif backend.
> > > > > > > >
> > > > > > > > I noticed it on 6.16.1, but looking at earlier test logs I see it with
> > > > > > > > 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
> > > > > > > > seemingly no relevant changes between rc2 and rc6).
> > > > > > > I forgot to include link for (a little) more details:
> > > > > > > https://github.com/QubesOS/qubes-linux-kernel/pull/1157
> > > > > > >
> > > > > > > Especially, there is another call trace with panic_on_warn enabled -
> > > > > > > slightly different, but looks related.
> > > > > > I'm pretty sure the PV variant for suspending is just wrong: it is calling
> > > > > > dpm_suspend_start() from do_suspend() without taking the required
> > > > > > system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
> > > > > >
> > > > > > It might be as easy as just adding the mutex() call to do_suspend(), but I'm
> > > > > > really not sure that will be a proper fix.
> > > > > Hm, this might explain the second call trace, but not the freeze failure
> > > > > quoted here above, I think?
> > > > While the patch I sent appears to fix this particular issue, it made me
> > > > wonder: is there any fundamental reason why do_suspend() is not using
> > > > pm_suspend() and register Xen-specific actions via platform_suspend_ops
> > > > (and maybe syscore_ops)? From a brief look at the code, it should
> > > > theoretically be possible, and should avoid issues like this.
> > > >
> > > > I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
> > > > surely made several mistakes there (and also left a ton of todo
> > > > comments). But before spending any more time at that, I'd like to ask
> > > > if this is a viable option at all.
> > > I think it might, but be careful with this, because there are two "System Low 
> > > power" paths in Linux
> > > 1) Suspend2RAM and Co
> > > 2) Hybernation
> > >
> > > While "Suspend2RAM and Co" path is relatively straight forward and expected to 
> > > be always
> > > started through pm_suspend(). In general, it's expected to happen
> > >    - from sysfs (User space)
> > >    - from autosuspend (wakelocks).
> > >
> > > the "hibernation" path is more complicated:(
> > > - Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
> > IIUC hibernation is very different as it puts Linux in charge of dumping
> > all the state to the disk. In case of Xen, the primary use case for
> > suspend is preparing VM for Xen toolstack serializing its state to disk
> > (or migrating to another host).
> > Additionally, VM suspend may be used as preparation for host suspend
> > (this is what I actually do here). This is especially relevant if the VM
> > has some PCI passthrough - to properly suspend (and resume) devices
> > across host suspend.
> >
> > > I'm not sure what path Xen originally implemented :( It seems like 
> > > "suspend2RAM",
> > > but, at the same time "hybernation" specific staff is used, like 
> > > PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
> > > As result, Linux suspend/hybernation code moves forward while Xen stays behind 
> > > and unsync.
> > Yeah, I think it's supposed to be suspend2RAM. TBH the
> > PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a
> > patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
> >
> > > So it sounds reasonable to avoid custom implementation, but may be not easy :(
> > >
> > > Suspending Xen features can be split between suspend stages, but
> > > not sure if platform_suspend_ops can be used.
> > >
> > > Generic suspend stages list
> > > - freeze
> > > - prepare
> > > - suspend
> > > - suspend_late
> > > - suspend_noirq (SPIs disabled, except wakeups)
> > >     [most of Xen specific staff has to be suspended at this point]
> > > - disable_secondary_cpus
> > > - arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
> > > - syscore_suspend
> > >     [rest here]
> > > - platform->enter() (suspended)
> > >
> > > You can't just overwrite platform_suspend_ops, because ARM64 is expected to 
> > > enter
> > > suspend through PSCI FW interface:
> > > drivers/firmware/psci/psci.c
> > >    static const struct platform_suspend_ops psci_suspend_ops = {
> > Does this apply to a VM on ARM64 too? At least on x86, the VM is
> > supposed to make a hypercall to tell Xen it suspended (the hypercall
> > will return only on resume).
> >
> > > As an option, some Xen components could be converted to use syscore_ops (but 
> > > not xenstore),
> > > and some might need to use DD(dev_pm_ops).
> > >
> > > > [1] 
> > > > https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
> > > --
> > > Best regards,
> > > -grygorii
> > [2] 
> > https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
> On my setup I get a weird behavior when trying to suspend (s2idle) a
> Linux guest.
> Doing echo freeze > /sys/power/state in the guest seems to "freeze" the
> guest for good, I could not unfreeze it afterward.
> VCPU goes to 100% according to XenOrchestra
> xl list shows state "r" but xl console blocks forever
> xl shutdown would block for some time and then print:
> Shutting down domain 721
> ?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge
> control request: -9
> shutdown failed (rc=-9)
>
> Do you think it's related to your current issue?
idle=halt on the Linux command line addresses the 100% CPU usage.  Or 
alternatively C2 needs to be implemented for guest vcpus.  I forget 
preceisely, but I think the 100% CPU is because there are no C-states 
available and Linux/cpuidle won't use halt by default.
To wake up, you need a wake up source.  The ACPI buttons presses will do 
that:
xl trigger $dom power
xl trigger $dom sleep

However, I think without changes, domU s2idle/S3 will detach all its PV 
devices.  Naturally they don't get reconnected on resume.  You can hack 
around that to skip the detach.
Actually, maybe we just need:
--- i/drivers/xen/xenbus/xenbus_probe_frontend.c
+++ w/drivers/xen/xenbus/xenbus_probe_frontend.c
@@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct 
device *_dev)
 }

 static const struct dev_pm_ops xenbus_pm_ops = {
-       .suspend        = xenbus_dev_suspend,
-       .resume         = xenbus_frontend_dev_resume,
        .freeze         = xenbus_dev_suspend,
        .thaw           = xenbus_dev_cancel,
-       .restore        = xenbus_dev_resume,
+       .restore        = xenbus_frontend_dev_resume,
 };

b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for 
suspend/resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to 
PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks 
remained.  But freeze and suspend being identical doesn't seem correct.
This would leave xl save/restore/migrate using the hibernate 
freeze/thaw/resume.  S3/s2idle would no touch the PV devices, so they 
would still be present on resume.  Maybe there are cases I am not 
thinking of though.
Regards,
Jason