Status of S0ix with Xen

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

S0ix came up in a recent discussion, so let me post a small status
update. We have patches that makes Qubes suspend with ~90% S0ix
residency. It's significantly worse than native Linux on the same system
- that gets 99+%, but still a significant progress.
We do have a tricky case though, because we use PCI passthrough, with
stubdomains.

A summary of changes required is posted by Simon in a tracking issue[1],
but let me post it here too:

    So currently on my test system (NUC11TNHi5) I'm being able to reach
    S0ix residency with only dom0 running for nearly 90 % of the suspend to
    idle time. (That's still worse than native Linux with > 99 %.)

    Used changes:

     - Xen
       - Disable HPET legacy mode after it has been userd for IOAPIC
         test during boot ([upstream 
discussion](https://lore.kernel.org/xen-devel/cb408368-077d-edb5-b4ad-f80086db48c1@xxxxxxxxxxxxxxxxxxxxxx/),
         including minimally required diff)
       - Tiger Lake support in Xen's mwait-idle driver. Unlike Linux'
         version of that driver Xen's only supports the CPUs for which
         it contains a hardcoded config. For the moment I just manualy added the
         config Linux is reading from ACPI. Needs to be tested if this is
         actually required (see also
         [Jan's 
comment](https://lore.kernel.org/xen-devel/f6c27788-bdd9-e5b1-a874-7f48a27c66a9@xxxxxxxx/))
       - Allow reading of S0ix related MSRs. Not sure if any of them is
         strictly required to get into S0ix states, but at the very
         least you want them to be able to debug things (for example via
         `/sys/kernel/debug/pmc_core/...`)
       - Teach `cpu_idle.c` that Tiger Lake also has PC{8..10} (for proper 
debug output)
       - Disable `ondemand` cpufreq governor (both `powersave` as well as 
`perfromance` work)
     - Linux in dom0
       - Add wakup support to `xen-pirq`
         
([patch](https://lore.kernel.org/xen-devel/20230313134102.3157-1-simon@xxxxxxxxxxxxxxxxxxxxxx/)
 sent upstream)
       - If using a SATA device you need to enable low power mode for
         link management (`echo min_power > 
/sys/class/scsi_host/host0/link_power_management_policy`, tried to get
         it [enabled by 
default](https://lore.kernel.org/linux-ide/20230213102450.1604-1-simon@xxxxxxxxxxxxxxxxxxxxxx/)
         but that's
         [not so 
easy](https://lore.kernel.org/linux-ide/ad02467d-d623-5933-67e0-09925c185568@xxxxxxxxxxxxx))

    First tests with a running guest and investigating the difference to
    native Linux are progress (the later is probably due to some timer).

Since the above update, few more things came up:
1. Thunderbolt driver in Linux does not support suspending when kernel
is built with CONFIG_HOTPLUG_PCI=n (which is the case in Qubes). We have
a hacky workaround at [2] (the first patch), together with more details
about the issue in the patch description.

2. When toolstack issues "suspend" command (via xenstore
control/shutdown), Linux uses "freeze" path, that doesn't put devices
into low power state. Changing that to "suspend" path helps (the second
patch in [2]), and for our case seems to have no negative consequences.
But we do not use live migration nor save/restore - which is another use
of this feature, and we have not tested if this still works.

3. QEMU emulates (instead of passthrough) power management related
config space. This is fixed at [3]

The relevant patches are all linked in pull requests connected to the issue.

This is still work in progress. Some of the patches were already posted
to relevant mailing lists, other still require some more work and will
be posted later. I'm posting it here mostly to share that we are working
on it. But also, if anybody wants to try it out and maybe help
improving, the patches are out there.

[1] https://github.com/QubesOS/qubes-issues/issues/6411#issuecomment-1538089344
[2] https://github.com/QubesOS/qubes-linux-kernel/pull/910/files
[3] https://github.com/QubesOS/qubes-vmm-xen-stubdom-linux/pull/63/files
-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature