Re: IOMMU faults after S3

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Wed, Apr 01, 2026 at 10:52:37AM +0200, Jan Beulich wrote:
> On 01.04.2026 09:14, Jan Beulich wrote:
> > On 27.03.2026 11:19, Marek Marczykowski-Górecki wrote:
> >> I noticed that on some systems, there are a lot of IOMMU faults after
> >> S3. I can see it also on a laptop with MTL, but it affects also the ADL
> >> gitlab runner:
> >>
> >>     https://gitlab.com/xen-project/hardware/xen/-/jobs/13661033722
> >>     (XEN) [   37.201160] [VT-D]DMAR:[DMA Write] Request device 
> >> [0000:00:1e.6] fault addr 0
> >>     (XEN) [   37.201164] [VT-D]DMAR: reason 02 - Present bit in context 
> >> entry is clear
> >>     (XEN) [   37.202332] [VT-D]DMAR:[DMA Write] Request device 
> >> [0000:00:1e.6] fault addr 0
> >>     (XEN) [   37.202339] [VT-D]DMAR: reason 02 - Present bit in context 
> >> entry is clear
> >>
> >> Interestingly, the 0000:00:1e.6 device is not even listed by lspci.
> >>
> >> The issue is present only on staging, not staging-4.21.
> >>
> >> Bisect says:
> >>
> >> 5ec93b2f19ff8873fca65d38c1164b0a56d3898b is the first bad commit
> >> commit 5ec93b2f19ff8873fca65d38c1164b0a56d3898b
> >> Author: Jan Beulich <jbeulich@xxxxxxxx>
> >> Date:   Thu Jan 22 14:13:35 2026 +0100
> >>
> >>     x86/HPET: drop .set_affinity hook
> > 
> > Looking into this, I find several things I can't quite understand (yet).
> > First there is
> > 
> > (XEN) [000000456c0fe39f] Disabling HPET for being unreliable
> > 
> > which looks to only affect clocksource selection, but not use as
> > broadcast source for CPU-idle management. (This may be an independent
> > issue.)
> > 
> > Then there is
> > 
> > (XEN) [    2.760248] HPET: 8 timers usable for broadcast (8 total)
> > 
> > which should only occur on ARAT-incapable systems. That should only be
> > older hardware. (On my much older Skylake I don't see this line, for
> > example.) What does CPUID leaf 6 have on this system? Sadly xen-cpuid
> > is purely featureset based, and hence doesn't expose info about that
> > leaf. The leaf also isn't exposed to domains, so CPUID output in Dom0
> > isn't useful to look at either. It would need to be CPUID output on a
> > bare metal kernel.
> > 
> > Further I suspect the fingered commit may only have uncovered an issue
> > elsewhere. I don't think we clear any context table entries during
> > suspend or resume. Hence in
> > 
> > (XEN) [   20.554813] [VT-D]DMAR:[DMA Write] Request device [0000:00:1e.6] 
> > fault addr 0
> > (XEN) [   20.554819] [VT-D]DMAR: reason 02 - Present bit in context entry 
> > is clear
> > 
> > the latter message is confusing me.
> > 
> > The fault address being zero may, otoh, be a hint of hpet_msi_write()
> > never having run post-resume. Which may be the connection to the
> > dropping of hpet_msi_set_affinity(), as that did call that function.
> 
> There clearly is an issue with the handling of the max_cstate variable,
> but I expect you don't use xenpm to limit usable C-states (there clearly
> is no respective command line option in the log you referenced)?

No, I don't think so.

> From what the log has, I conclude hpet_broadcast_resume() is called.

I don't think so... I applied changes as attached and got this on
resume:

(XEN) [   69.486120] Enabling non-boot CPUs  ...
(XEN) [   69.486404] mwait-idle: state C1 is disabled
(XEN) [   69.587869] mwait-idle: state C1 is disabled
(XEN) [   69.588008] mwait-idle: state C1 is disabled
(XEN) [   69.689438] mwait-idle: state C1 is disabled
(XEN) [   69.689608] mwait-idle: state C1 is disabled
(XEN) [   69.791066] mwait-idle: state C1 is disabled
(XEN) [   69.791334] mwait-idle: state C1 is disabled
(XEN) [   69.892938] mwait-idle: state C1 is disabled
(XEN) [   69.893209] mwait-idle: state C1 is disabled
(XEN) [   69.994890] mwait-idle: state C1 is disabled
(XEN) [   69.995096] mwait-idle: state C1 is disabled
(XEN) [   70.096638] mwait-idle: state C1 is disabled
(XEN) [   70.096915] mwait-idle: state C1 is disabled
(XEN) [   70.097093] mwait-idle: state C1 is disabled
(XEN) [   70.097272] mwait-idle: state C1 is disabled
(XEN) [   70.203357] [VT-D]DMAR:[DMA Write] Request device [0000:00:1e.6] fault 
addr 0
(XEN) [   70.203363] [VT-D]DMAR: reason 02 - Present bit in context entry is 
clear

> Question is whether it does what we want it to. Could you instrument it
> some, so we have confirmation that it is called, and we also know whether
> __hpet_setup_msi_irq() is not only called on all 8 channels, but also
> succeeds there? (If it failed, I suppose we better wouldn't set
> HPET_TN_FSB and/or HPET_TN_ENABLE.) If, however, it succeeds, I couldn't
> explain why the fault address would be reported as 0, as then we
> definitely must have written HPET_Tn_ROUTE.
> 
> Jan

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: xen-debug.diff
Description: Text document

Attachment: signature.asc
Description: PGP signature