Re: IOMMU faults after S3

[Jan Beulich <jbeulich@xxxxxxxx>]

On 01.04.2026 09:14, Jan Beulich wrote:
> On 27.03.2026 11:19, Marek Marczykowski-Górecki wrote:
>> I noticed that on some systems, there are a lot of IOMMU faults after
>> S3. I can see it also on a laptop with MTL, but it affects also the ADL
>> gitlab runner:
>>
>>     https://gitlab.com/xen-project/hardware/xen/-/jobs/13661033722
>>     (XEN) [   37.201160] [VT-D]DMAR:[DMA Write] Request device 
>> [0000:00:1e.6] fault addr 0
>>     (XEN) [   37.201164] [VT-D]DMAR: reason 02 - Present bit in context 
>> entry is clear
>>     (XEN) [   37.202332] [VT-D]DMAR:[DMA Write] Request device 
>> [0000:00:1e.6] fault addr 0
>>     (XEN) [   37.202339] [VT-D]DMAR: reason 02 - Present bit in context 
>> entry is clear
>>
>> Interestingly, the 0000:00:1e.6 device is not even listed by lspci.
>>
>> The issue is present only on staging, not staging-4.21.
>>
>> Bisect says:
>>
>> 5ec93b2f19ff8873fca65d38c1164b0a56d3898b is the first bad commit
>> commit 5ec93b2f19ff8873fca65d38c1164b0a56d3898b
>> Author: Jan Beulich <jbeulich@xxxxxxxx>
>> Date:   Thu Jan 22 14:13:35 2026 +0100
>>
>>     x86/HPET: drop .set_affinity hook
> 
> Looking into this, I find several things I can't quite understand (yet).
> First there is
> 
> (XEN) [000000456c0fe39f] Disabling HPET for being unreliable
> 
> which looks to only affect clocksource selection, but not use as
> broadcast source for CPU-idle management. (This may be an independent
> issue.)
> 
> Then there is
> 
> (XEN) [    2.760248] HPET: 8 timers usable for broadcast (8 total)
> 
> which should only occur on ARAT-incapable systems. That should only be
> older hardware. (On my much older Skylake I don't see this line, for
> example.) What does CPUID leaf 6 have on this system? Sadly xen-cpuid
> is purely featureset based, and hence doesn't expose info about that
> leaf. The leaf also isn't exposed to domains, so CPUID output in Dom0
> isn't useful to look at either. It would need to be CPUID output on a
> bare metal kernel.
> 
> Further I suspect the fingered commit may only have uncovered an issue
> elsewhere. I don't think we clear any context table entries during
> suspend or resume. Hence in
> 
> (XEN) [   20.554813] [VT-D]DMAR:[DMA Write] Request device [0000:00:1e.6] 
> fault addr 0
> (XEN) [   20.554819] [VT-D]DMAR: reason 02 - Present bit in context entry is 
> clear
> 
> the latter message is confusing me.
> 
> The fault address being zero may, otoh, be a hint of hpet_msi_write()
> never having run post-resume. Which may be the connection to the
> dropping of hpet_msi_set_affinity(), as that did call that function.

There clearly is an issue with the handling of the max_cstate variable,
but I expect you don't use xenpm to limit usable C-states (there clearly
is no respective command line option in the log you referenced)?

>From what the log has, I conclude hpet_broadcast_resume() is called.
Question is whether it does what we want it to. Could you instrument it
some, so we have confirmation that it is called, and we also know whether
__hpet_setup_msi_irq() is not only called on all 8 channels, but also
succeeds there? (If it failed, I suppose we better wouldn't set
HPET_TN_FSB and/or HPET_TN_ENABLE.) If, however, it succeeds, I couldn't
explain why the fault address would be reported as 0, as then we
definitely must have written HPET_Tn_ROUTE.

Jan