Re: IOMMU faults after S3

[Jan Beulich <jbeulich@xxxxxxxx>]

On 02.04.2026 01:17, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 01, 2026 at 10:52:37AM +0200, Jan Beulich wrote:
>> On 01.04.2026 09:14, Jan Beulich wrote:
>>> On 27.03.2026 11:19, Marek Marczykowski-Górecki wrote:
>>>> I noticed that on some systems, there are a lot of IOMMU faults after
>>>> S3. I can see it also on a laptop with MTL, but it affects also the ADL
>>>> gitlab runner:
>>>>
>>>>     https://gitlab.com/xen-project/hardware/xen/-/jobs/13661033722
>>>>     (XEN) [   37.201160] [VT-D]DMAR:[DMA Write] Request device 
>>>> [0000:00:1e.6] fault addr 0
>>>>     (XEN) [   37.201164] [VT-D]DMAR: reason 02 - Present bit in context 
>>>> entry is clear
>>>>     (XEN) [   37.202332] [VT-D]DMAR:[DMA Write] Request device 
>>>> [0000:00:1e.6] fault addr 0
>>>>     (XEN) [   37.202339] [VT-D]DMAR: reason 02 - Present bit in context 
>>>> entry is clear
>>>>
>>>> Interestingly, the 0000:00:1e.6 device is not even listed by lspci.
>>>>
>>>> The issue is present only on staging, not staging-4.21.
>>>>
>>>> Bisect says:
>>>>
>>>> 5ec93b2f19ff8873fca65d38c1164b0a56d3898b is the first bad commit
>>>> commit 5ec93b2f19ff8873fca65d38c1164b0a56d3898b
>>>> Author: Jan Beulich <jbeulich@xxxxxxxx>
>>>> Date:   Thu Jan 22 14:13:35 2026 +0100
>>>>
>>>>     x86/HPET: drop .set_affinity hook
>>>
>>> Looking into this, I find several things I can't quite understand (yet).
>>> First there is
>>>
>>> (XEN) [000000456c0fe39f] Disabling HPET for being unreliable
>>>
>>> which looks to only affect clocksource selection, but not use as
>>> broadcast source for CPU-idle management. (This may be an independent
>>> issue.)
>>>
>>> Then there is
>>>
>>> (XEN) [    2.760248] HPET: 8 timers usable for broadcast (8 total)
>>>
>>> which should only occur on ARAT-incapable systems. That should only be
>>> older hardware. (On my much older Skylake I don't see this line, for
>>> example.) What does CPUID leaf 6 have on this system? Sadly xen-cpuid
>>> is purely featureset based, and hence doesn't expose info about that
>>> leaf. The leaf also isn't exposed to domains, so CPUID output in Dom0
>>> isn't useful to look at either. It would need to be CPUID output on a
>>> bare metal kernel.
>>>
>>> Further I suspect the fingered commit may only have uncovered an issue
>>> elsewhere. I don't think we clear any context table entries during
>>> suspend or resume. Hence in
>>>
>>> (XEN) [   20.554813] [VT-D]DMAR:[DMA Write] Request device [0000:00:1e.6] 
>>> fault addr 0
>>> (XEN) [   20.554819] [VT-D]DMAR: reason 02 - Present bit in context entry 
>>> is clear
>>>
>>> the latter message is confusing me.
>>>
>>> The fault address being zero may, otoh, be a hint of hpet_msi_write()
>>> never having run post-resume. Which may be the connection to the
>>> dropping of hpet_msi_set_affinity(), as that did call that function.
>>
>> There clearly is an issue with the handling of the max_cstate variable,
>> but I expect you don't use xenpm to limit usable C-states (there clearly
>> is no respective command line option in the log you referenced)?
> 
> No, I don't think so.
> 
>> From what the log has, I conclude hpet_broadcast_resume() is called.
> 
> I don't think so... I applied changes as attached and got this on
> resume:
> 
> (XEN) [   69.486120] Enabling non-boot CPUs  ...
> (XEN) [   69.486404] mwait-idle: state C1 is disabled
> (XEN) [   69.587869] mwait-idle: state C1 is disabled
> (XEN) [   69.588008] mwait-idle: state C1 is disabled
> (XEN) [   69.689438] mwait-idle: state C1 is disabled
> (XEN) [   69.689608] mwait-idle: state C1 is disabled
> (XEN) [   69.791066] mwait-idle: state C1 is disabled
> (XEN) [   69.791334] mwait-idle: state C1 is disabled
> (XEN) [   69.892938] mwait-idle: state C1 is disabled
> (XEN) [   69.893209] mwait-idle: state C1 is disabled
> (XEN) [   69.994890] mwait-idle: state C1 is disabled
> (XEN) [   69.995096] mwait-idle: state C1 is disabled
> (XEN) [   70.096638] mwait-idle: state C1 is disabled
> (XEN) [   70.096915] mwait-idle: state C1 is disabled
> (XEN) [   70.097093] mwait-idle: state C1 is disabled
> (XEN) [   70.097272] mwait-idle: state C1 is disabled
> (XEN) [   70.203357] [VT-D]DMAR:[DMA Write] Request device [0000:00:1e.6] 
> fault addr 0
> (XEN) [   70.203363] [VT-D]DMAR: reason 02 - Present bit in context entry is 
> clear

That was on the serial console or from xl dmesg? I ask because console_resume()
runs after time_resume(), so nothing appearing on the serial console would be
expected (I think).

Without hpet_broadcast_resume() running, I don't think I could explain how the
channels (and their FSB interrupts) would get enabled.

Jan