[Xen-devel] IPI delivery showing very long latency

["Jan Beulich" <JBeulich@xxxxxxxx>]

(re-sending with xen-devel no longer left off the Cc-list, and also with
one further bit of information)

All,

as you may have seen mentioned in other threads, we're observing
very long latencies resulting from APIC ICR writes taking unexpected
amounts of time when the remote CPU is in a (perhaps deep) C-state:

Nehalem:                120us (waiting for the ICR to become idle again)
Dinar:          186us (dito)
Ivybridge:      230us (for the WRMSR to ICR to complete)

as opposed to Sandybridge, where the WRMSR to ICR completes in
at most 25us (which is still close to 50,000 cycles).

While I've been working on avoidance strategies for the most
severe observed case (vlapic_ipi() broadcasting to all vCPU-s of
a guest with beyond 40 of them), I'm relatively certain this is not
the only place this could bite us going forward, and hence I'm
(a) trying to understand the reasons for this and (b) looking for
suggestions on how to avoid this (other than turning off C-state
management).

This is in particular because at least on the Ivybridge system we're
seeing occasional spikes on the ICR write completion latency of up
to 1.5ms even with "cpuidle=0".

Thanks, Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel