Re: [PATCH 0/2] Xen real-time x86

[Stefano Stabellini <stefano.stabellini@xxxxxxx>]

On Wed, 9 Jul 2025, Jan Beulich wrote:
> On 08.07.2025 19:11, Stefano Stabellini wrote:
> > On Tue, 8 Jul 2025, Jan Beulich wrote:
> >> On 08.07.2025 12:11, Roger Pau Monné wrote:
> >>> On Mon, Jul 07, 2025 at 05:06:53PM -0700, Stefano Stabellini wrote:
> >>>> Hi all,
> >>>>
> >>>> This short patch series improves Xen real-time execution on AMD x86
> >>>> processors.
> >>>>
> >>>> The key to real-time performance is deterministic guest execution times
> >>>> and deterministic guest interrupt latency. In such configurations, the
> >>>> null scheduler is typically used, and there should be no IPIs or other
> >>>> sources of vCPU execution interruptions beyond the guest timer interrupt
> >>>> as configured by the guest, and any passthrough interrupts for
> >>>> passthrough devices.
> >>>>
> >>>> This is because, upon receiving a critical interrupt, the guest (such as
> >>>> FreeRTOS or Zephyr) typically has a very short window of time to
> >>>> complete the required action. Being interrupted in the middle of this
> >>>> critical section could prevent the guest from completing the action
> >>>> within the allotted time, leading to malfunctions.
> >>>
> >>> There's IMO still one pending issue after this series on x86, maybe
> >>> you have addressed this with some local patch.
> >>
> >> Not just one, I think. We use IPIs for other purposes as well. The way
> >> I read the text above, all of them are a (potential) problem.
> > 
> > Yes, all of them are potentially a problem. If you know of any other
> > IPI, please let me know and I'll try to remove them.
> 
> INVALIDATE_TLB_VECTOR, EVENT_CHECK_VECTOR, and CALL_FUNCTION_VECTOR, maybe
> also others in that group of vectors (see irq-vectors.h).

Thanks Jan, I'll look into those.


> > One of my goals
> > posting this series was to raise awareness on this issue and attempting
> > to fix it with your help. It is not just IPIs, also Xen timers and other
> > things that could cause the guest to trap into Xen without the guest
> > knowledge. Typically IPIs are the worst offenders in my experience.
> > 
> > On ARM, I have done several experiments where, after the system is
> > configured correctly, I can see that if the RTOS does nothing, there are
> > no traps in Xen on the RTOS vCPU/pCPU for seconds.
> 
> Being quiescent when the system is idle is only part of the overall
> requirement, though?

Actually being quiescent when the system is idle is not a requirement.

The only requirements are:
1) quick interrupt injection into the RTOS
2) the RTOS must be undisturbed while executing the critical region

1) mostly means that the physical interrupt should be delivered to the
same pCPU running the RTOS vCPU. Otherwise the extra IPI causes unwanted
delays.

2) means that the RTOS must be undisturbed when executing the critical
section, which is typically right after receiving the interrupt and only
last for less than 1ms. In practice, it means the RTOS should absolutely
not be descheduled and there should be no (unnecessary) traps into Xen
while the RTOS is executing the critical section. It is expected that
the RTOS will run the critical section with interrupts disabled.

That's pretty much it. If we get this right, we have solved 99% of the
problem.