Re: [PATCH 0/2] Xen real-time x86

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Jul 09, 2025 at 05:44:33PM -0700, Stefano Stabellini wrote:
> On Wed, 9 Jul 2025, Jan Beulich wrote:
> > On 08.07.2025 19:11, Stefano Stabellini wrote:
> > > On Tue, 8 Jul 2025, Jan Beulich wrote:
> > >> On 08.07.2025 12:11, Roger Pau Monné wrote:
> > >>> On Mon, Jul 07, 2025 at 05:06:53PM -0700, Stefano Stabellini wrote:
> > >>>> Hi all,
> > >>>>
> > >>>> This short patch series improves Xen real-time execution on AMD x86
> > >>>> processors.
> > >>>>
> > >>>> The key to real-time performance is deterministic guest execution times
> > >>>> and deterministic guest interrupt latency. In such configurations, the
> > >>>> null scheduler is typically used, and there should be no IPIs or other
> > >>>> sources of vCPU execution interruptions beyond the guest timer 
> > >>>> interrupt
> > >>>> as configured by the guest, and any passthrough interrupts for
> > >>>> passthrough devices.
> > >>>>
> > >>>> This is because, upon receiving a critical interrupt, the guest (such 
> > >>>> as
> > >>>> FreeRTOS or Zephyr) typically has a very short window of time to
> > >>>> complete the required action. Being interrupted in the middle of this
> > >>>> critical section could prevent the guest from completing the action
> > >>>> within the allotted time, leading to malfunctions.
> > >>>
> > >>> There's IMO still one pending issue after this series on x86, maybe
> > >>> you have addressed this with some local patch.
> > >>
> > >> Not just one, I think. We use IPIs for other purposes as well. The way
> > >> I read the text above, all of them are a (potential) problem.
> > > 
> > > Yes, all of them are potentially a problem. If you know of any other
> > > IPI, please let me know and I'll try to remove them.
> > 
> > INVALIDATE_TLB_VECTOR, EVENT_CHECK_VECTOR, and CALL_FUNCTION_VECTOR, maybe
> > also others in that group of vectors (see irq-vectors.h).
> 
> Thanks Jan, I'll look into those.
> 
> 
> > > One of my goals
> > > posting this series was to raise awareness on this issue and attempting
> > > to fix it with your help. It is not just IPIs, also Xen timers and other
> > > things that could cause the guest to trap into Xen without the guest
> > > knowledge. Typically IPIs are the worst offenders in my experience.
> > > 
> > > On ARM, I have done several experiments where, after the system is
> > > configured correctly, I can see that if the RTOS does nothing, there are
> > > no traps in Xen on the RTOS vCPU/pCPU for seconds.
> > 
> > Being quiescent when the system is idle is only part of the overall
> > requirement, though?
> 
> Actually being quiescent when the system is idle is not a requirement.
> 
> The only requirements are:
> 1) quick interrupt injection into the RTOS
> 2) the RTOS must be undisturbed while executing the critical region
> 
> 1) mostly means that the physical interrupt should be delivered to the
> same pCPU running the RTOS vCPU. Otherwise the extra IPI causes unwanted
> delays.

This should already be the case, in the Xen model interrupts follow
vCPUs, so if you use pinning the vCPU should always be running
on the pCPU that's the target of the physical interrupt.

> 2) means that the RTOS must be undisturbed when executing the critical
> section, which is typically right after receiving the interrupt and only
> last for less than 1ms. In practice, it means the RTOS should absolutely
> not be descheduled and there should be no (unnecessary) traps into Xen
> while the RTOS is executing the critical section. It is expected that
> the RTOS will run the critical section with interrupts disabled.

What about other external interrupts?  While the guest runs the
critical interrupt handling section with interrupts disabled, an
external interrupt from a device targeting the pCPU could cause a
vmexit.  I'm not aware of a nice way to solve this however, as for
PVH/HVM Xen doesn't know when the guest has finished interrupt
processing (iret).  Maybe this is not an issue in practice if you
isolate interrupts to different vCPUs (you might have to do this
already to ensure deterministic latency).

Roger.