Re: [PATCH 0/2] Xen real-time x86

[Stefano Stabellini <stefano.stabellini@xxxxxxx>]

On Tue, 8 Jul 2025, Jan Beulich wrote:
> On 08.07.2025 12:11, Roger Pau Monné wrote:
> > On Mon, Jul 07, 2025 at 05:06:53PM -0700, Stefano Stabellini wrote:
> >> Hi all,
> >>
> >> This short patch series improves Xen real-time execution on AMD x86
> >> processors.
> >>
> >> The key to real-time performance is deterministic guest execution times
> >> and deterministic guest interrupt latency. In such configurations, the
> >> null scheduler is typically used, and there should be no IPIs or other
> >> sources of vCPU execution interruptions beyond the guest timer interrupt
> >> as configured by the guest, and any passthrough interrupts for
> >> passthrough devices.
> >>
> >> This is because, upon receiving a critical interrupt, the guest (such as
> >> FreeRTOS or Zephyr) typically has a very short window of time to
> >> complete the required action. Being interrupted in the middle of this
> >> critical section could prevent the guest from completing the action
> >> within the allotted time, leading to malfunctions.
> > 
> > There's IMO still one pending issue after this series on x86, maybe
> > you have addressed this with some local patch.
> 
> Not just one, I think. We use IPIs for other purposes as well. The way
> I read the text above, all of them are a (potential) problem.

Yes, all of them are potentially a problem. If you know of any other
IPI, please let me know and I'll try to remove them. One of my goals
posting this series was to raise awareness on this issue and attempting
to fix it with your help. It is not just IPIs, also Xen timers and other
things that could cause the guest to trap into Xen without the guest
knowledge. Typically IPIs are the worst offenders in my experience.

On ARM, I have done several experiments where, after the system is
configured correctly, I can see that if the RTOS does nothing, there are
no traps in Xen on the RTOS vCPU/pCPU for seconds.

As I tried to describe in the email, typically the real time
application, which tends to be based on an RTOS like FreeRTOS or Zephyr
(think of them like Unikernels), has a very small window of time from
receiving an interrupt to accomplish a critical task. Nothing should be
disturbing the execution of the RTOS during the critical window. The
operation the RTOS needs to perform is typically on a passthrough device
without Xen interactions.

In general from the hypervisor point of view, the idea is that Xen
should inject the interrupt and then leave the RTOS alone and
undisturbed to do its job.


 
> >  Interrupt forwarding
> > from Xen into HVM/PVH guests uses a softirq to do the injection, which
> > means there's a non-deterministic window of latency between when the
> > interrupt is received by Xen, as to when it's injected to the guest,
> > because the softirq might not get processed right after being set as
> > pending (there might be other softirqs to process, or simply Xen might
> > be busy doing some other operation).
> > 
> > I think you want to look into adding a new command line option or
> > similar, that allows selecting whether guest IRQs are deferred to a
> > softirq for injection, or are injected as part of the processing done
> > in the IRQ handler itself.
> > 
> > Otherwise there will always be a non-deterministic amount of latency
> > on x86 w.r.t. HVM/PVH passthrough guest interrupts.  Haven't you seen
> > some weird/unexpected variance when doing this passthrough interrupt
> > latency measurements on x86?

While this is not great and I agree with Roger that it should be
improved (I'll try to do so), in a well configured system I expect that
there should be no other softirqs on the RTOS vCPU/pCPU so it shouldn't
matter much if it is raise as a softirq or not?