Re: [Xen-devel] [RFC PATCH 21/24] ARM: vITS: handle INVALL command

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 6 Dec 2016, Julien Grall wrote:
> On 06/12/2016 22:01, Stefano Stabellini wrote:
> > On Tue, 6 Dec 2016, Stefano Stabellini wrote:
> > > moving a vCPU with interrupts assigned to it is slower than moving a
> > > vCPU without interrupts assigned to it. You could say that the
> > > slowness is directly proportional do the number of interrupts assigned
> > > to the vCPU.
> > 
> > To be pedantic, by "assigned" I mean that a physical interrupt is routed
> > to a given pCPU and is set to be forwarded to a guest vCPU running on it
> > by the _IRQ_GUEST flag. The guest could be dom0. Upon receiving one of
> > these physical interrupts, a corresponding virtual interrupt (could be a
> > different irq) will be injected into the guest vCPU.
> > 
> > When the vCPU is migrated to a new pCPU, the physical interrupts that
> > are configured to be injected as virtual interrupts into the vCPU, are
> > migrated with it. The physical interrupt migration has a cost. However,
> > receiving physical interrupts on the wrong pCPU has an higher cost.
> 
> I don't understand why it is a problem for you to receive the first interrupt
> to the wrong pCPU and moving it if necessary.
> 
> While this may have an higher cost (I don't believe so) on the first received
> interrupt, migrating thousands of interrupts at the same time is very
> expensive and will likely get Xen stuck for a while (think about ITS with a
> single command queue).
> 
> Furthermore, the current approach will move every single interrupt routed a
> the vCPU, even those disabled. That's pointless and a waste of resource. You
> may argue that we can skip the ones disabled, but in that case what would be
> the benefits to migrate the IRQs while migrate the vCPUs?
>
> So I would suggest to spread it over the time. This also means less headache
> for the scheduler developers.

The most important aspect of interrupts handling in Xen is latency,
measured as the time between Xen receiving a physical interrupt and the
guest receiving it. This latency should be both small and deterministic.

We all agree so far, right?


The issue with spreading interrupts migrations over time is that it makes
interrupt latency less deterministic. It is OK, in the uncommon case of
vCPU migration with interrupts, to take a hit for a short time. This
"hit" can be measured. It can be known. If your workload cannot tolerate
it, vCPUs can be pinned. It should be a rare event anyway. On the other
hand, by spreading interrupts migrations, we make it harder to predict
latency. Aside from determinism, another problem with this approach is
that it ensures that every interrupt assigned to a vCPU will first hit
the wrong pCPU, then it will be moved. It guarantees the worst-case
scenario for interrupt latency for the vCPU that has been moved. If we
migrated all interrupts as soon as possible, we would minimize the
amount of interrupts delivered to the wrong pCPU. Most interrupts would
be delivered to the new pCPU right away, reducing interrupt latency.

Regardless of how we implement interrupts migrations on ARM, I think it
still makes sense for the scheduler to know about it. I realize that
this is a separate point. Even if we spread interrupts migrations over
time, it still has a cost, in terms of latency as I wrote above, but also
in terms of interactions with interrupt controllers and ITSes. A vCPU
with no interrupts assigned to it poses no such problems. The scheduler
should be aware of the difference. If the scheduler knew, I bet that
vCPU migration would be a rare event for vCPUs that have many interrupts
assigned to them. For example, Dom0 vCPU0 would never be moved, and
dom0_pin_vcpus would be superfluous.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel