Re: [PATCH v2] xen/arm: implement GICD_I[S/C]ACTIVER reads

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Apr 3, 2020, at 9:27 PM, Julien Grall <julien.grall.oss@xxxxxxxxx> wrote:
> 
> On Fri, 3 Apr 2020 at 20:41, Stefano Stabellini <sstabellini@xxxxxxxxxx> 
> wrote:
>> 
>> On Thu, 2 Apr 2020, Julien Grall wrote:
>>> As we discussed on Tuesday, the cost for other vCPUs is only going to be a
>>> trap to the hypervisor and then back again. The cost is likely smaller than
>>> receiving and forwarding an interrupt.
>>> 
>>> You actually agreed on this analysis. So can you enlighten me as to why
>>> receiving an interrupt is a not problem for latency but this is?
>> 
>> My answer was that the difference is that an operating system can
>> disable interrupts, but it cannot disable receiving this special IPI.
> 
> An OS can *only* disable its own interrupts, yet interrupts will still
> be received by Xen even if the interrupts are masked at the processor
> (e.g local_irq_disable()).
> 
> You would need to disable interrupts one by one as the GIC level (use
> ICENABLER) in order to not receive any interrupts. Yet, Xen may still
> receive interrupts for operational purposes (e.g serial, console,
> maintainance IRQ...). So trap will happen.

I think Stefano’s assertion is that the users he has in mind will be 
configuring the system such that RT workloads get a minimum number of 
hypervisor-related interrupts possible.  On a 4-core system, you could  have 
non-RT workloads running on cores 0-1, and RT workloads running with the NULL 
scheduler on cores 2-3.  In such a system, you’d obviously arrange that serial 
and maintenance IRQs are delivered to cores 0-1.

Julien, I think your argument above about interrupts somewhat undermines your 
point about deadlock.  Your basic argument is, that a guest will be interrupted 
by Xen quite frequently anyway for lots of reasons; adding one more to the list 
shouldn’t measurably increase the jitter.  But if it’s true that a guest will 
be interrupted by Xen frequently, then deadlock shouldn’t be much of an issue; 
and insofar as deadlock is an issue, it’s because there were no interrupts — 
and thus, adding an extra IPI will have a statistically significant effect on 
jitter.

On the other had, Stefano’s argument about deadlock (if I understand him 
correctly) has been that guests shouldn’t really be spinning on that register 
anyway.  But it sounds from Julien’s other email that perhaps spinning on the 
register is exactly what newer versions of Linux do — so Linux would certainly 
hang on such systems if Stefano’s assertion about the low number of Xen-related 
interrupts is true.

If the latter is true, then it sounds like addressing the deadlock issue will 
be necessary?  And so effort needs to be put towards figuring out how to 
minimize jitter due to Xen-related IPIs.

Wei Xu / Peng Fan, any opinions here?

 - George