Re: [PATCH v2] xen/arm: implement GICD_I[S/C]ACTIVER reads

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Apr 7, 2020, at 5:50 PM, Julien Grall <julien@xxxxxxx> wrote:
> 
> 
> 
> On 07/04/2020 17:16, George Dunlap wrote:
>>> On Apr 6, 2020, at 7:47 PM, Julien Grall <julien@xxxxxxx> wrote:
>>> 
>>> On 06/04/2020 18:58, George Dunlap wrote:
>>>>> On Apr 3, 2020, at 9:27 PM, Julien Grall <julien.grall.oss@xxxxxxxxx> 
>>>>> wrote:
>>>>> 
>>>>> On Fri, 3 Apr 2020 at 20:41, Stefano Stabellini <sstabellini@xxxxxxxxxx> 
>>>>> wrote:
>>>>>> 
>>>>>> On Thu, 2 Apr 2020, Julien Grall wrote:
>>>>>>> As we discussed on Tuesday, the cost for other vCPUs is only going to 
>>>>>>> be a
>>>>>>> trap to the hypervisor and then back again. The cost is likely smaller 
>>>>>>> than
>>>>>>> receiving and forwarding an interrupt.
>>>>>>> 
>>>>>>> You actually agreed on this analysis. So can you enlighten me as to why
>>>>>>> receiving an interrupt is a not problem for latency but this is?
>>>>>> 
>>>>>> My answer was that the difference is that an operating system can
>>>>>> disable interrupts, but it cannot disable receiving this special IPI.
>>>>> 
>>>>> An OS can *only* disable its own interrupts, yet interrupts will still
>>>>> be received by Xen even if the interrupts are masked at the processor
>>>>> (e.g local_irq_disable()).
>>>>> 
>>>>> You would need to disable interrupts one by one as the GIC level (use
>>>>> ICENABLER) in order to not receive any interrupts. Yet, Xen may still
>>>>> receive interrupts for operational purposes (e.g serial, console,
>>>>> maintainance IRQ...). So trap will happen.
>>>> I think Stefano’s assertion is that the users he has in mind will be 
>>>> configuring the system such that RT workloads get a minimum number of 
>>>> hypervisor-related interrupts possible.  On a 4-core system, you could  
>>>> have non-RT workloads running on cores 0-1, and RT workloads running with 
>>>> the NULL scheduler on cores 2-3.  In such a system, you’d obviously 
>>>> arrange that serial and maintenance IRQs are delivered to cores 0-1.
>>> Well maintenance IRQs are per pCPU so you can't route to another one...
>>> 
>>> But, I think you missed my point that local_irq_disable() from the guest 
>>> will not prevent the hypervisor to receive interrupts *even* the one routed 
>>> to the vCPU itself. They will just not be delivered to the guest context 
>>> until local_irq_enable() is called.
>> My understanding, from Stefano was that what his customers are concerned 
>> about is the time between the time a physical IRQ is delivered to the guest 
>> and the time the guest OS can respond appropriately.  The key thing here 
>> isn’t necessarily speed, but predictability — system designers need to know 
>> that, with a high probability, their interrupt routines will complete within 
>> X amount of cycles.
>> Further interrupts delivered to a guest are not a problem in this scenario, 
>> if the guest can disable them until the critical IRQ has been handled.
> 
> You keep saying a guest can disable interrupts, but it can't do it via 
> local_irq_disable(). So what method are you thinking? Disabling at the GIC 
> level? That is involving traps and most likely not going to help with 
> predictability...

So you’ll have to forgive me for making educated guesses here, as I’m trying to 
collect all the information.  On x86, if you use device pass-through on a 
system with a virtualized APIC and posted interrupts, then when when the device 
generates interrupts, those are delivered directly to the guest without 
involvement of Xen; and when the guest disables interrupts in the vAPIC, those 
interrupts will be disabled, and be delivered when the guest re-enables 
interrupts.  Given what Stefano said about disabling interrupts, I assumed that 
ARM had the same sort of functionality.  Is that not the case?

>> Xen-related IPIs, however, could potentially cause a problem if not 
>> mitigated. Consider a guest where vcpu 1 loops over the register, while vcpu 
>> 2 is handling a latency-critical IRQ.  A naive implementation might send an 
>> IPI every time vcpu 1 does a read, spamming vcpu 2 with dozens of IPIs.  
>> Then an IRQ routine which reliably finishes well within the required time 
>> normally suddenly overruns and causes an issue.
> 
> I never suggested the naive implementation would be perfect. That's why I 
> said it can be optimized...

It didn’t seem to me that you understood what Stefano’s concerns were; so I was 
trying to explain the situation he is trying to avoid (as well as making sure 
that I had a clear understanding myself).  The reason I said “a naive 
implementation” was to make clear that I knew that’s not what you were 
suggesting. :-)

>> I don’t know what maintenance IRQs are, but if they only happen 
>> intermittently, it’s possible that you’d never get more than a single one in 
>> a latency-critical IRQ routine; and as such, the variatibility in execution 
>> time (jitter) wouldn’t be an issue in practice.  But every time you add a 
>> new unblockable IPI, you increase this jitter; particularly if this 
>> unblockable IPI might be repeated an arbitrary number of times.
>> (Stefano, let me know if I’ve misunderstood something.)
>> So stepping back a moment, here’s all the possible ideas that I think have 
>> been discussed (or are there implicitly) so far.
>> 1. [Default] Do nothing; guests using this register continue crashing
>> 2. Make the I?ACTIVER registers RZWI.
>> 3. Make I?ACTIVER return the most recent known value; i.e. KVM’s current 
>> behavior (as far as we understand it)
>> 4. Use a simple IPI with do_noop to update I?ACTIVER
>> 4a.  Use an IPI, but come up with clever tricks to avoid interrupting guests 
>> handling IRQs.
>> 5. Trap to Xen on guest EOI, so that we know when the
>> 6. Some clever paravirtualized option
> 
> 7. Use an IPI if we are confident the interrupts may be active.

I don’t understand this one.  How is it different than 4 or 4a?  And in 
particular, how does it evaluate on the “how much additional design work would 
it take” criteria?

 -George