[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [PATCH v2] xen/arm: implement GICD_I[S/C]ACTIVER reads
> On Apr 7, 2020, at 5:50 PM, Julien Grall <julien@xxxxxxx> wrote: > > > > On 07/04/2020 17:16, George Dunlap wrote: >>> On Apr 6, 2020, at 7:47 PM, Julien Grall <julien@xxxxxxx> wrote: >>> >>> On 06/04/2020 18:58, George Dunlap wrote: >>>>> On Apr 3, 2020, at 9:27 PM, Julien Grall <julien.grall.oss@xxxxxxxxx> >>>>> wrote: >>>>> >>>>> On Fri, 3 Apr 2020 at 20:41, Stefano Stabellini <sstabellini@xxxxxxxxxx> >>>>> wrote: >>>>>> >>>>>> On Thu, 2 Apr 2020, Julien Grall wrote: >>>>>>> As we discussed on Tuesday, the cost for other vCPUs is only going to >>>>>>> be a >>>>>>> trap to the hypervisor and then back again. The cost is likely smaller >>>>>>> than >>>>>>> receiving and forwarding an interrupt. >>>>>>> >>>>>>> You actually agreed on this analysis. So can you enlighten me as to why >>>>>>> receiving an interrupt is a not problem for latency but this is? >>>>>> >>>>>> My answer was that the difference is that an operating system can >>>>>> disable interrupts, but it cannot disable receiving this special IPI. >>>>> >>>>> An OS can *only* disable its own interrupts, yet interrupts will still >>>>> be received by Xen even if the interrupts are masked at the processor >>>>> (e.g local_irq_disable()). >>>>> >>>>> You would need to disable interrupts one by one as the GIC level (use >>>>> ICENABLER) in order to not receive any interrupts. Yet, Xen may still >>>>> receive interrupts for operational purposes (e.g serial, console, >>>>> maintainance IRQ...). So trap will happen. >>>> I think Stefano’s assertion is that the users he has in mind will be >>>> configuring the system such that RT workloads get a minimum number of >>>> hypervisor-related interrupts possible. On a 4-core system, you could >>>> have non-RT workloads running on cores 0-1, and RT workloads running with >>>> the NULL scheduler on cores 2-3. In such a system, you’d obviously >>>> arrange that serial and maintenance IRQs are delivered to cores 0-1. >>> Well maintenance IRQs are per pCPU so you can't route to another one... >>> >>> But, I think you missed my point that local_irq_disable() from the guest >>> will not prevent the hypervisor to receive interrupts *even* the one routed >>> to the vCPU itself. They will just not be delivered to the guest context >>> until local_irq_enable() is called. >> My understanding, from Stefano was that what his customers are concerned >> about is the time between the time a physical IRQ is delivered to the guest >> and the time the guest OS can respond appropriately. The key thing here >> isn’t necessarily speed, but predictability — system designers need to know >> that, with a high probability, their interrupt routines will complete within >> X amount of cycles. >> Further interrupts delivered to a guest are not a problem in this scenario, >> if the guest can disable them until the critical IRQ has been handled. > > You keep saying a guest can disable interrupts, but it can't do it via > local_irq_disable(). So what method are you thinking? Disabling at the GIC > level? That is involving traps and most likely not going to help with > predictability... So you’ll have to forgive me for making educated guesses here, as I’m trying to collect all the information. On x86, if you use device pass-through on a system with a virtualized APIC and posted interrupts, then when when the device generates interrupts, those are delivered directly to the guest without involvement of Xen; and when the guest disables interrupts in the vAPIC, those interrupts will be disabled, and be delivered when the guest re-enables interrupts. Given what Stefano said about disabling interrupts, I assumed that ARM had the same sort of functionality. Is that not the case? >> Xen-related IPIs, however, could potentially cause a problem if not >> mitigated. Consider a guest where vcpu 1 loops over the register, while vcpu >> 2 is handling a latency-critical IRQ. A naive implementation might send an >> IPI every time vcpu 1 does a read, spamming vcpu 2 with dozens of IPIs. >> Then an IRQ routine which reliably finishes well within the required time >> normally suddenly overruns and causes an issue. > > I never suggested the naive implementation would be perfect. That's why I > said it can be optimized... It didn’t seem to me that you understood what Stefano’s concerns were; so I was trying to explain the situation he is trying to avoid (as well as making sure that I had a clear understanding myself). The reason I said “a naive implementation” was to make clear that I knew that’s not what you were suggesting. :-) >> I don’t know what maintenance IRQs are, but if they only happen >> intermittently, it’s possible that you’d never get more than a single one in >> a latency-critical IRQ routine; and as such, the variatibility in execution >> time (jitter) wouldn’t be an issue in practice. But every time you add a >> new unblockable IPI, you increase this jitter; particularly if this >> unblockable IPI might be repeated an arbitrary number of times. >> (Stefano, let me know if I’ve misunderstood something.) >> So stepping back a moment, here’s all the possible ideas that I think have >> been discussed (or are there implicitly) so far. >> 1. [Default] Do nothing; guests using this register continue crashing >> 2. Make the I?ACTIVER registers RZWI. >> 3. Make I?ACTIVER return the most recent known value; i.e. KVM’s current >> behavior (as far as we understand it) >> 4. Use a simple IPI with do_noop to update I?ACTIVER >> 4a. Use an IPI, but come up with clever tricks to avoid interrupting guests >> handling IRQs. >> 5. Trap to Xen on guest EOI, so that we know when the >> 6. Some clever paravirtualized option > > 7. Use an IPI if we are confident the interrupts may be active. I don’t understand this one. How is it different than 4 or 4a? And in particular, how does it evaluate on the “how much additional design work would it take” criteria? -George
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |