Re: [Xen-devel] [PATCH] xen/arm: introduce vwfi parameter

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 21/02/17 17:49, Stefano Stabellini wrote:
> On Tue, 21 Feb 2017, Dario Faggioli wrote:
>> On Tue, 2017-02-21 at 13:46 +0000, George Dunlap wrote:
>>>
>>> A.  Don't trap guest WFI at all -- allow it to 'halt' in
>>> moderate-power-but-ready-for-interrupt mode.
>>>
>>> [..]
>>>
>>> A is safe because the scheduler should already have set a timer to
>>> break
>>> out of it if necessary.  The only potential issue here is that the
>>> guest
>>> is burning its credits, meaning that other vcpus with something
>>> potentially useful to do aren't being allowed to run; and then later
>>> when this vcpu has something useful to do it may be prevented from
>>> running because of low credits.  (This may be what Dario means when
>>> he
>>> says it "breaks scheduling").
>>>
>> Are you also referring to the case when there are less vCPUs around
>> than the host has pCPUs (and, ideally, all vCPUs are pinned 1:1 to a
>> pCPU)? If yes, I agree that we're probably fine, but we have to check
>> and enforce all this to be the case.
>>
>> If no, think at a situation where there is 1 vCPU running on a pCPU and
>> 3 vCPUs in the runqueue (it may be a per-CPU Credit1 runqueue or a
>> shared among some pCPUs Credit2 runqueue). If the running vCPU goes
>> idle, let's say with WFI, we _don't_ want the pCPU to enter neither
>> moderate nor deep sleep, we want to pick up the first of the 3 other
>> vCPUs that are waiting in the runqueue.
>>
>> This is what I mean when I say "breaks scheduling". :-)
> 
> That's right, I cannot see how this can be made to work correctly.
> 
> 
>> Oh, actually, if --which I only now realize may be what you are
>> referring to, since you're talking about "guest burning its credits"--
>> you let the vCPU put the pCPU to sleep *but*, when it wakes up (or when
>> the scheduler runs again for whatever reason), you charge to it for all
>> the time the the pCPU was actually idle/sleeping, well, that may
>> actually  not break scheduling, or cause disruption to the service of
>> other vCPUs.... But indeed I'd consider it rather counter intuitive a
>> behavior.
> 
> How can this be safe? There could be no interrupts programmed to wake up
> the pcpu at all. In fact, I don't think today there would be any, unless
> we set one up in Xen for the specific purpose of interrupting the pcpu
> sleep.
> 
> I don't know the inner working of the scheduler, but does it always send
> an interrupt to other pcpu to schedule something?

Letting a guest call WFI is as safe as letting a guest `while(1);`.  Xen
*always* has to set a timer interrupt before switching to the guest
context to make sure that it can pre-empt the vcpu -- otherwise any vcpu
could perform a DoS by simply continually executing instructions.

As long as the guest can't disable host interrupts, and as long as the
scheduler interrupt will break out of a guest WFI into Xen, it should be
safe.

> What if there are 2 vcpu pinned to the same pcpu? This cannot be fair.

From the scheduler's perspective, the WFI would be the same as the
`while(1)`.  If you had two vcpus doing while(1) on the same vcpu, the
credit2 scheduler would (approximately) let one run for 2ms, then the
other for 2ms, and so on, each getting 50%.  If instead you have one
doing a WFI, then the credit2 scheduler would do the same -- let the
first one do WFI for 2ms, then preempt it and let the other one spin for
2ms, then let the first one WFI for 2ms, &c.

(Obviously the exact behaviour would be significantly more random.)

>> What would be the benefit of this? That we don't context switch (either
>> to idle or to someone else)?
> 
> Yes, that would be the benefit.

If ARM had the equivalent of posted interrupts, then a pinned guest
could efficiently wait for interrupts with no additional latency from
virtualization without having to poll.

(Speaking of which -- that could be an interesting optimization even on
x86... if a pcpu has no vcpus waiting to run, then disable HLT exit.)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel