[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH] xen/arm: introduce vwfi parameter



Hi Stefano,

On 21/02/2017 00:38, Stefano Stabellini wrote:
On Mon, 20 Feb 2017, Dario Faggioli wrote:
On Mon, 2017-02-20 at 19:38 +0000, Julien Grall wrote:
On 20/02/17 19:20, Dario Faggioli wrote:
E.g., if vCPU x of domain A wants to go idle with a WFI/WFE, but
the
host is overbooked and currently really busy, Xen wants to run some
other vCPU (of either the same of another domain).

That's actually the whole point of virtualization, and the reason
why
overbooking an host with more vCPUs (from multiple guests) than it
has
pCPUs works at all. If we start letting guests put the host's pCPUs
to
sleep, not only the scheduler, but many things would break, IMO!

I am not speaking about general case but when you get 1 vCPU pinned
to 1
pCPU (I think this is Stefano use case). No other vCPU will run on
this
pCPU. So it would be fine to let the guest do the WFI.

Mmm... ok, yes, in that case, it may make sense and work, from a, let's
say, purely functional perspective. But still I struggle to place this
in a bigger picture.

I feel the same way as you, Dario. That said, if we could make it work
without breaking too many assumptions in Xen, it would be a great
improvement for this use-case.

Again, there is no assumption broken. Using WFI/WFE is just a nice way to say: "You can sleep and save power" so a scheduler can decide to schedule another vCPU. Obviously a guest is free to not use them and could do a busy loop instead.

But as far as I can tell, the use-case you are trying to solve is each vCPU pinned to a specific pCPU. If you don't pin them, your patch will not improve much because both yield and block may context switch you vCPU.


For instance, as you say, executing a WFI from a guest directly on
hardware, only makes sense if we have 1:1 static pinning. Which means
it can't just be done by default, or with a boot parameter, because we
need to check and enforce that there's only 1:1 pinning around.

That's right, but we don't have a way to recognize or enforce 1:1 static
pinning at the moment, do we? But maybe the nop scheduler we discussed
could be a step in the right direction.


Is it possible to decide whether to trap and emulate WFI, or just
execute it, online, and change such decision dynamically? And even if
yes, how would the whole thing work? When the direct execution is
enabled for a domain we automatically enforce 1:1 pinning for that
domain, and kick all the other domain out of its pcpus? What if they
have their own pinning, what if they also have 'direct WFI' behavior
enabled?

Right, I asked myself those questions as well. That is why I wrote "it
breaks the scheduler" in the previous email. I don't think it can work
today, but it could work one day, building on top of the nop scheduler.

WFE/WFI is only a hint for the scheduler to reschedule. If you don't trap them, the guest will still run until the end of its credit. It doesn't break anything.



If it is not possible to change all this online and on a per-domain
basis, what do we do? When dooted with the 'direct WFI' flag, we only
accept 1:1 pinning? Who should enforce that, the setvcpuaffinity
hypercall?

These are just examples, my point being that in theory, if we consider
a very specific usecase or set of usecase, there's a lot we can do. But
when you say "why don't you let the guest directly execute WFI", in
response to a patch and a discussion like this, people may think that
you are actually proposing doing it as a solution, which is not
possible without figuring out all the open questions above (actually,
probably, more) and without introducing a lot of cross-subsystem
policing inside Xen, which is often something we don't want.

+1


But, if you let me say this again, it looks to me we are trying to
solve too many problem all at once in this thread, should we try
slowing down/refocusing? :-)

Indeed. I think this patch can improve some use-cases with little
maintenance cost. It's pretty straightforward to me.

I am still missing the bit of your use-case and I can only speculate on it so far.

But in
that case, using WFI in the guest may not have been the right things
to do.

But if the guest is, let's say, Linux, does it use WFI or not? And is
it the right thing or not?

Again, the fact you're saying this probably means there's something I
am either missing or ignoring about ARM.

Linux uses WFI

Linux may or may not use WFI. It is up to his scheduler to decide.


I have heard use case where people wants to disable the scheduler
(e.g a
nop scheduler) because they know only 1 vCPU will ever run on the
pCPU.
This is exactly the use case I am thinking about.

Sure! Except that, in Xen, we don't know whether we have, and always
will, 1 vCPU ever run on each pCPU. Nor we have a way to enforce that,
neither in toolstack nor in the hypervisor. :-P

Exactly! I should have written it more clearly from the beginning.


--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.