[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC
- To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
- From: Julien Grall <julien.grall@xxxxxxx>
- Date: Wed, 21 Sep 2016 20:21:40 +0100
- Cc: Juergen Gross <jgross@xxxxxxxx>, Peng Fan <peng.fan@xxxxxxx>, Steve Capper <Steve.Capper@xxxxxxx>, George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Dario Faggioli <dario.faggioli@xxxxxxxxxx>, Punit Agrawal <punit.agrawal@xxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Peng Fan <van.freenix@xxxxxxxxx>
- Delivery-date: Wed, 21 Sep 2016 19:21:48 +0000
- List-id: Xen developer discussion <xen-devel.lists.xen.org>
On 21/09/2016 20:11, Julien Grall wrote:
Hi Stefano,
On 21/09/2016 19:13, Stefano Stabellini wrote:
On Wed, 21 Sep 2016, Julien Grall wrote:
(CC a couple of ARM folks)
On 21/09/16 11:22, George Dunlap wrote:
On 21/09/16 11:09, Julien Grall wrote:
On 20/09/16 21:17, Stefano Stabellini wrote:
On Tue, 20 Sep 2016, Julien Grall wrote:
Hi Stefano,
On 20/09/2016 20:09, Stefano Stabellini wrote:
On Tue, 20 Sep 2016, Julien Grall wrote:
Hi,
On 20/09/2016 12:27, George Dunlap wrote:
On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
<van.freenix@xxxxxxxxx>
wrote:
On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli
wrote:
On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
On Tue, 20 Sep 2016, Dario Faggioli wrote:
I'd like to add a computing capability in xen/arm, like this:
struct compute_capatiliby
{
char *core_name;
uint32_t rank;
uint32_t cpu_partnum;
};
struct compute_capatiliby cc=
{
{"A72", 4, 0xd08},
{"A57", 3, 0xxxx},
{"A53", 2, 0xd03},
{"A35", 1, ...},
}
Then when identify cpu, we decide which cpu is big and which
cpu is
little
according to the computing rank.
Any comments?
I think we definitely need to have Xen have some kind of idea
the
order between processors, so that the user doesn't need to
figure out
which class / pool is big and which pool is LITTLE. Whether
this
sort
of enumeration is the best way to do that I'll let Julien and
Stefano
give their opinion.
I don't think an hardcoded list of processor in Xen is the right
solution.
There are many existing processors and combinations for big.LITTLE
so it
will
nearly be impossible to keep updated.
I would expect the firmware table (device tree, ACPI) to provide
relevant
data
for each processor and differentiate big from LITTLE core.
Note that I haven't looked at it for now. A good place to start is
looking
at
how Linux does.
That's right, see
Documentation/devicetree/bindings/arm/cpus.txt. It
is
trivial to identify the two different CPU classes and which cores
belong
to which class.t, as
The class of the CPU can be found from the MIDR, there is no need to
use the
device tree/acpi for that. Note that I don't think there is an easy
way in
ACPI (i.e not in AML) to find out the class.
It is harder to figure out which one is supposed to be
big and which one LITTLE. Regardless, we could default to using the
first cluster (usually big), which is also the cluster of the boot
cpu,
and utilize the second cluster only when the user demands it.
Why do you think the boot CPU will usually be a big one? In the case
of Juno
platform it is configurable, and the boot CPU is a little core on r2
by
default.
In any case, what we care about is differentiate between two set of
CPUs. I
don't think Xen should care about migrating a guest vCPU between big
and
LITTLE cpus. So I am not sure why we would want to know that.
No, it is not about migrating (at least yet). It is about giving
useful
information to the user. It would be nice if the user had to choose
between "big" and "LITTLE" rather than "class 0x1" and "class
0x100", or
even "A7" or "A15".
I don't think it is wise to assume that we may have only 2 kind of
CPUs
on the platform. We may have more in the future, if so how would you
name them?
I would suggest that internally Xen recognize an arbitrary number of
processor "classes", and order them according to more powerful -> less
powerful. Then if at some point someone makes a platform with three
processors, you can say "class 0", "class 1" or "class 2". "big" would
be an alias for "class 0" and "little" would be an alias for "class 1".
As mentioned earlier, there is no upstreamed yet device tree bindings
to know
the "power" of a CPU (see [1]
And in my suggestion, we allow a richer set of labels, so that the user
could also be more specific -- e.g., asking for "A15" specifically, for
example, and failing to build if there are no A15 cores present, while
allowing users to simply write "big" or "little" if they want
simplicity
/ things which work across different platforms.
Well, before trying to do something clever like that (i.e naming
"big" and
"little"), we need to have upstreamed bindings available to
acknowledge the
difference. AFAICT, it is not yet upstreamed for Device Tree (see
[1]) and I
don't know any static ACPI tables providing the similar information.
I like George's idea that "big" and "little" could be just convenience
aliases. Of course they are predicated on the necessary device tree
bindings being upstream. We don't need [1] to be upstream in Linux, just
the binding:
http://marc.info/?l=linux-arm-kernel&m=147308556729426&w=2
which has already been acked by the relevant maintainers.
This is device tree only. What about ACPI?
I had few discussions and more thought about big.LITTLE support in
Xen. The
main goal of big.LITTLE is power efficiency by moving task around and
been
able to idle one cluster. All the solutions suggested (including
mine) so far,
can be replicated by hand (except the VPIDR) so they are mostly an
automatic
way. This will also remove the real benefits of big.LITTLE because
Xen will
not be able to migrate vCPU across cluster for power efficiency.
The goal of the architects of big.LITTLE might have been power
efficiency, but of course we are free to use any features that the
hardware provides in the best way for Xen and the Xen community.
This is very dependent on how the big.LITTLE has been implemented by the
hardware. Some platform can not run both big and LITTLE cores at the
same time. You need a proper switch in the firmware/hypervisor.
If we care about power efficiency, we would have to handle seamlessly
big.LITTLE in Xen (i.e a guess would only see a kind of CPU). This
arise quite
few problem, nothing insurmountable, similar to migration across two
platforms
with different micro-architecture (e.g processors): errata, features
supported... The guest would have to know the union of all the errata
(this is
done so far via the MIDR, so we would a PV way to do it), and only the
intersection of features would be exposed to the guest. This also
means the
scheduler would have to be modified to handle power efficiency (not
strictly
necessary at the beginning).
I agree that a such solution would require some work to implement,
although
Xen will have a better control of the energy consumption of the
platform.
So the question here, is what do we want to achieve with big.LITTLE?
I don't think that handling seamlessly big.LITTLE in Xen is the best way
to do it in the scenarios where Xen on ARM is being used today. I
understand the principles behind it, but I don't think that it will lead
to good results in a virtualized environment, where there is more
activity and more vcpus than pcpus.
Can you detail why you don't think it will give good results?
What we discussed in this thread so far is actionable, and gives us
big.LITTLE support in a short time frame. It is a good fit for Xen on
ARM use cases and still leads to lower power consumption with an wise
allocation of big and LITTLE vcpus and pcpus to guests.
How this would lead to lower power consumption? If there is nothing
running of the processor we would have a wfi loop which will never put
the physical CPU in deep sleep. The main advantage of big.LITTLE is too
be able to switch off a cluster/cpu when it is not used.
Without any knowledge in Xen (such as CPU freq), I am afraid the the
power consumption will still be the same.
I would start from this approach, then if somebody comes along with a
plan to implement a big.LITTLE switcher in Xen, I welcome her to do it
and I would be happy to accept the code in Xen. We'll just make it
optional.
I think we are discussing here a simple design for big.LITTLE. I never
asked Peng to do all the work. I am worry that if we start to expose the
big.LITTLE to the userspace it will be hard in the future to step back
from it.
Thinking a bit more to this, after looking at Dario's mail [1], his
suggestion could be seen as a high level use of soft/hard affinity.
I will answer to this.
Regards,
[1] https://lists.xen.org/archives/html/xen-devel/2016-09/msg02293.html
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
