Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan <van.freenix@xxxxxxxxx>]

On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>On Tue, 20 Sep 2016, Julien Grall wrote:
>> Hi Stefano,
>> 
>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>> > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > Hi,
>> > > 
>> > > On 20/09/2016 12:27, George Dunlap wrote:
>> > > > On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.freenix@xxxxxxxxx>
>> > > > wrote:
>> > > > > On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>> > > > > > On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>> > > > > > > On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > > > > I'd like to add a computing capability in xen/arm, like this:
>> > > > > 
>> > > > > struct compute_capatiliby
>> > > > > {
>> > > > >    char *core_name;
>> > > > >    uint32_t rank;
>> > > > >    uint32_t cpu_partnum;
>> > > > > };
>> > > > > 
>> > > > > struct compute_capatiliby cc=
>> > > > > {
>> > > > >   {"A72", 4, 0xd08},
>> > > > >   {"A57", 3, 0xxxx},
>> > > > >   {"A53", 2, 0xd03},
>> > > > >   {"A35", 1, ...},
>> > > > > }
>> > > > > 
>> > > > > Then when identify cpu, we decide which cpu is big and which cpu is
>> > > > > little
>> > > > > according to the computing rank.
>> > > > > 
>> > > > > Any comments?
>> > > > 
>> > > > I think we definitely need to have Xen have some kind of idea the
>> > > > order between processors, so that the user doesn't need to figure out
>> > > > which class / pool is big and which pool is LITTLE.  Whether this sort
>> > > > of enumeration is the best way to do that I'll let Julien and Stefano
>> > > > give their opinion.
>> > > 
>> > > I don't think an hardcoded list of processor in Xen is the right 
>> > > solution.
>> > > There are many existing processors and combinations for big.LITTLE so it
>> > > will
>> > > nearly be impossible to keep updated.
>> > > 
>> > > I would expect the firmware table (device tree, ACPI) to provide relevant
>> > > data
>> > > for each processor and differentiate big from LITTLE core.
>> > > Note that I haven't looked at it for now. A good place to start is 
>> > > looking
>> > > at
>> > > how Linux does.
>> > 
>> > That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>> > trivial to identify the two different CPU classes and which cores belong
>> > to which class.t, as
>> 
>> The class of the CPU can be found from the MIDR, there is no need to use the
>> device tree/acpi for that. Note that I don't think there is an easy way in
>> ACPI (i.e not in AML) to find out the class.
>> 
>> > It is harder to figure out which one is supposed to be
>> > big and which one LITTLE. Regardless, we could default to using the
>> > first cluster (usually big), which is also the cluster of the boot cpu,
>> > and utilize the second cluster only when the user demands it.
>> 
>> Why do you think the boot CPU will usually be a big one? In the case of Juno
>> platform it is configurable, and the boot CPU is a little core on r2 by
>> default.
>> 
>> In any case, what we care about is differentiate between two set of CPUs. I
>> don't think Xen should care about migrating a guest vCPU between big and
>> LITTLE cpus. So I am not sure why we would want to know that.
>
>No, it is not about migrating (at least yet). It is about giving useful
>information to the user. It would be nice if the user had to choose
>between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>even "A7" or "A15".

As Dario mentioned in previous email,
for dom0 provide like this:

dom0_vcpus_big = 4
dom0_vcpus_little = 2

to dom0.

If these two no provided, we could let dom0 runs on big pcpus or big.little.
Anyway this is not the important point for dom0 only big or big.little.

For domU, provide "vcpus.big" and "vcpus.little" in xl configuration file.
Such as:

vcpus.big = 2
vcpus.litle = 4


According to George's comments,
Then, I think we could use affinity to restrict little vcpus be scheduled on 
little vcpus,
and restrict big vcpus on big vcpus. Seems no need to consider soft affinity, 
use hard
affinity is to handle this.

We may need to provide some interface to let xl can get the information such as
big.little or smp. if it is big.little, which is big and which is little.

For how to differentiate cpus, I am looking the linaro eas cpu topology code,
The code has not been upstreamed (:, but merged into google android kernel.
I only plan to take some necessary code, such as device tree parse and
cpu topology build, because we only need to know the computing capacity of each 
pcpu.

Some doc about eas piece, including dts node examples:
https://git.linaro.org/arm/eas/kernel.git/blob/refs/heads/lsk-v4.4-eas-v5.2:/Documentation/devicetree/bindings/scheduler/sched-energy-costs.txt

I pasted partial eas code:
                        for (i = 0, val = prop->value; i < nstates; i++) {
                                cap_states[i].cap = be32_to_cpup(val++);
                                cap_states[i].power = be32_to_cpup(val++);
                        }

                        sge->nr_cap_states = nstates;
                        sge->cap_states = cap_states;

The upper code is to get the computing capacity and power from device tree.

For us, I think we only need to cap entry.
Add a "cap" entry in cpuinfo_arm, fill cap when parsing dt.
Add a "cap" entry in arch_vcpu or vcpu.

When creating a vcpu, fill cap according to this is big vcpu or little vcpu.
The cap should be same with the cap of physical cpu.
Then set the hard affinity of vcpu when creat a vcpu.

User may change the hard affinity of a vcpu, so we also need to block a little
vcpu be scheduled to a big physical cpu. Add some checking code in xen,
when chaning the hard affnity, check whether the cap of a vcpu is compatible
with the cap of the physical cpus.

I am not sure, but we may also need to handle mpidr for ARM, because big and 
little vcpus are supported.

All the above is that I would like to implement according the to discussion of
this thread. No cpupool and scheduler part included.

Please comments. 

Thanks,
Peng.
-- 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel