Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 19 Sep 2016, Juergen Gross wrote:
> On 19/09/16 12:06, Julien Grall wrote:
> > Hi George,
> > 
> > On 19/09/2016 11:45, George Dunlap wrote:
> >> On Mon, Sep 19, 2016 at 9:53 AM, Julien Grall <julien.grall@xxxxxxx>
> >> wrote:
> >>>>> As mentioned in the mail you pointed above, this series is not
> >>>>> enough to
> >>>>> make
> >>>>> big.LITTLE working on then. Xen is always using the boot CPU to detect
> >>>>> the
> >>>>> list of features. With big.LITTLE features may not be the same.
> >>>>>
> >>>>> And I would prefer to see Xen supporting big.LITTLE correctly before
> >>>>> beginning to think to expose big.LITTLE to the userspace (via cpupool)
> >>>>> automatically.
> >>>>
> >>>>
> >>>> Do you mean vcpus be scheduled between big and little cpus freely?
> >>>
> >>>
> >>> By supporting big.LITTLE correctly I meant Xen thinks that all the
> >>> cores has
> >>> the same set of features. So the feature detection is only done the boot
> >>> CPU. See processor_setup for instance...
> >>>
> >>> Moving vCPUs between big and little cores would be a hard task (cache
> >>> line
> >>> issue, and possibly feature) and I don't expect to ever cross this in
> >>> Xen.
> >>> However, I am expecting to see big.LITTLE exposed to the guest (i.e
> >>> having
> >>> big and little vCPUs).
> >>
> >> So it sounds like the big and LITTLE cores are architecturally
> >> different enough that software must be aware of which one it's running
> >> on?
> > 
> > That's correct. Each big and LITTLE cores may have different errata,
> > different features...
> > 
> > It has also the advantage to let the guest dealing itself with its own
> > power efficiency without introducing a specific Xen interface.
> > 
> >>
> >> Exposing varying numbers of big and LITTLE vcpus to guests seems like
> >> a sensible approach.  But at the moment cpupools only allow a domain
> >> to be in exactly one pool -- meaning if we use cpupools to control the
> >> big.LITTLE placement, you won't be *able* to have guests with both big
> >> and LITTLE vcpus.
> >>
> >>>> If need to create all the pools, need to decided how many pools need
> >>>> to be
> >>>> created.
> >>>> I thought about this, but I do not come out a good idea.
> >>>>
> >>>> The cpupool0 is defined in xen/common/cpupool.c, if need to create many
> >>>> pools,
> >>>> need to alloc cpupools dynamically when booting. I would not like to
> >>>> change a
> >>>> lot to common code.
> >>>
> >>>
> >>> Why? We should avoid to choose a specific design just because the common
> >>> code does not allow you to do it without heavy change.
> >>>
> >>> We never came across the big.LITTLE problem on x86, so it is normal to
> >>> modify the code.
> >>
> >> Julien is correct; there's no reason we couldn't have a default
> >> multiple pools on boot.
> >>
> >>>> The implementation in this patchset I think is an easy way to let
> >>>> Big and
> >>>> Little
> >>>> CPUs all run.
> >>>
> >>>
> >>> I care about having a design allowing an easy use of big.LITTLE on
> >>> Xen. Your
> >>> solution requires the administrator to know the underlying platform and
> >>> create the pool.
> >>>
> >>> In the solution I suggested, the pools would be created by Xen (and
> >>> the info
> >>> exposed to the userspace for the admin).
> >>
> >> FWIW another approach could be the one taken by "xl
> >> cpupool-numa-split": you could have "xl cpupool-bigLITTLE-split" or
> >> something that would automatically set up the pools.
> >>
> >> But expanding the schedulers to know about different classes of cpus,
> >> and having vcpus specified as running only on specific types of pcpus,
> >> seems like a more flexible approach.
> > 
> > So, if I understand correctly, you would not recommend to extend the
> > number of CPU pool per domain, correct?
> 
> Before deciding in which direction to go (multiple cpupools, sub-pools,
> kind of implicit cpu pinning) I think we should think about the
> implications regarding today's interfaces:
> 
> - Do we want to be able to use different schedulers for big/little
>   (this would mean some cpupool related solution)? I'd prefer to
>   have only one scheduler type for each domain. :-)
> 
> - What about scheduling parameters like weight and cap? How would
>   those apply (answer probably influencing pinning solution).
>   Remember that especially the downsides of pinning led to the
>   introduction of cpupools.

It isn't easy to answer these questions, but there might be a reason to
have different schedulers because they are supposed to run different
classes of workloads. big cores are of cpu intensive tasks (think of V8,
the Javascript engine), while LITTLE cores are for low key background
applications (think of the whatsapp daemon that runs in the background
on your phone).


> - Is big.LITTLE to be expected to be combined with NUMA?

NUMA is not a popular way to design hardware in the ARM ecosystem today.
If it did become more widespread, I would expect it to happen on server
hardware, where big.LITTLE is not used.

However if there will ever be NUMA systems combined with big.LITTLE, I
would expect each NUMA node to be big.LITTLE.


> - Do we need to support live migration for domains containing both
>   types of cpus?

I think it will happen. Once we have live-migration on ARM and support
for bit.LITTLE guests, then it is only natural to support live-migration
for them too.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel