[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [RFC PATCH 0/2] Boot time cpupools
- To: Julien Grall <julien@xxxxxxx>
- From: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>
- Date: Thu, 18 Nov 2021 17:29:40 +0200
- Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Luca Fancellu <Luca.Fancellu@xxxxxxx>, Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Wei Chen <Wei.Chen@xxxxxxx>, Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, Juergen Gross <jgross@xxxxxxxx>, Dario Faggioli <dfaggioli@xxxxxxxx>
- Delivery-date: Thu, 18 Nov 2021 15:30:05 +0000
- List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
Hi Julien, all
[Sorry for the possible format issues]
(Prunning some CC to just leave Arm and sched folks)
On 17/11/2021 12:07, Bertrand Marquis wrote:
> Hi Julien,
Hi Bertrand,
>> On 17 Nov 2021, at 11:48, Julien Grall <julien@xxxxxxx> wrote:
>>
>> On 17/11/2021 11:16, Bertrand Marquis wrote:
>>> Hi Julien,
>>
>> Hi,
>>
>>>> On 17 Nov 2021, at 10:26, Julien Grall <julien@xxxxxxx> wrote:
>>>>
>>>> Hi Luca,
>>>>
>>>> On 17/11/2021 09:57, Luca Fancellu wrote:
>>>>> Currently Xen creates a default cpupool0 that contains all the cpu brought up
>>>>> during boot and it assumes that the platform has only one kind of CPU.
>>>>> This assumption does not hold on big.LITTLE platform, but putting different
>>>>> type of CPU in the same cpupool can result in instability and security issues
>>>>> for the domains running on the pool.
>>>>
>>>> I agree that you can't move a LITTLE vCPU to a big pCPU. However...
>>>>
>>>>> For this reason this serie introduces an architecture specific way to create
>>>>> different cpupool at boot time, this is particularly useful on ARM big.LITTLE
>>>>> platform where there might be the need to have different cpupools for each type
>>>>> of core, but also systems using NUMA can have different cpu pool for each node.
>>>>
>>>> ... from my understanding, all the vCPUs of a domain have to be in the same cpupool. So with this approach it is not possible:
>>>> 1) to have a mix of LITTLE and big vCPUs in the domain
>>>> 2) to create a domain spanning across two NUMA nodes
>>>>
>>>> So I think we need to make sure that any solutions we go through will not prevent us to implement those setups.
>>> The point of this patch is to make all cores available without breaking the current behaviour of existing system.
>>
>> I might be missing some context here. By breaking current behavior, do you mean user that may want to add "hmp-unsafe" on the command line?
>
> Right, with hmp-unsafe the behaviour is now the same as without, only extra cores are parked in other cpupools.
>
> So you have a point in fact that behaviour is changed for someone who was using hmp-unsafe before if this is activated.
> The command line argument suggested by Jurgen definitely makes sense here.
>
> We could instead do the following:
> - when this is activated in the configuration, boot all cores and park them in different pools (depending on command line argument). Current behaviour not change if other pools are not used (but more cores will be on in xen)
From my understanding, it is possible to move a pCPU in/out a pool
afterwards. So the security concern with big.LITTLE is still present,
even though it would be difficult to hit it.
I am also concerned that it would be more difficult to detect any
misconfiguration. So I think this option would still need to be turned
on only if hmp-unsafe are the new command line one are both on.
If we want to enable it without hmp-unsafe on, we would need to at least
lock the pools.
> - when hmp-unsafe is on, this feature will be turned of (if activated in configuration) and all cores would be added in the same pool.
>
> What do you think ?
I am split. On one hand, this is making easier for someone to try
big.LITTLE as you don't have manually pin vCPUs. On the other hand, this
is handling a single use-case and you would need to use hmp-unsafe and
pinning if you want to get more exotic setup (e.g. a domain with
big.LITTLE).
Another possible solution is to pin dom0 vCPUs (AFAIK they are just
sticky by default) and then create the pools from dom0 userspace. My
assumption is for dom0less we would want to use pinning instead.
That said I would like to hear from Xilinx and EPAM as, IIRC, they are
already using hmp-unsafe in production.
We have been using hmp-unsafe since it's introduction, yes we are aware of possible consequences of enabling that option (as different CPU types might have different errata, cache lines, PA bits (?), etc), so we are trying to deal with it carefully.
In our target system we pin Dom0's vCPUs to the pCPUs of the same type from userspace via "xl vcpu-pin" command, for other domains more exotic configuration can be used.
I share Stefano's opinion not to tie new feature (boot time MIDR-based cpupools) to existing hmp-unsafe option and create new command line option to control new feature.
The proposed algorithm (copy it here for the convenience) looks fine to me.
"So in short I think it should be:
- midr-cpupools alone
cpupools created at boot, warning/errors on changing cpupools
- midr-cpupools + hmp-unsafe
cpupools created at boot, changing cpupools is allowed (we could still
warn but no errors)
- hmp-unsafe alone
same as today with hmp-unsafe"
For the default behaviour I also don't have a strong preference. One thing is clear: default behaviour should be safe. I think, the command line option is preferred over the config option as new feature can be enabled/disabled without a need to re-build Xen, moreover, if we follow the proposed algorithm route, the behaviour of new feature at runtime (whether the changing of cpupools is allowed or not) are going to be depended on the hmp-unsafe state which is under command line control currently. But, there is no strong preference here as well.
Cheers,
--
Julien Grall
--
|
