Re: [PATCH] x86/x2apic: introduce a mixed physical/cluster mode

[Jan Beulich <jbeulich@xxxxxxxx>]

On 30.10.2023 17:10, Roger Pau Monné wrote:
> On Mon, Oct 30, 2023 at 03:32:56PM +0100, Jan Beulich wrote:
>> On 24.10.2023 15:51, Roger Pau Monne wrote:
>>> The current implementation of x2APIC requires to either use Cluster Logical 
>>> or
>>> Physical mode for all interrupts.  However the selection of Physical vs 
>>> Logical
>>> is not done at APIC setup, an APIC can be addressed both in Physical or 
>>> Logical
>>> destination modes concurrently.
>>>
>>> Introduce a new x2APIC mode called mixed, which uses Logical Cluster mode 
>>> for
>>> IPIs, and Physical mode for external interrupts, thus attempting to use the
>>> best method for each interrupt type.
>>>
>>> Using Physical mode for external interrupts allows more vectors to be used, 
>>> and
>>> interrupt balancing to be more accurate.
>>>
>>> Using Logical Cluster mode for IPIs allows less accesses to the ICR register
>>> when sending those, as multiple CPUs can be targeted with a single ICR 
>>> register
>>> write.
>>>
>>> A simple test calling flush_tlb_all() 10000 times in a tight loop on a 96 
>>> CPU
>>> box gives the following average figures:
>>>
>>> Physical mode: 26617931ns
>>> Mixed mode:    23865337ns
>>>
>>> So ~10% improvement versus plain Physical mode.
>>
>> Nice.
>>
>>>  Note that Xen uses Cluster
>>> mode by default, and hence is already using the fastest way for IPI 
>>> delivery at
>>> the cost of reducing the amount of vectors available system-wide.
>>>
>>> Make the newly introduced mode the default one.
>>>
>>> Suggested-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>>> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
>>> ---
>>> Do we want to keep a full Logical Cluster mode available?  I don't see a 
>>> reason
>>> to target external interrupts in Logical Cluster mode, but maybe there's
>>> something I'm missing.
>>>
>>> It's not clear to me whether the ACPI FADT flags are meant to apply only to
>>> external interrupt delivery mode, or also to IPI delivery.  If
>>> ACPI_FADT_APIC_PHYSICAL is only meant to apply to external interrupts and 
>>> not
>>> IPIs then we could possibly get rid of physical mode IPI delivery.
>>>
>>> Still need to put this under XenServer extensive testing, but wanted to get
>>> some feedback before in case approach greatly changes.
>>
>> Looks quite okay, just a couple of minor remarks:
> 
> Thanks.
> 
> Do we still want to keep the pure cluster mode?

I think we should keep it for a full release cycle or two.

Jan