Re: [Xen-devel] [PATCH 0/3] Enable L2 Cache Allocation Technology

[Meng Xu <xumengpanda@xxxxxxxxx>]

On Sat, Sep 17, 2016 at 10:34 PM, Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx> wrote:
> On 16-09-17 11:31:54, Meng Xu wrote:
>> On Thu, Aug 25, 2016 at 1:21 AM, Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx> wrote:
>> >
>> > Design document is below:
>> > =======================================================================
>> > % Intel L2 Cache Allocation Technology (L2 CAT) Feature
>> > % Revision 1.0
>> >
>> > \clearpage
>> >
>> > Hi all,
>> >
>> > We plan to bring a new PSR (Platform Shared Resource) feature called
>> > Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
>> >
>> > This is the initial design of L2 CAT. It might be a little long and
>> > detailed, hope it doesn't matter.
>> >
>> > Besides the L2 CAT implementaion, we refactor the psr.c to make it more
>> > flexible to add new features and fulfill the principle, open for extension
>> > but closed for modification.
>> >
>> > Comments and suggestions are welcome :-)
>>
>>
>> I have some comments/questions. ;-)
>>
>>
> Thanks for your comments and questions! :)
>
>> >
>> >
>> > # Basics
>> >
>> > ---------------- ----------------------------------------------------
>> >          Status: **Tech Preview**
>> >
>> > Architecture(s): Intel x86
>> >
>> >    Component(s): Hypervisor, toolstack
>> >
>> >        Hardware: Atom codename Goldmont and beyond
>> > ---------------- ----------------------------------------------------
>> >
>> > # Overview
>> >
>> > L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
>> > CPU's shared L2 cache based on application priority or Class of Service
>> > (COS). Each CLOS is configured using capacity bitmasks (CBM) which
>> > represent cache capacity and indicate the degree of overlap and
>> > isolation between classes. Once L2 CAT is configured, the processor
>> > allows access to portions of L2 cache according to the established
>> > class of service (COS).
>>
>>
>>
>> The very earlier version of the design at [1] said, "In the initial
>> implementation, L2 CAT is shown up on Atom codename
>> Goldmont firstly and there is no platform support both L2 & L3 CAT so far."
>>
>> I'm wondering if there is any hardware supporting both L2 & L3 CAT now.
>>
>> [1]http://www.gossamer-threads.com/lists/xen/devel/431142
>>
>>
> Per my info, CURRENT HWs does not support both L2 and L3. But from SW view,
> we cannot limit this. If both features are enabled in the future HW, SW
> should be able to support it.

OK. I'm not sure if this is the best way to do it. Since the hw will
be manufactured by Intel, I guess you guys know what the future HW
will be. ;-)

Actually, the Atom processor that support L2 CAT is similar to the
Broadwell processor in the sense that the CAT is only for the
Last-Level Cache.
I'm looking forward to the L2 CAT in the processor that has three
level cache. ;-)

>
>> >
>> >
>> > # Technical information
>> >
>> > L2 CAT is a member of Intel PSR features and part of CAT, it shares
>> > some base PSR infrastructure in Xen.
>> >
>> > ## Hardware perspective
>> >
>> > L2 CAT defines a new range MSRs to assign different L2 cache access
>> > patterns which are known as CBMs (Capacity BitMask), each CBM is
>> > associated with a COS.
>> >
>> > ```
>> >
>> >                         +----------------------------+----------------+
>> >    IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
>> >  +----+---+-------+     +----------------------------+----------------+
>> >  |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
>> >  +----+---+-------+     +----------------------------+----------------+
>> >         └-------------> | ...                        |  ...           |
>> >                         +----------------------------+----------------+
>> >                         | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
>> >                         +----------------------------+----------------+
>> > ```
>> >
>> > When context switch happens, the COS of VCPU is written to per-thread
>> > MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
>> > according to the corresponding CBM.
>> >
>> > ## The relationship between L2 CAT and L3 CAT/CDP
>> >
>> > L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
>> > while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
>>
>> This sentence indicates that both L2 and L3 CAT can be enabled at the same 
>> time.
>> How can I know which CPU supports both L2 and L3 CAT?
>> Is the support architectural or just on several types of CPUs?
>> IMHO, it will be good to provide a reference about the type of CPUs
>> that supports L2 only, and that both L2 and L3.
>>
> Like above comment, no HWs support both L2 and L3 SO FAR but we should not
> limit this in SW. By CPUID tool or HW spec, you should know which feature
> is enabled on CPU.
>
>> >
>> > L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
>> > the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
>> > patterns from L3 cache.
>> >
>> > N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
>> > associate register `IA32_PQR_ASSOC`, which means one COS associate to a
>> > pair of L2 CBM and L3 CBM.
>> > Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
>> > other PSR features in future). In some cases, a VM is permitted to have a
>> > COS that is beyond one (or more) of PSR features but within the others.
>> > For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
>> > L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
>> > COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
>> > limit) since COS 9 is beyond the max COS (8) of L2 CAT.
>> >
>> > ## Design Overview
>> >
>> > * Core COS/CBM association
>> >
>> >   When enforcing L2 CAT, all cores of domains have the same default
>> >   COS (COS0) which associated to the fully open CBM (all ones bitmask)
>> >   to access all L2 cache. The default COS is used only in hypervisor
>> >   and is transparent to tool stack and user.
>> >
>> >   System administrator can change PSR allocation policy at runtime by
>> >   tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
>> >   to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
>> >   is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
>> >   L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
>> >   things would be easier, one COS corresponds to one L2 CBM.
>>
>>
>> I felt that this paragraph is kind of confusing. Can system admin
>> configure both L2 and L3 at the same time for a VM?
>> What does the 2-tuple, like [L2 CBM, L3 CBM], indicates here?
>>
> If HW can support both L2 and L3, admin can configure them one by one from
> SW view. To explain 2-tuple and 3-tuple, please refer below figure.
>
> 2-tuple:
>         -----------------------
>         | COS 0 | COS 1 | ... |
>         -----------------------
> L3 CAT  | 0x7ff | 0x3ff | ... |
>         -----------------------
> L2 CAT  | 0xff  | 0x3f  | ... |
>         -----------------------
>
> 3-tuple:
>         -----------------------------------------------------
>         | COS 0 DATA | COS 0 CODE | COS 1 DATA | COS 1 CODE |
>         -----------------------------------------------------
> L3 CDP  | 0x7ff      | 0x7ff      | 0x3ff      | 0x3ff      |
>         -----------------------------------------------------
>         |          COS 0          |         COS 1           |
>         -----------------------------------------------------
> L2 CAT  |          0xff           |         0x3f            |
>         -----------------------------------------------------
>
> There is only one ASSOC MSR register, so a CBM combination(e.g. 3-tuple) will
> work When you set COS id(e.g. 1) into ASSOC register. That means below CBM set
> works on the CPU.
> L3 CDP DATA=0x3ff
> L3 CDP CODE=0x3ff
> L2 CAT = 0x3f

I see. Thank you very much for the explanation!

>
>> In addition, does the bits in CBM have to be continuous?
>>
> Yes, all CBM bits should be continuous. Thanks for pointing this out!

;-)

>
>> > # User information
>> >
>> > * Feature Enabling:
>> >
>> >   Add "psr=cat" to boot line parameter to enable all supported level CAT
>> >   features.
>> >
>> > * xl interfaces:
>> >
>> >   1. `psr-cat-show [OPTIONS] domain-id`:
>> >
>> >      Show domain L2 or L3 CAT CBM.
>> >
>> >      New option `-l` is added.
>> >      `-l2`: Specify cbm for L2 cache.
>> >      `-l3`: Specify cbm for L3 cache.
>> >
>> >      If neither `-l2` nor `-l3` is given, LLC (Last Level Cache) is
>> >      default behavior.
>>
>> Does it mean the LLC CAT setting will be reset or just kept to the
>> latest configuration?
>>
> Here, it means LLC CAT value will be shown.

Then the explanation " If neither `-l2` nor `-l3` is given, LLC (Last
Level Cache) is  default behavior." could change to
"If neither `-l2` nor `-l3` is given, it will output the current LLC
(Last Level Cache) configuration"

Thank you very much!

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel