Xen project Mailing List

Re: [Xen-devel] [PATCH 0/3] Enable L2 Cache Allocation Technology

On 16-09-17 11:31:54, Meng Xu wrote: > On Thu, Aug 25, 2016 at 1:21 AM, Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx> wrote: > > > > Design document is below: > > ======================================================================= > > % Intel L2 Cache Allocation Technology (L2 CAT) Feature > > % Revision 1.0 > > > > \clearpage > > > > Hi all, > > > > We plan to bring a new PSR (Platform Shared Resource) feature called > > Intel L2 Cache Allocation Technology (L2 CAT) to Xen. > > > > This is the initial design of L2 CAT. It might be a little long and > > detailed, hope it doesn't matter. > > > > Besides the L2 CAT implementaion, we refactor the psr.c to make it more > > flexible to add new features and fulfill the principle, open for extension > > but closed for modification. > > > > Comments and suggestions are welcome :-) > > > I have some comments/questions. ;-) > > Thanks for your comments and questions! :) > > > > > > # Basics > > > > ---------------- ---------------------------------------------------- > > Status: **Tech Preview** > > > > Architecture(s): Intel x86 > > > > Component(s): Hypervisor, toolstack > > > > Hardware: Atom codename Goldmont and beyond > > ---------------- ---------------------------------------------------- > > > > # Overview > > > > L2 CAT allows an OS or Hypervisor/VMM to control allocation of a > > CPU's shared L2 cache based on application priority or Class of Service > > (COS). Each CLOS is configured using capacity bitmasks (CBM) which > > represent cache capacity and indicate the degree of overlap and > > isolation between classes. Once L2 CAT is configured, the processor > > allows access to portions of L2 cache according to the established > > class of service (COS). > > > > The very earlier version of the design at [1] said, "In the initial > implementation, L2 CAT is shown up on Atom codename > Goldmont firstly and there is no platform support both L2 & L3 CAT so far." > > I'm wondering if there is any hardware supporting both L2 & L3 CAT now. > > [1]http://www.gossamer-threads.com/lists/xen/devel/431142 > > Per my info, CURRENT HWs does not support both L2 and L3. But from SW view, we cannot limit this. If both features are enabled in the future HW, SW should be able to support it. > > > > > > # Technical information > > > > L2 CAT is a member of Intel PSR features and part of CAT, it shares > > some base PSR infrastructure in Xen. > > > > ## Hardware perspective > > > > L2 CAT defines a new range MSRs to assign different L2 cache access > > patterns which are known as CBMs (Capacity BitMask), each CBM is > > associated with a COS. > > > > ``` > > > > +----------------------------+----------------+ > > IA32_PQR_ASSOC | MSR (per socket) | Address | > > +----+---+-------+ +----------------------------+----------------+ > > | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | > > +----+---+-------+ +----------------------------+----------------+ > > └-------------> | ... | ... | > > +----------------------------+----------------+ > > | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | > > +----------------------------+----------------+ > > ``` > > > > When context switch happens, the COS of VCPU is written to per-thread > > MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation > > according to the corresponding CBM. > > > > ## The relationship between L2 CAT and L3 CAT/CDP > > > > L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled > > while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled. > > This sentence indicates that both L2 and L3 CAT can be enabled at the same > time. > How can I know which CPU supports both L2 and L3 CAT? > Is the support architectural or just on several types of CPUs? > IMHO, it will be good to provide a reference about the type of CPUs > that supports L2 only, and that both L2 and L3. > Like above comment, no HWs support both L2 and L3 SO FAR but we should not limit this in SW. By CPUID tool or HW spec, you should know which feature is enabled on CPU. > > > > L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by > > the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing > > patterns from L3 cache. > > > > N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same > > associate register `IA32_PQR_ASSOC`, which means one COS associate to a > > pair of L2 CBM and L3 CBM. > > Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or > > other PSR features in future). In some cases, a VM is permitted to have a > > COS that is beyond one (or more) of PSR features but within the others. > > For instance, let's assume the max COS of L2 CAT is 8 but the max COS of > > L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to > > COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no > > limit) since COS 9 is beyond the max COS (8) of L2 CAT. > > > > ## Design Overview > > > > * Core COS/CBM association > > > > When enforcing L2 CAT, all cores of domains have the same default > > COS (COS0) which associated to the fully open CBM (all ones bitmask) > > to access all L2 cache. The default COS is used only in hypervisor > > and is transparent to tool stack and user. > > > > System administrator can change PSR allocation policy at runtime by > > tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds > > to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP > > is enabled, one COS corresponds to a 3-tuple, like [L2 CBM, > > L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled, > > things would be easier, one COS corresponds to one L2 CBM. > > > I felt that this paragraph is kind of confusing. Can system admin > configure both L2 and L3 at the same time for a VM? > What does the 2-tuple, like [L2 CBM, L3 CBM], indicates here? > If HW can support both L2 and L3, admin can configure them one by one from SW view. To explain 2-tuple and 3-tuple, please refer below figure. 2-tuple: ----------------------- | COS 0 | COS 1 | ... | ----------------------- L3 CAT | 0x7ff | 0x3ff | ... | ----------------------- L2 CAT | 0xff | 0x3f | ... | ----------------------- 3-tuple: ----------------------------------------------------- | COS 0 DATA | COS 0 CODE | COS 1 DATA | COS 1 CODE | ----------------------------------------------------- L3 CDP | 0x7ff | 0x7ff | 0x3ff | 0x3ff | ----------------------------------------------------- | COS 0 | COS 1 | ----------------------------------------------------- L2 CAT | 0xff | 0x3f | ----------------------------------------------------- There is only one ASSOC MSR register, so a CBM combination(e.g. 3-tuple) will work When you set COS id(e.g. 1) into ASSOC register. That means below CBM set works on the CPU. L3 CDP DATA=0x3ff L3 CDP CODE=0x3ff L2 CAT = 0x3f > In addition, does the bits in CBM have to be continuous? > Yes, all CBM bits should be continuous. Thanks for pointing this out! > > # User information > > > > * Feature Enabling: > > > > Add "psr=cat" to boot line parameter to enable all supported level CAT > > features. > > > > * xl interfaces: > > > > 1. `psr-cat-show [OPTIONS] domain-id`: > > > > Show domain L2 or L3 CAT CBM. > > > > New option `-l` is added. > > `-l2`: Specify cbm for L2 cache. > > `-l3`: Specify cbm for L3 cache. > > > > If neither `-l2` nor `-l3` is given, LLC (Last Level Cache) is > > default behavior. > > Does it mean the LLC CAT setting will be reset or just kept to the > latest configuration? > Here, it means LLC CAT value will be shown. > > > > 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: > > > > Set domain L2 or L3 CBM. > > > > New option `-l` is added. > > `-l2`: Specify cbm for L2 cache. > > `-l3`: Specify cbm for L3 cache. > > > > If neither `-l2` nor `-l3` is given, LLC (Last Level Cache) is > > default behavior. Since there is no CDP support on L2 cache, so if > > `-l2`, `--code` or `--data` are both given, error message shows. > > > > 3. `psr-hwinfo [OPTIONS]`: > > > > Show L2 & L3 CAT HW informations on every socket. > > > > # References > > > > [Intel® 64 and IA-32 Architectures Software Developer > > Manuals](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) > > Would you mind provide the chapter for the L2 CAT and the version of > the SDM you are referring? > It makes people's life easier to find the reference. ;-) > Thank you! I will add it in next version. You can download the latest version from the link. L2 CAT is vol3, chapter 17.17. > > > > # History > > > > ------------------------------------------------------------------------ > > Date Revision Version Notes > > ---------- -------- -------- ------------------------------------------- > > 2016-08-12 1.0 Xen 4.7 Initial design > > It seems there is a RFC design in May this year: > http://www.gossamer-threads.com/lists/xen/devel/431142 > > Shouldn't it be counted? > Ok, thanks! > Thanks, > > Meng > > ----------- > Meng Xu > PhD Student in Computer and Information Science > University of Pennsylvania > http://www.cis.upenn.edu/~mengxu/ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@xxxxxxxxxxxxx > https://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.