Re: [Xen-devel] [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document

[Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>]

On 17-01-20 09:39:41, Tian, Kevin wrote:
> > From: Yi Sun
> > Sent: Thursday, January 19, 2017 2:01 PM
> > 
> > This patch creates L2 CAT feature document in doc/features/.
> > It describes details of L2 CAT.
> 
> A good write-up, but still some improvements required. :-)
> 
Thanks a lot for review and the good suggestions!

> > 
> > Signed-off-by: Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>
> > ---
> >  docs/features/intel_psr_l2_cat.pandoc | 347
> > ++++++++++++++++++++++++++++++++++
> >  1 file changed, 347 insertions(+)
> >  create mode 100644 docs/features/intel_psr_l2_cat.pandoc
> > 
> > diff --git a/docs/features/intel_psr_l2_cat.pandoc
> > b/docs/features/intel_psr_l2_cat.pandoc
> > new file mode 100644
> > index 0000000..77bd61f
> > --- /dev/null
> > +++ b/docs/features/intel_psr_l2_cat.pandoc
> > @@ -0,0 +1,347 @@
> > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> > +% Revision 1.0
> > +
> > +\clearpage
> > +
> > +# Basics
> > +
> > +---------------- ----------------------------------------------------
> > +         Status: **Tech Preview**
> > +
> > +Architecture(s): Intel x86
> > +
> > +   Component(s): Hypervisor, toolstack
> > +
> > +       Hardware: Atom codename Goldmont and beyond CPUs
> > +---------------- ----------------------------------------------------
> > +
> > +# Overview
> > +
> > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> > +CPU's shared L2 cache based on application priority or Class of Service
> > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> > +represent cache capacity and indicate the degree of overlap and
> > +isolation between classes. Once L2 CAT is configured, the processor
> > +allows access to portions of L2 cache according to the established
> > +class of service.
> 
> I would suggest make this doc for all CAT features, otherwise some
> content looks incomplete when you say adding new options or new
> ranges with assumption that reader understands the existing stuff.
> Yes I do notice you provide many background about L3 CAT/CDP. Let's
> just make it complete.
> 
> Also as a permanent design, 'new' is not preferred since it will become
> 'existing' once it's checked in.
> 
We have discussed this before. I proposed to add L3 CAT/CDP feature documents
later. But considering readers may not have background knowledge, I will try
to change this feature document to cover all CAT/CDP features.

> > +
> > +     New option `-l` is added.
> > +     `-l2`: Specify cbm for L2 cache.
> > +     `-l3`: Specify cbm for L3 cache.
> > +
> > +     If neither `-l2` nor `-l3` is given, level 3 is the default option.
> > +
> > +  3. `psr-hwinfo [OPTIONS]`:
> > +
> > +     Show L2 & L3 CAT HW informations on every socket.
> 
> informations -> information
> 
> and which HW information? please elaborate.
> 
> and where is the interface of setting COS for VCPU?
> 
What do you mean 'the interface of setting COS for VCPU'? User can set CBM
for one domain through 'psr-cat-cbm-set' now. Then, hypervisor will find
a matched COS ID or pick an available COS ID for the domain. User does not
need know which COS ID it is using.

> > +When context switch happens, the COS of VCPU is written to per-thread
> > +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> > +according to the corresponding CBM.
> > +
> > +## The relationship between L2 CAT and L3 CAT/CDP
> > +
> > +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> > +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
> > +
> > +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
> > +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> > +patterns from L3 cache. 
> 
> "L2 CAT supports ..." or "L3 CAT supports..." for the last sentence? 
> 
It is 'L2 CAT'. Sorry for confusion.

> > Like L3 CAT/CDP requirement, the bits of CBM of
> > +L2 CAT must be continuous too.
> > +
> > +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> > +associate register `IA32_PQR_ASSOC`, which means one COS associates to a
> > +pair of L2 CBM and L3 CBM.
> > +
> > +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> > +other PSR features in future). In some cases, a VM is permitted to have a
> > +COS that is beyond one (or more) of PSR features but within the others.
> > +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> > +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> > +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> > +limit) since COS 9 is beyond the max COS (8) of L2 CAT.
> 
> Does user space need to know such difference since L2 CAT may not 
> be effective for this vCPU?
> .."
> 
In fact, L2 CAT will use default value for such case. This is done by HW.
User space can know the max COS differences through 'psr-hwinfo'.

> > +
> > +* Multi-sockets
> > +
> > +  Different sockets may have different L2 CAT capability (e.g. max COS)
> > +  although it is consistent on the same socket. So the capability of
> > +  per-socket L2 CAT is specified.
> 
> VCPU schedule design is important here. e.g. when migrating a VCPU 
> from socket 1 to socket 2 (socket 1 has max COS as 16 while socket
> 2 as 8), will you prevent migration if VCPU has a COS (9)?
>
'psr-cat-cbm-set' can set CBM for one domain per socket. On each socket, we
maintain a COS array for all domains. One domain uses one COS at one time. One
COS saves the CBM of to work. So, when a VCPU of the domain is migrated from
socket 1 to socket 2, it follows configuration on socket 2. This mechanism has
been implemented since L3 CAT. L2 CAT follows it.

E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets
domain 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
is migrated from socket 1 to 2, the COS ID used will be 7, that means 0x3f
will be the CBM to work for this domain.

Thanks,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel