Re: [Xen-devel] [PATCH v3 01/15] docs: create Memory Bandwidth Allocation (MBA) feature document

[Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>]

On 17-09-18 18:16:40, Roger Pau Monn� wrote:
> On Tue, Sep 05, 2017 at 05:32:23PM +0800, Yi Sun wrote:
> > +* xl interfaces:
> > +
> > +  1. `psr-mba-show [domain-id]`:
> 
> Is this limited to domain-id, or one can also use the domain name?
> Most of the xl commands accept either a domain-id or a domain-name.
> 
Both domain-id and domain-name can show it. I thought this is by default and
no need to explicitly declare. If I am wrong, I will change it as below:
    `psr-mba-show [domain-id/domain-name]`

[...]

> > +  2. `psr-mba-set [OPTIONS] <domain-id> <throttling>`:
> > +
> > +     Set memory bandwidth throttling for domain.
> > +
> > +     Options:
> > +     '-s': Specify the socket to process, otherwise all sockets are 
> > processed.
> > +
> > +     Throttling value set in register implies the approximate amount of 
> > delaying
> > +     the traffic between core and memory. The higher throttling value 
> > results in
> > +     lower bandwidth. The max throttling value (MBA_MAX) supported can be 
> > got
> 
> s/got/obtained/
> 
Thanks!

> > +     through CPUID.
> 
> How can one get this value empirically? Do I need to use a external
> tool?
> 
Sorry for confusion. In fact, the MBA_MAX is got through CPUID in hypervisor.
User can know it through psr-hwinfo. Will explain it.

> > +
> > +     Linear mode: the input precision is defined as 100-(MBA_MAX). For 
> > instance,
> > +     if the MBA_MAX value is 90, the input precision is 10%. Values not an 
> > even
> > +     multiple of the precision (e.g., 12%) will be rounded down (e.g., to 
> > 10%
> > +     delay applied) by HW automatically.
> > +
> > +     Non-linear mode: input delay values are powers-of-two from zero to the
> > +     MBA_MAX value from CPUID. In this case any values not a power of two 
> > will
> > +     be rounded down the next nearest power of two by HW automatically.
> 
> Both of the above descriptions should be moved to mba-show IMHO, the
> description there is incomplete and not helpful.
> 
Ok, thanks!

> > +
> > +# Technical details
> > +
> > +MBA is a member of Intel PSR features, it shares the base PSR 
> > infrastructure
> > +in Xen.
> > +
> > +## Hardware perspective
> > +
> > +  MBA defines a range of MSRs to support specifying a delay value (Thrtl) 
> > per
> > +  COS, with details below.
> > +
> > +  ```
> > +   +----------------------------+----------------+
> > +   | MSR (per socket)           |    Address     |
> > +   +----------------------------+----------------+
> > +   | IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
> > +   +----------------------------+----------------+
> > +   | ...                        |  ...           |
> > +   +----------------------------+----------------+
> > +   | IA32_L2_QOS_Ext_BW_Thrtl_n |     0xD50+n    |
> > +   +----------------------------+----------------+
> > +  ```
> > +
> > +  When context switch happens, the COS ID of domain is written to 
> > per-thread MSR
> > +  `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation 
> > according
> 
> I think this is missing some context of the relation between a thread
> and the MSR. I assume it's related to IA32_PQR_ASSOC, but I have no
> idea what that constant means.
> 
> What's more, Xen doesn't have threads, so you should maybe speak about
> vCPUs instead?
> 
As Jan's comment, this is for 'per-hyper-thread'.

[...]

> > +## Implementation Description
> > +
> > +* Hypervisor interfaces:
> > +
> > +  1. Boot line param: "psr=mba" to enable the feature.
> > +
> > +  2. SYSCTL:
> > +          - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
> 
> So this is likely how one gets the mentioned MBA_MAX?
> 
Yup.

> > +
> > +  3. DOMCTL:
> > +          - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get throttling for a domain.
> > +          - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set throttling for a domain.
> > +
> > +* xl interfaces:
> > +
> > +  1. psr-mba-show [domain-id]
> > +          Show system/domain runtime MBA throttling value. For linear mode,
> > +          it shows the decimal value. For non-linear mode, it shows 
> > hexadecimal
> > +          value.
> > +          => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
> > +
> > +  2. psr-mba-set [OPTIONS] <domain-id> <throttling>
> > +          Set bandwidth throttling for a domain.
> > +          => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
> > +
> > +  3. psr-hwinfo
> > +          Show PSR HW information, including L3 CAT/CDP/L2 CAT/MBA.
> > +          => XEN_SYSCTL_PSR_MBA_get_info
> 
> 'psr-hwinfo' seems to be completely missing from the 'xl interfaces:'
> section above.
> 
Because this is not a newly added interface, I do not describe it in 'xl
interfaces'. Is that necessary?

> > +* Key data structure:
> > +
> > +  1. Feature HW info
> > +
> > +     ```
> > +     struct {
> > +         unsigned int thrtl_max;
> > +         bool linear;
> > +     } mba;
> > +
> > +     - Member `thrtl_max`
> > +
> > +       `thrtl_max` is the max throttling value to be set, i.e. MBA_MAX.
> > +
> > +     - Member `linear`
> > +
> > +       `linear` means the response of delay value is linear or not.
> > +
> > +     As mentioned above, MBA is a member of Intel PSR features, it would
> > +     share the base PSR infrastructure in Xen. For example, the 'cos_max'
> > +     is a common HW property for all features. So, for other data structure
> > +     details, please refer 'intel_psr_cat_cdp.pandoc'.
>                              ^ to
Thanks!

> > +
> > +# Limitations
> > +
> > +MBA can only work on HW which enables it (check by CPUID).
>                                  ^ s/enables/supports/.
Thanks!

> > +
> > +# Testing
> > +
> > +We can execute these commands to verify MBA on different HWs supporting 
> > them.
> > +
> > +For example:
> > +  1. User can get the MBA hardware info through 'psr-hwinfo' command. From
> > +     result, user can know if this hardware works under linear mode or non-
> > +     linear mode, the max throttling value (MBA_MAX) and so on.
> > +
> > +    root@:~$ xl psr-hwinfo --mba
> > +    Memory Bandwidth Allocation (MBA):
> > +    Socket ID       : 0
> > +    Linear Mode     : Enabled
> > +    Maximum COS     : 7
> > +    Maximum Throttling Value: 90
> > +    Default Throttling Value: 0
> > +
> > +  2. Then, user can set a throttling value to a domain. For example, set 
> > '0xa',
> > +     i.e 10% delay.
> > +
> > +    root@:~$ xl psr-mba-set 1 0xa
> > +
> > +  3. User can check the current configuration of the domain through
> > +     'psr-mab-show'. For linear mode, the decimal value is shown.
> > +
> > +    root@:~$ xl psr-mba-show 1
> > +    Socket ID       : 0
> > +    Default THRTL   : 0
> > +       ID                     NAME            THRTL
> > +        1                 ubuntu14             10
> 
> The example seems better now IMHO.
> 
> Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel