Re: [PATCH v8 09/13] xen: add cache coloring allocator for domains

[Jan Beulich <jbeulich@xxxxxxxx>]

On 02.05.2024 18:55, Carlo Nonato wrote:
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -270,6 +270,20 @@ and not running softirqs. Reduce this if softirqs are 
> not being run frequently
>  enough. Setting this to a high value may cause boot failure, particularly if
>  the NMI watchdog is also enabled.
>  
> +### buddy-alloc-size (arm64)

I'd like to ask for consistency in the command line doc additions: Either
"(arm64)" here and then also in patch 1, or (less desirable) nowhere.

> @@ -1481,6 +1487,14 @@ static void free_heap_pages(
>              pg[i].count_info |= PGC_need_scrub;
>              poison_one_page(&pg[i]);
>          }
> +
> +        if ( pg->count_info & PGC_colored )
> +        {
> +            /* Colored pages can be 0-order only, so ignore it */
> +            free_color_heap_page(pg, need_scrub);
> +            spin_unlock(&heap_lock);
> +            return;
> +        }

Why "ignore it"? You pass the page to free_color_heap_page() after all.
(later) Oh, you may mean to ignore "order". Yet besides this not really
becoming clear from the wording, I think that this constraint could do
with asserting here.

> @@ -1945,6 +1959,156 @@ static unsigned long avail_heap_pages(
>      return free_pages;
>  }
>  
> +/*************************
> + * COLORED SIDE-ALLOCATOR
> + *
> + * Pages are grouped by LLC color in lists which are globally referred to as 
> the
> + * color heap. Lists are populated in end_boot_allocator().
> + * After initialization there will be N lists where N is the number of
> + * available colors on the platform.
> + */
> +static struct page_list_head *__ro_after_init _color_heap;
> +#define color_heap(color) (&_color_heap[color])
> +
> +static unsigned long *__ro_after_init free_colored_pages;
> +
> +/* Memory required for buddy allocator to work with colored one */
> +#ifdef CONFIG_LLC_COLORING
> +static unsigned long __initdata buddy_alloc_size =
> +    MB(CONFIG_BUDDY_ALLOCATOR_SIZE);

I think it would be quite nice if this and ...

> +size_param("buddy-alloc-size", buddy_alloc_size);
> +
> +#define domain_num_llc_colors(d) (d)->num_llc_colors
> +#define domain_llc_color(d, i)   (d)->llc_colors[i]
> +#else
> +static unsigned long __initdata buddy_alloc_size;

... this were folded. Which I think would be possible if the Kconfig
addition went like this:

config BUDDY_ALLOCATOR_SIZE
        int "Buddy allocator reserved memory size (MiB)" if LLC_COLORING
        default "0" if !LLC_COLORING
        default "64"

> +static struct page_info *alloc_color_heap_page(unsigned int memflags,
> +                                               const struct domain *d)
> +{
> +    struct page_info *pg = NULL;
> +    unsigned int i, color = 0;
> +    unsigned long max = 0;
> +    bool need_tlbflush = false;
> +    uint32_t tlbflush_timestamp = 0;
> +    bool need_scrub;
> +
> +    if ( memflags & ~(MEMF_no_owner | MEMF_no_refcount | MEMF_no_scrub |
> +                      MEMF_no_tlbflush) )
> +        return NULL;
> +
> +    spin_lock(&heap_lock);
> +
> +    for ( i = 0; i < domain_num_llc_colors(d); i++ )
> +    {
> +        unsigned long free = free_colored_pages[domain_llc_color(d, i)];
> +
> +        if ( free > max )
> +        {
> +            color = domain_llc_color(d, i);
> +            pg = page_list_first(color_heap(color));
> +            max = free;
> +        }
> +    }
> +
> +    if ( !pg )
> +    {
> +        spin_unlock(&heap_lock);
> +        return NULL;
> +    }
> +
> +    need_scrub = pg->count_info & (PGC_need_scrub);

Nit: No need for parentheses here.

Jan