Re: [PATCH v5 09/13] xen: add cache coloring allocator for domains

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 9 Jan 2024, Jan Beulich wrote:
> On 02.01.2024 10:51, Carlo Nonato wrote:
> > This commit adds a new memory page allocator that implements the cache
> > coloring mechanism. The allocation algorithm enforces equal frequency
> > distribution of cache partitions, following the coloring configuration of a
> > domain. This allows an even utilization of cache sets for every domain.
> > 
> > Pages are stored in a color-indexed array of lists. Those lists are filled
> > by a simple init function which computes the color of each page.
> > When a domain requests a page, the allocator extract the page from the list
> > with the maximum number of free pages between those that the domain can
> > access, given its coloring configuration.
> > 
> > The allocator can only handle requests of order-0 pages. This allows for
> > easier implementation and since cache coloring targets only embedded 
> > systems,
> > it's assumed not to be a major problem.
> 
> I'm curious about the specific properties of embedded systems that makes
> the performance implications of deeper page walks less of an issue for
> them.

I think Carlo meant to say that embedded systems tend to have a smaller
amount of RAM (our boards today have 4-8GB of total memory). So higher
level allocations (2MB/1GB) might not be possible.

Also, domains that care about interrupt latency tend to be RTOSes (e.g.
Zephyr, FreeRTOS) and RTOSes are happy to run with less than 1MB of
total memory available. This is so true that I vaguely remember hitting
a bug in xl/libxl when I tried to create a domain with 128KB of memory. 


> Nothing is said about address-constrained allocations. Are such entirely
> of no interest to domains on Arm, not even to Dom0 (e.g. for filling
> Linux'es swiotlb)?

Cache coloring is useful if you can use an IOMMU with all the
dma-capable devices. If that is not the case, then not even Dom0 would
be able to boot with cache coloring enabled (because it wouldn't be 1:1
mapped).

On ARM we only support booting Dom0 1:1 mapped, or not-1:1-mapped but
relying on the IOMMU.


> Certainly alloc_color_heap_page() should at least fail when it can't
> satisfy the passed in memflags.

[...]


> > +void dump_color_heap(void)
> > +{
> > +    unsigned int color;
> > +
> > +    printk("Dumping color heap info\n");
> > +    for ( color = 0; color < get_nr_llc_colors(); color++ )
> > +        if ( free_colored_pages[color] > 0 )
> > +            printk("Color heap[%u]: %lu pages\n",
> > +                   color, free_colored_pages[color]);
> > +}
> 
> What's a typical range of number of colors on a system? I expect more
> than 9, but I'm not sure about a reasonable upper bound. For the
> output to be easy to consume, [%u] may want to become at least [%2u].

16 colors on our boards