Re: [PATCH DO NOT APPLY] docs: Document allocator properties and the rubric for using them

[Jan Beulich <jbeulich@xxxxxxxx>]

On 12.03.2021 15:32, George Dunlap wrote:
>> On Feb 16, 2021, at 3:29 PM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>> On 16.02.2021 11:28, George Dunlap wrote:
>>> --- /dev/null
>>> +++ b/docs/hypervisor-guide/memory-allocation-functions.rst
>>> @@ -0,0 +1,118 @@
>>> +.. SPDX-License-Identifier: CC-BY-4.0
>>> +
>>> +Xenheap memory allocation functions
>>> +===================================
>>> +
>>> +In general Xen contains two pools (or "heaps") of memory: the *xen
>>> +heap* and the *dom heap*.  Please see the comment at the top of
>>> +``xen/common/page_alloc.c`` for the canonical explanation.
>>> +
>>> +This document describes the various functions available to allocate
>>> +memory from the xen heap: their properties and rules for when they should 
>>> be
>>> +used.
>>
>> Irrespective of your subsequent indication of you disliking the
>> proposal (which I understand only affects the guidelines further
>> down anyway) I'd like to point out that vmalloc() does not
>> allocate from the Xen heap. Therefore a benefit of always
>> recommending use of xvmalloc() would be that the function could
>> fall back to vmalloc() (and hence the larger domain heap) when
>> xmalloc() failed.
> 
> OK, that’s good to know.
> 
> So just trying to think this through: address space is limiting factor for 
> how big the xenheap can be, right?

Yes, with the current direct-map model only memory which has a
permanent mapping can be "Xen heap". Obviously, for the mapping
to be permanent, its VA range needs to be set up front (at
build time in reality). FAOD the distinction (at least on x86)
matters only on systems with a lot of memory.

>  Presumably “vmap” space is also limited, and will be much smaller?

Yes and yes, albeit for the 2nd one I'd like to add "currently",
because once we do away with the direct map, I'd envision to use
all the VA space for such on-demand mapping purposes.

>  So in a sense the “fallback” is less about getting more memory,
> but about using up that extra little bit of virtual address space?

Not really, no. If no memory is left on the Xen heap, there may
still be some left on the domain heap.

Falling back could also be the other way around, yes - if we've
run out of vmalloc() address space, we may still have a chance
find the requested space in the Xen heap.

> Another question that raises:  Are there times when it’s
> advantageous to specify which heap to allocate from?  If there
> are good reasons for allocations to be in the xenheap or in the
> domheap / vmap area, then the guidelines should probably say
> that as well.

I can't think of such reasons (beyond ones already named, like
e.g. wanting to avoid mapping overhead), but I agree that if
there are any, mentioning them would be desirable.

> And, of course, will the whole concept of the xenheap / domheap
> split go away if we ever get rid of the 1:1 map?

I expect so, yes.

>>> +Properties of various allocation functions
>>> +------------------------------------------
>>> +
>>> +Ultimately, the underlying allocator for all of these functions is
>>> +``alloc_xenheap_pages``.  They differ on several different properties:
>>> +
>>> +1. What underlying allocation sizes are.  This in turn has an effect
>>> +   on:
>>> +
>>> +   - How much memory is wasted when requested size doesn't match
>>> +
>>> +   - How such allocations are affected by memory fragmentation
>>> +
>>> +   - How such allocations affect memory fragmentation
>>> +
>>> +2. Whether the underlying pages are physically contiguous
>>> +
>>> +3. Whether allocation and deallocation require the cost of mapping and
>>> +   unmapping
>>> +
>>> +``alloc_xenheap_pages`` will allocate a physically contiguous set of
>>> +pages on orders of 2.  No mapping or unmapping is done.
>>
>> That's the case today, but meant to change rather sooner than later
>> (when the 1:1 map disappears).
> 
> Is that the kind of thing we want to add into this document?

Not sure what to answer here - my intention with raising the point
was ...

>  I suppose it would be good to make the guidelines now such
> that they produce code which is as easy as possible to adapt
> to the new way of doing things.

... precisely this.

Jan