Re: [PATCH DO NOT APPLY] docs: Document allocator properties and the rubric for using them

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Feb 16, 2021, at 3:29 PM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
> 
> On 16.02.2021 11:28, George Dunlap wrote:
>> --- /dev/null
>> +++ b/docs/hypervisor-guide/memory-allocation-functions.rst
>> @@ -0,0 +1,118 @@
>> +.. SPDX-License-Identifier: CC-BY-4.0
>> +
>> +Xenheap memory allocation functions
>> +===================================
>> +
>> +In general Xen contains two pools (or "heaps") of memory: the *xen
>> +heap* and the *dom heap*.  Please see the comment at the top of
>> +``xen/common/page_alloc.c`` for the canonical explanation.
>> +
>> +This document describes the various functions available to allocate
>> +memory from the xen heap: their properties and rules for when they should be
>> +used.
> 
> Irrespective of your subsequent indication of you disliking the
> proposal (which I understand only affects the guidelines further
> down anyway) I'd like to point out that vmalloc() does not
> allocate from the Xen heap. Therefore a benefit of always
> recommending use of xvmalloc() would be that the function could
> fall back to vmalloc() (and hence the larger domain heap) when
> xmalloc() failed.

OK, that’s good to know.

So just trying to think this through: address space is limiting factor for how 
big the xenheap can be, right?  Presumably “vmap” space is also limited, and 
will be much smaller?  So in a sense the “fallback” is less about getting more 
memory, but about using up that extra little bit of virtual address space?

Another question that raises:  Are there times when it’s advantageous to 
specify which heap to allocate from?  If there are good reasons for allocations 
to be in the xenheap or in the domheap / vmap area, then the guidelines should 
probably say that as well.

And, of course, will the whole concept of the xenheap / domheap split go away 
if we ever get rid of the 1:1 map?

> 
>> +TLDR guidelines
>> +---------------
>> +
>> +* By default, ``xvmalloc`` (or its helper cognates) should be used
>> +  unless you know you have specific properties that need to be met.
>> +
>> +* If you need memory which needs to be physically contiguous, and may
>> +  be larger than ``PAGE_SIZE``...
>> +  
>> +  - ...and is order 2, use ``alloc_xenheap_pages``.
>> +    
>> +  - ...and is not order 2, use ``xmalloc`` (or its helper cognates)..
> 
> ITYM "an exact power of 2 number of pages"?

Yes, I’ll fix that.

> 
>> +* If you don't need memory to be physically contiguous, and know the
>> +  allocation will always be larger than ``PAGE_SIZE``, you may use
>> +  ``vmalloc`` (or one of its helper cognates).
>> +
>> +* If you know that allocation will always be less than ``PAGE_SIZE``,
>> +  you may use ``xmalloc``.
> 
> As per Julien's and your own replies, this wants to be "minimum
> possible page size", which of course depends on where in the
> tree the piece of code is to live. (It would be "maximum
> possible page size" in the earlier paragraph.)

I’ll see if I can clarify this.

> 
>> +Properties of various allocation functions
>> +------------------------------------------
>> +
>> +Ultimately, the underlying allocator for all of these functions is
>> +``alloc_xenheap_pages``.  They differ on several different properties:
>> +
>> +1. What underlying allocation sizes are.  This in turn has an effect
>> +   on:
>> +
>> +   - How much memory is wasted when requested size doesn't match
>> +
>> +   - How such allocations are affected by memory fragmentation
>> +
>> +   - How such allocations affect memory fragmentation
>> +
>> +2. Whether the underlying pages are physically contiguous
>> +
>> +3. Whether allocation and deallocation require the cost of mapping and
>> +   unmapping
>> +
>> +``alloc_xenheap_pages`` will allocate a physically contiguous set of
>> +pages on orders of 2.  No mapping or unmapping is done.
> 
> That's the case today, but meant to change rather sooner than later
> (when the 1:1 map disappears).

Is that the kind of thing we want to add into this document?  I suppose it 
would be good to make the guidelines now such that they produce code which is 
as easy as possible to adapt to the new way of doing things.

 -George