Re: [Xen-devel] [PATCH v2] docs: clarify xl mem-max semantics

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 24/01/2017 21:47, Jim Fehlig wrote:
> On 01/23/2017 02:49 AM, Juergen Gross wrote:
>> The information given in the xl man page for the mem-max command is
>> rather brief. Expand it in order to let the reader understand what it
>> is really doing.
>>
>> As the related libxl function libxl_domain_setmaxmem() isn't much
>> clearer add a comment to it explaining the desired semantics.
>>
>> Signed-off-by: Juergen Gross <jgross@xxxxxxxx>
>> ---
>> V2: rewording as suggested by Jim Fehlig and Ian Jackson
>> ---
>>  docs/man/xl.pod.1.in | 10 ++++++++++
>>  tools/libxl/libxl.c  |  6 ++++++
>>  2 files changed, 16 insertions(+)
>>
>> diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
>> index 09c1faa..7caed08 100644
>> --- a/docs/man/xl.pod.1.in
>> +++ b/docs/man/xl.pod.1.in
>> @@ -401,6 +401,16 @@ for bytes.
>>  The mem-max value may not correspond to the actual memory used in the
>>  domain, as it may balloon down its memory to give more back to the OS.
>>
>> +The value given just sets the memory amount the domain is allowed to
>> allocate
>> +in the hypervisor. It can't be set lower than the current
>> reservation, but
>> +it is allowed to be higher than the configured maximum memory size
>> of the
>> +domain (B<maxmem> parameter in the domain's configuration). Using
>> B<xl mem-max>
>> +to set the maximum memory above the initial B<maxmem> value will not
>> allow the
>> +additional memory to be used via B<xl mem-set>. The initial
>> B<maxmem> value is
>> +still used as an upper limit for B<xl mem-set>.
>> +
>> +The domain is not receiving any signal regarding the changed memory
>> limit.
>> +
>>  =item B<mem-set> I<domain-id> I<mem>
>>
>>  Set the domain's used memory using the balloon driver; append 't' for
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 0622311..ed59510 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -4018,6 +4018,12 @@ out:
>>
>>  
>> /******************************************************************************/
>>
>>
>> +/*
>> + * Set the maximum memory size of the domain in the hypervisor.
>> There is no
>> + * change of the current memory size involved. The allowed memory
>> size can
>
> s/allowed/specified/ ? Or perhaps "The value of max_memkb can even..."
> ? Sorry I didn't look closely at this hunk in V1.
>
>> + * even be above the configured maxmem size of the domain, but the
>> related
>> + * Xenstore entry memory/static-max isn't modified!
>> + */
>>  int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint64_t
>> max_memkb)
>>  {
>>      GC_INIT(ctx);
>
> I have a related question for you (or Ian, Wei, ...): What does
> max_pages (as seen with 'q' debug key') represent? How is it related
> to memory/static-max in xenstore, if at all?

That is a can of worms.  They have grown teeth and are biting XenServer
as well. :(

Accurate accounting of memory is basically impossible at the moment,
because it is an $N variable equation (for $N = 14 IIRC last time it was
counted) with $M entities knowing some but not all of the information.

There is active work underway to try and quantify and remedy the
situation.  As to your specific question though...

>
> When a domain is running, libvirt reports maximum memory through
> max_memkb field of libxl_dominfo struct. max_memkb is filled with
> PAGE_TO_MEMKB(xcinfo->max_pages). AFAICT, xcinfo->max_pages contains
> the same value as max_pages from 'q' debug key, but neither of those
> represent the maxmem value used at domain creation. E.g.
>
> # cat test.xl | grep mem
> maxmem = 8192
> memory = 2048
> # xl create test.xl
> # xl debug-key q && xl dmesg | grep max_pages
> (XEN)     nr_pages=524205 xenheap_pages=6 shared_pages=0 paged_pages=0
> dirty_cpus={20,42} max_pages=524544
>
> On this system, 524544*4096=2049M which is closer to 'memory' than
> 'maxmem'.
>
> I have a patch which fixes this in libvirt, but I'm having a hard time
> articulating why the current use of max_pages is incorrect :-).

nr_pages is (to an approximation) the number of pages currently making
up guest RAM.  It becomes complicated/fuzzy with PoD, VRAM, option roms,
ballooned out pages, etc.  (Part of the problem for XenServer is that it
doesn't relate to "the amount RAM is Xen using to look after this
domain"; there is absolutely no quantification of the overhead of
running a VM.)

max_pages is a value set directly by the toolstack
(XEN_DOMCTL_set_maxmem iirc).  It is an upper limit, which the guest may
not exceed.  However, it may be lower than nr_pages, at which point the
VM may release pages (i.e. balloon them out), but cant claim them back.

Part of the fuzzy problems come from the fact that multiple entities
play with set_maxmem, including qemu when trying to allocate space for
VRAM/option roms.  This particular information is never conveyed back to
libxl, which is most likely where your discrepancies originate.

Having said that, by having maxmem and memory different, you have
invoked PoD and that is a separate world of fun.  A value which isn't
printed is the size of the PoD cache, which is the number of pages
nominally allocated to a domain, which will magic themselves into
existence if the VM happens to touch a non-populated page.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel