Re: [Xen-users] 'xl reboot domid' ignores all in-memory changes AND the domu's config file changes

["Austin S. Hemmelgarn" <ahferroin7@xxxxxxxxx>]

On 2016-09-21 12:04, msd+xen-users@xxxxxx wrote:
> Hello !
>
> Thank you for your reply !
>
> The reboot 'bug'
> ----------------
>
> So, what I have done :
> - installing a Debian 8 jessie with xen 4.4.1
> - creating a domu1 with 2048 Mb of RAM with 'xl create /etc/xen/domu1.cfg'
> - 'xl li' says that domu1 has 2048 Mb of RAM
> - use the command 'xl mem-set domu1 1024'
> - 'xl li' says that domu1 has 1024 Mb of RAM
> - use the command 'xl reboot domu1'
> - waiting for the reboot to complete
> - 'xl li' says that domu1 has ***2048 Mb*** of RAM # and this is not
> what I expect
>
> Then :
> - Upgrade my Debian from jessie to stretch with xen 4.6.0.
> - 'xl li' says that domu1 has 2048 Mb of RAM
> - use the command 'xl mem-set domu1 1024'
> - 'xl li' says that domu1 has 1024 Mb of RAM
> - use the command 'xl reboot domu1'
> - waiting for the reboot to complete
> - 'xl li' says that domu1 has ***1024 Mb*** of RAM # and this is what I
> expect
>
> So I really think that it is a bug in xen 4.4.1.
>
> No ?
>
> xl config-update
> ----------------
>
> Just for information.
>
> With Debian stretch, I have tried the command 'xl config-update domu
> /etc/xen/domu1.cfg'. No segfault this time, but :
>
> > WARNING: xl now has better capability to manage domain configuration,
> avoid using this command when possible
> > setting dom3 configuration
> And the config '/etc/xen/domu1.cfg' file was not changed !
>
> self-ballooning
> ---------------
>
> Austin S. Hemmelgarn <ahferroin7@xxxxxxxxx> wrote :
> > and I have all my PV domains set up for self-ballooning, so 90% of the
> time they're using significantly less memory than they're assigned
>
> How self-ballooning works ? How do you configure your Dom-0 and your
> Dom-u ?
The general concept is pretty simple.  In essence, when a domain which 
has it enabled (and is running a kernel that supports it) sees that it 
has a lot of free memory, it starts to inflate the balloon device itself 
until it hits some threshold (based on a bunch of factors, I've not even 
figured out exactly how it decides).  When the amount of free memory 
goes below that threshold for long enough, the balloon device tries to 
claim more memory up to the configured limit from Xen.
To use a slightly more concrete example:  I run a dedicated PV domain on 
my home server that provides DHCP for the other VM's and a couple of 
other systems.  I have this VM set to use at most 1024 MB or memory. 
When it boots, it starts out with that amount of memory, but once the 
balloon driver initializes, it sees that most of that is free, and 
starts slowly reclaiming memory to Xen.  This process continues until 
things stabilize at around 350MB of RAM, which constitutes the normal 
working set for everything running in this domain (including the page 
cache and network buffers) plus a small amount of extra (about 120MB 
usually in this domain).  When I go to install updates on this domain, 
the package manager needs more memory, and as it starts allocating it, 
the balloon driver sees the free memory decreasing, and starts deflating 
the balloon until it satisfies whatever threshold it's looking at.  Once 
the updates are done, the excess free memory triggers the balloon driver 
shrinking the allocation again, and things stabilize again around 370MB.
There are a number of things needed in both Xen and the user domain to 
support this:
1. Xen has to have support for transcendent memory.  I don't remember 
when this first got merged, but I've not tested it on any version of Xen 
before 4.6.
2. Xen needs to be booted with the 'tmem' option.  You just need to 
specify this option, not any others starting with it (self-ballooning 
doesn't benefit much from deduplication or compression being enabled).
3. The user domain has to have a PV balloon driver which supports this. 
As of right now, the only guest OS that supports it as far as I know is 
Linux, and the kernel needs to be built with at least the following 
config options:
CLEANCACHE=y
FRONTSWAP=y
XEN_BALLOON=y
XEN_SELFBALLOONING=y
4. The user domain has to be running with PV drivers.  I've never tested 
this in anything but pure PV mode, but it should work in PVH mode, and 
it might work with PV drivers on HVM.
5. For Linux, the user domain must be configured with a reasonable 
amount of swap space.  I don't know exactly how much is needed, but my 
testing indicates that having at least as much swap as the max amount of 
RAM for the domain meets this criteria.
6. For Linux, you need to boot with the following command-line options:
tmem=1 tmem,selfshrink=1 tmem.selfballooning=1 frontswap=selfshrinking

Assuming that all the criteria are met, you should see a directory 
called 'selfballooning' under /sys/devices/system/xen_memory/xen_memory0
That directory shows that you at least have support for it, and you can 
check the 'selfballooning' and 'frontswap_selfshrinking' files in that 
directory to see if it's enabled (they'll both contain '1' if it is). 
The other files there control various parameters of the driver.
There are a couple of caveats to using this though:
1. Starting a domain pre-shrunk (mem<maxmem in the config) doesn't seem 
to work with selfballooning (the domain just starts with maxmem 
allocated).  Because of this, if you're over-committing memory (which is 
possible, and can be safe if you are careful), you'll have to sequence 
startup of the VM's so that they don't all start at the same time.  In 
my case, I do this with a script that starts domains one by one waiting 
to be able to SSH into each one before starting the next.
2. If you're doing this to overcommit memory (like I am), you need to 
have a very good understanding of your workload's memory requirements 
and how everything behaves, because trying to claim more memory from Xen 
than exists will cause things to break (sometimes domains will crash, 
sometimes you'll just get an OOM condition in a domain, sometimes they 
just start hitting swap really hard).
3. Some applications behave differently based on how much RAM they see 
in the system.  BOINC applications are a classic example of this (they 
use memory based on the maximum amount in the system), but there are 
others, so doing this can impact how the system behaves.
4. Some applications try to allocate huge amounts of memory very 
quickly.  The general symptom of this when using self-shrinking is that 
you start seeing random out-of-memory conditions when trying to use a 
specific program.  These applications can be made to work, but you'll 
have to bump up the minimum free RAM.
5. Depending on your workload, this can be a performance hit.  If you 
are using something that regularly allocates and frees lots of memory, 
it will almost certainly perform worse with self-shrinking enabled.  In 
contrast, something like a DHCP server (which usually allocates 
everything it needs at startup) will see near zero performance hit from 
using this.
> Austin S. Hemmelgarn <ahferroin7@xxxxxxxxx> wrote :
> > (like a nearly complete lack of documentation)
> Do you know where I can find some documentation about it ?
Pretty much everything I know about it is stuff I've figured out from 
reading source code.  What I've got above is as close to a complete 
description as I've ever seen, maybe it should go on the Wiki?
> shutdown + create
> -----------------
>
> Simon Hobson <linux@xxxxxxxxxxxxxxxx> wrote :
> > Is to shutdown the machine and create it. So instead of "xl reboot"
> which simply tells the guest OS to restart USING THE SAME VIRTUAL
> MACHINE, use xl shutdown followed by xl create which will create a new
> virtual machine with the current settings in the config file.
>
> Thank you !
>
> But in fact my problem is that I don't want as a Dom-0 admin choose when
> the Dom-U will reboot. I don't want to reboot the Dom-U because there is
> services running on, I want that when the admin of the Dom-U wants (in
> the night, on sunday !), he reboots the Dom-U (with the 'reboot' command
> inside the Dom-U). And when the Dom-U admin do it, I just want the Dom-U
> to keep the good amount of RAM ! :)
>
> But, for now, I will just do as you advise me. Reboot the Dom-U from the
> Dom-0 when I change its amount of RAM.
For what it's worth, it is technically possible to set things up so that 
a reboot command in the user domain will trigger a full shutdown+create 
in domain 0, but it's not easy to do, and it's very tricky to do securely.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users