Re: domU memory exceeded =?=> spontaneous reboots

[Elliott Mitchell <ehem+xen@xxxxxxx>]

On Fri, Dec 06, 2024 at 06:18:09AM +0000, Mike wrote:
> Elliott Mitchell wrote:
> > > The command that I
> > > ran in it that triggered the reboot was `kubectl delete -f` of a 
> > > Deployment
> > > that was already running from an `apply`.
> > 
> > Okay, do you have a full list of what this command does?
> 
> I'm sort of counting on people to know what Kubernetes does.  I know that the
> command will contact the k8s API Server on the control node, then that will
> cause some cascade of communication among the k8s services running on the
> control node, eventually leading the API Server to contact the worker node's
> Kubelet to delete the objects.

Notice how this mailing list is xen-users@xxxxxxxxxxxxxxxxxxxx?  It is
reasonable to think people here might have surface familiarity with
Kubernetes, but you're going to need to fill in many pieces for us.

> > Might it cause a crucial Xen domain to panic (domain 0) and this in turn
> > cause Xen to panic?
> 
> The only way that I could see the dom0 involved is maybe via the network
> stack?  The communication between the domUs is going through the xenbr0, of
> course.

Not knowing any of the details of how you've got things setup, do you
have any pieces of Kubernetes in domain 0?  These exhausting memory
would cause problems.

> > How much free memory does Xen have?
> 
> 265.  It was sthg like 313 (= 265 + 2048 - 2000) before I moved more memory
> to the control node.

Did you in fact look at the output of `xl info` (this needs to be run in
domain 0)?  Are you instead guessing based on what you think your
computer's setup is?

If you're not looking at the output of `xl info` (or
`xl info | grep -e^free_memory`), then you're guessing and you could be
wrong.  Xen needs some memory to keep track of memory/addresses which are
assigned to particular domains.  Mapping pages between domains (for
virtual network/block devices) does require Xen to use some memory.

The estimate from Qubes OS is to start worrying if Xen's free memory
drops below 50MB.  If Xen's free memory is <20MB then the "network"
activity causing exhaustion of Xen's memory seems plausible.  If below
10MB then this seems likely.  If listed as "0" then this is near-certain
as the immediate cause.

> > Might be 0 if Xen is ballooning memory from domain 0 to handle
> > allocations.  If ballooning memory from domain 0 has been disabled this
> > should stay above 50 so Xen can allocate memory to handle activity.
> 
> No ballooning:
> dom0_mem=4G;max:4G dom0_max_vcpus=4 dom0_vcpus_pin xpti=dom0=false,domu=true 
> no-real-mode edd=off

Okay, but 4GB for domain 0 strikes me as an odd value.  If this computer
is primarily a server then 1GB should be plenty.  Needing 4GB would
suggest you've got way too much going on for a hypervisor control VM.  Is
this computer primarily your desktop and you're using Xen to generate VMs
to experiment with Kubernetes?

If this computer is primarily your desktop then 4GB of memory and 4 vCPUs
is rather low.  Many Linux desktop environments will really want 8GB and
can readily benefit from 8-12 processors (lighter weight desktop
environments don't need that much).

> > > The domUs are a k8s control and worker node,
> > 
> > Is either of these also domain 0?  Domain 0 exhausting its free memory
> > and panicing might cause the issue you're describing.
> 
> No, they are domUs.

I'm starting to suspect this computer has a total of 8GB of memory?  All
desktop processors support dual-channel memory.  If you've only got 8GB
then you're losing performance due to limited memory bandwidth.  This
though is assuming a desktop computer.

The other trick is you *will* find you want more memory.  Rule of thumb
is the moment you have a VM machine, it will start eating your network.
You will start finding every task on your network looks like it can be
turned into a VM.  There is a reason server systems have been claiming
support for 2TB of memory/socket for a while.

My personal VM server is enveloping my router (just needed 802.11 plus
dual ethernet interfaces).  I'm a bit scared since it looks hungry and
seems to be eyeing my desktop.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445