Re: [Xen-users] query memory allocation per NUMA node

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Tue, 2017-01-10 at 01:50 +0000, Kun Cheng wrote:
> On Mon, Jan 9, 2017 at 10:47 PM Eike Waldt <waldt@xxxxxxxxxxxxx>
> wrote:
> > Xen has to have a mechanism to get to know which NUMA-Node is
> > most-empty/preferred then.
> > I even read about different "NUMA placement policies" in [1], but
> > didn't
> > find a way to set them.
> 
> The placement seems to be automatic (perhaps in a greedy way) .
>
It is automatic, but it's not greedy. It's actually "pretty exact", as
it generates and analyzes the various possible combinations of
placements.

> > A command line parameter for "xl" is what I'm looking here for.
> > A handy alternative to "xl debug-keys u; xl dmesg"...
> 
> Then as I said, perhaps no cmdline tool could do the trick.
> 
Not right now, no.

> > Why should it be harder to count then? "xl debug-keys u; xm dmesg"
> > does
> > already give me this information (but you cannot really parse this
> > or
> > execute this periodically).
> 
> Becuase it is changing rapidly in some situations (high load) due to
> load balancing then hypervisor may have to migrate some VCPUs (then
> memory) to another node. 
>
We do migrate vcpus for balancing load. We never migrate memory.

> About 14 months ago I asked Wei Liu and  Dario about the possiblility
> of optimization NUMA support (mainly VCPU & memory load-balancing &
> migration) and at that time they said it was not in their coming
> plan. 
>
Well, as briefly said already, it'd definitely be super-nice to have.
Unfortunately, moving the memory is not easy.

> So I tried to it myself for fun but the first problem I encountered
> was counting how much memory consumed on each node for each VM, which
> a more accurate NUMA scheduling & load-balancing could depend on such
> info. 
>
Yes, I remember talking with you about this, but I do not remember that
the showstopper was knowing how many pages of a certain domain are
allocated on a certain NUMA node. That is quite straightforward to tell
and, at present, never changes after domain creation!

If, OTOH, you mean how frequently a certain page is _accessed_ from a
CPU of a certain node, that's indeed a different story. But it's not
what is being asked here.

> However, as in a high load situation (that's where you could use a
> good laod balancing), memory usage is changing rapidly, at some time
> you may find a more suitable node with enough memory space but after
> probing is finished you may find it no longer a candidate due to 1)
> some VMs have been already migrated to that node 2) memory ballooning
> of existing VMs on that node.  
> 
In a typical virtualization host, while scheduling can indeed be quite
dynamic, memory usage should not move that fast.

In fact, one of the challenges of implementing the kind of load
balancing you're hinting at, is reconciling these two velocities by
setting proper thresholds...

> > But to do hard-pinning the correct way I need to know on which
> > NUMA-nodes the DomU runs...Otherwise performance will be impacted
> > again.
> > 
> > As I cannot change on which NUMA-node the DomU is started (unless I
> > specify pCPUs to the DomU's config [which would require something
> > "intelligent" to figure out which Node/CPUs to know]), I have to do
> > it
> > this way around, or am I getting it totally wrong?
> 
> In Xen the manual NUMA node preference is done through VCPU pinning,
> no matter how you do it, either soft or hard pinning, or even with a
> cpu pool. 
> 
Yep, this is correct.

> If all your VMs are already pinned to some certain nodes, maybe you
> can parse the configuration files or xenstore to retrive existing
> placement info (Yes it certainly involves some programming work) and
> find the best place for the next VM?  
>
Indeed, except there's no need to parse confg file or look in xenstore.
All it takes is:

xl list -n

or

xl vcpu-list

(and parse the output of these, if needing automating things).

> > As far as I understand numactl/numastat will not work in Dom0.
> 
> I remembered, as  Xen runs under Domain-0, numactl/numastat  will not
> work. But xl info can return some numa topology info, have you tried
> to get more outputs in the latest version?
> 
xl info -n

> Or maybe libvirt could do some help?
> 
libvirt can tell the same than `xl info -n', yes.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users