Re: [Xen-devel] Possible memory leak in qemu-dm (qemu-dm swapping 20GB+, adding 2gb+ per day)

[Niklas Bivald <niklas@xxxxxxxxxx>]

Hi,

Summary: Me and ilon@xxxxxxxxxx has independently confirmed that the patch solves the memory leak and is running the patched binary live. Big thanks to everyone who helped - specially Matthew who "patched the patch" to work with xen.

With the help of Matthew I've successfully compiled the patch on xen 4.1.4 (git checkout tags/RELEASE-4.1.4 and make dist-tools) and me and ilon@xxxxxxxxxx has confirmed independently that the patch does solve the memory leak in qemu-dm. To make sure we've changed nothing except the patch, we also compiled from source without the patch to confirm the memory leak was actually there before (which is was)

The final patch (again, thanks to Matthew) is available on https://gist.github.com/bivald/7691087

To use the patched version of qemu, we've:

1. Compiled binaries from source (with patch)

2. Used the qemu-dm binary (symlinked to /usr/lib/xen-4.1/bin/qemu-dm since 4.1.4 doesn't support device_model_override)

3. Used the compiled binaries we were missing on dom0: libblktap.so.3.0, libxenctrl.so.4.0, libxenguest.so.4.0 

We're currently running the patched binaries since roughly a week ago and so far they are completely stable. 

Things that might means something to someone else then me:

- We needed libblktap.so.3.0, libxenctrl.so.4.0, libxenguest.so.4.0 for some reason, instead of the 4.1 versions (libxenguest-4.1.so, libxenctrl-4.1.so, xen-4.1/lib/libblktapctl.so)

- Memory usage of qemu-dm process for a freshly started VPS is slightly higher then the initial memory usage of the leaking binary but doesn't leak

Again, thanks to everyone involved for the help.

In case someone else finds this in the archives and need the patched binary (or if I need to remember how to patch it in the future), I've uploaded the binaries and how they were compiled to https://github.com/bivald/xen-tools-4.1.4-patched-qemu

Regards,

Niklas

On 27 Nov 2013, at 12:06, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:

On Mon, 2013-11-25 at 13:59 +0100, Niklas Bivald wrote:

Assuming my Debian 7.0 with xen 4.1.4 uses qemu-xen-traditional then
yes. Otherwise I’ve observed it in default xen qemu for debian 7.
Currently all mine (and ilon@xxxxxxxxxx’s) qemu-dm instances keeps
growing, adding several GB in swap per day. Then dom0 runs out of swap
and the qemu-dm segfaults and I have to xl destroy it. Then I start
the domain again and qemu-dm starts growing in swap.

Although I've not run it on qemu you might be able to use valgrind's
support for Xen to help debug this. See:
http://blog.xen.org/index.php/2013/01/18/using-valgrind-to-debug-xen-toolstacks/

It works for debugging xl create and similar, but I didn't try qemu, if
it complains about hypercalls it doesn't understand please let me know
and I'll see about implementing them.

Ian.

On 25 nov 2013, at 13:40, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> wrote:

Niklas Bivald writes ("Re: [Xen-devel] Possible memory leak in qemu-dm (qemu-dm swapping 20GB+, adding 2gb+ per day)"):

Can I help in the process of confirming this bug in qemu-xen-traditional? 

Do you mean to say that you have observed qemu-xen-traditional's
qemu-dm growing, as described in the bug report ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel