[Xen-bugs] [Bug 762] xen_net: Memory squeeze in netback driver.

[bugzilla-daemon@xxxxxxxxxxxxxxxxxxx]

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=762


sebastian.malcolm@xxxxxxxxx changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sebastian.malcolm@xxxxxxxxx
           Severity|normal                      |major




------- Comment #3 from sebastian.malcolm@xxxxxxxxx  2006-11-08 01:36 -------
I'm using Debian packages of Xen (2.6.17-2 kernel) installed from the Etch
(testing) distribution on a Dell SC430 (dual core PentiumD with 2Gig Ram).
I have a minimal installation of Debian Etch (amd64) running as host Dom0.

To provide isolated networks for my DomU's across one dummy ethernet interface
and two physical ethernet cards (one addin 100mb/s card for Internet & the
onboard 1000mb/s Intel for private LAN), I configured Xend startup to create 3
Bridges using brctl -> 1 bridge for each interface (xenbrInet = ethInet,
xenbrDMZ = ethDMZ, xenbrPriv = ethPriv). I have then been able to startup
several DomUs with vif's in 1,2 or all 3 of these bridges.

I only see this error message after I add "one too many" DomUs (>3?) but I've
not done enough testing yet to be certain what quantity or combination of the
following quantities trigger this error message:
 * The total number of DomUs
 * The amount of Memory allocate per DomU
 (Eg: ~4x 256mb out of 2gig physical triggered the error, but does 8x 128mb?)
 * The total number (virtual) network interfaces
 * The number of virtual network interfaces per-domain

I have been able to reproduce this error message using either Debian package
"xen-hypervisor-3.0-unstable-1-amd64" or the hopefully stable+reliable package
version "xen-hypervisor-3.0.2-1-amd64" with the recommended 2.6.16 or 2.6.17
Xen kernel packages. The next scenario I think I must test is trying to
reproduce my current Xen Dom0 configuration with 4 or more DomU's  using the
"Demo CD Image" that I have downloaded from XenSource.com. If all the current
stable release binaries available are unable to run >3 DomU's on my hardware
without producing errors then I shall try checking out the latest source code
and compiling it to hopefully produce a working kernel and hypervisor.

Use the source browser on lxr.xensource.com to read thru the Xen source code to
see that the error is probably printed from within
"/linux-2.6-xen-sparse/drivers/xen/netback/netback.c".

The code from xen-3.0.3_0-src.tgz in netback.c:net_rx_action() is:

        if (!xen_feature(XENFEAT_auto_translated_physmap) &&
            check_mfn(nr_frags + 1)) {
            /* Memory squeeze? Back off for an arbitrary while. */
            if ( net_ratelimit() )
                WPRINTK("Memory squeeze in netback "
                    "driver.\n");
            mod_timer(&net_timer, jiffies + HZ);
            skb_queue_head(&rx_queue, skb);
            break;
        }

Researching this error message using Google and searching the mailing lists
suggests to me that this error message has been a problem for a long time
(perhaps since the 3.0 release?).

The must-read thread on xen-devel hinting at a solution to this error message
is dated "6 Jun 2006" from Anthony Liguori:
http://lists.xensource.com/archives/cgi-bin/mesg.cgi?a=xen-devel&i=4485F70E.4020602%40us.ibm.com
"Do this mean that the netback driver needs to be able to increase it's
reservation?"
Keir Fraser writes in reply: "The kernel should be able to do it for itself."

Relevant postings to the xen-users mailing list reporting the same or similar
problem are:

http://lists.xensource.com/archives/html/xen-users/2006-08/msg00947.html
30 Aug 2006: "Memory squeeze in netback driver" "...doesn't appear to affect
performance until you add one too many guest domains, then the whole lot drop
off the net. It's not hardware related as I can recreate the issue on 3
completely separate servers with dissimilar hardware."
[Increasing the quantity of DomUs certainly seems to be a common theme in all
reports of this bug. Does it occur only on 64bit (amd64 and em64t) systems
running 64bit enabled Kernels?]

http://lists.xensource.com/archives/html/xen-users/2005-11/msg00534.html
22 Nov 2005: "...Xen 3.0 release using the Fedora Core 4 Wiki instructions.."
[Distro used probably isn't a factor.]

http://lists.xensource.com/archives/html/xen-users/2006-02/msg00077.html
2 Feb 2006: "...if dom0_mem set to 196MB then I can start 20 domains using all 
available ram without any problems..."
"Before I rebooted with dom0_mem set to 196MB...
...I also saw:  "xen_net: Memory squeeze in netback driver." in dom0..."
"I have a solution now (using dom0_mem and 3.0.1)..."
[So it seems that the amount of RAM allocated to Dom0 is a factor?]

http://lists.xensource.com/archives/html/xen-users/2006-06/msg00524.html
13 Jun 2006: "> When I start more than about 3 domU's..."
"I'm seeing this error as soon I as start a second domU."
Follow up in the same thread, 14 Jun 2006:
"I had forgotten to copy the modules for the xen linux kernel from dom0
to the new domU. So, I did that, and lowered the memory allocated to
the first domU. The 2nd domU booted without any problems."
And again, 15 Jun 2006:
"...resolved the problem by lowering the memory allocated to the first domU.
After I did that, the 2nd domU started up and the "memory squeeze..." error
stopped."

http://www.linode.com/xen/irc/logs/xen.log-2006-08-15
Possibly relevant info about how setting "maxmem value < memory" might be
relevant to this error message: 
"17:21 <CosmicRay> rharper: oh, so it is a bug in xen 3.0.2 then?"
 17:22 <rharper> yes, in 3.0.2 they ignored the maxmem value"

------------------
In summary:

Nothing I've read indicates that downloading the latest version of Xen will
contain any changes that prevent the "Memory squeeze in netback driver" error
message from being printk'ed endlessly until the number of DomUs is reduced.
The thread on xen-devel started by Anthony Liguori in response to the detailed
error report by Erik Hensema is the best description I've seen of a workaround
that is stable (but might require Xen >=3.0.3 if the maxmem parameter is
ignored on all 3.0.2.X versions). I too shall try new configuration options to
see if I am able to run >4 DomUs without this error message infinitely repeated
in my system logs. I'll report back here with another comment on my success or
failure.


-- 
Configure bugmail: 
http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs