Re: [Xen-users] Stability of XEN under high load

["Dirk H. Schulz" <dirk.schulz@xxxxxxxxxxxxx>]

Thomas,

thanks for your input, but we are sure we do not have an arp problem.

The phenomenon is that suddenly under heavy load a vif stops working - no 
connection to others on the bridge is possible any more then, it does not 
send out anything (not even arp requests are sent, then). Since this is 
happening under heavy load the mac of this vif is well listed in all 
concerned arp caches of others on the bridge.
In the mean time we found out that we can trigger this phenomenon with 
certain tests that create massive network I/O. Is there any known bugs with 
xen 3.0.3 on ia64 concerning network load?
Dirk

--On 1. Dezember 2008 19:41:02 +0800 Thomas Goirand <thomas@xxxxxxxxxx> 
wrote:
> > On Sun, Nov 30, 2008 at 12:13 AM, Bernd Gotschy
> > <Bernd.Gotschy@xxxxxxxxxxxxxx <mailto:Bernd.Gotschy@xxxxxxxxxxxxxx>>
> > wrote: Major problem is  that under high load we loose some of the
> >     network interfaces and cannot get them working unless we reboot the
> >     guest.
> We have find out quite the opposite way. If one of the VM has no
> activity, the ARP vs IP cache of most switch will discard the MAC of the
> VMs. So we have made a very simple python script that we deploy on all
> of our Xen servers, that simply does a ping of all VMs it finds in the
> /etc/xen/auto folder. This way, the ARP cache gets refreshed, and
> everything is working fine. Maybe this is what you are experiencing, no?
> Anyway, bellow is the script, I hope that helps (take care, the procs =
> line might appear on 2 lines as I didn't do an attachment).
>
> Thomas
>
> # !/usr/bin/env python
>
> import glob
> import re
> import subprocess
> import os
> import sys
>
> pathspec = "/etc/xen/auto/*"
> regexp = re.compile(r"vif.*?=.*?ip=([0-9\. ]+)'")
>
> files = glob.glob(pathspec)
>
> contents = ( file(t).read(-1) for t in files )
>
> def ips(contents):
>         matches = regexp.findall("\n".join(list(contents)))
>         for match in matches:
>                 mips = match.split(" ")
>                 for ip in mips: yield ip
>
> devnull = file("/dev/null","w")
> procs = (
> subprocess.Popen(["ping",ip,"-c","1"],stdin=devnull,stdout=devnull,stderr
> =devnull) for ip in ips(contents) )
>
> returncodes = ( proc.wait() for proc in list(procs) )
>
> sys.exit(sum(returncodes))
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users


--------------------------------------------------------------
Dirk H. Schulz
IT Systems Service
Wiesenweg 12, 85567 Grafing
Tel. 0 80 92/86 25 68
Fax. 0 80 92/86 25 72
--------------------------------------------------------------
Technik vom Feinsten - und das nÃtige Tuning

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users