Re: [Xen-API] [XCP-1.1] High OVS cpu load and unresponsive host network while VMPR archive phase is running

[Christian Fischer <christian.fischer@xxxxxxxxxxxxxxxxxxx>]

On Thursday 02 August 2012 23:46:18 George Shuklin wrote:
> In product environment I saw that behavior few times. ovs-* processes
> starts to consume lot of cpu (over 100%) and start to cause packets drops.
> 
> That usually happens with 'hacked' customer VMs (sudden spike of
> outgoing traffic, cpu, and in few cases we assisted in research, actual
> trojans running on server because of some stupid php misconfiguration in
> yet another phpbb/cms/durpal/etc).

We have no customer VMs there, and we watch the vm traffic. Nothing unusual. 
The 
archive phase is running. It's 100% reproducible.

I suppose that, in my case, it has something to do with with the OpenFlow 
controller (Citrix DVS Controller) we tried to evaluate. Currently we do some 
tests in an testing environment to work out the problem.

But by the way, what do you do to protect your production environment against 
crashing caused by flooding the network? IIRC Jesse Gross told something about 
some work on patches preventing a single vm from being able to render the 
network unresponsive, maybe a year ago. What's the state?

> I'm not sure wat exactly happens, but my hypothesis is that it related
> to amount of flows. Then trojan starts to flood out traffic to different
> servers (smtp/www spam/etc) it cause lots of new connections..
> 
> On 01.08.2012 04:08, Ben Pfaff wrote:
> > Christian Fischer writes:
> >> On Tuesday 31 July 2012 18:08:18 Ben Pfaff wrote:
> >>> Christian Fischer
> >>> 
> >>> writes:
> >>>> We have no tagged vlans here, all physical switch ports running access
> >>>> mode. I wouldn't say that network load is increased when this happens,
> >>>> 15 kpps. Network performance could be poor due either a vswitch issue
> >>>> (runs at 180% CPU load if the vswitch log don't lie) or high load
> >>>> on/cheep hardware of the customer shared backup storage. I've never
> >>>> seen this stuff.
> >>> 
> >>> 180% CPU load is impossible for OVS 1.0.1, which has only a
> >>> single procsss with a single thread.
> >> 
> >> Yes, that's right, but we run OVS 1.4.2
> >> 
> >> XCP build: 1.1.0-50674c
> >> OVS build: 1.4.2
> >> NICs: BCM5709 Gigabit TOE iSCSI Offload
> >> OVS NIC bonding: active/active
> > 
> > Only the as-yet-unreleased post-1.8.0 Open vSwitch has more than
> > one process, and it still doesn't have multiple threads.
> > 
> > I suppose ovsdb-server and ovs-vswitchd could both go crazy at
> > the same time, but I haven't had any reports of that.
> > 
> > What process(es) add up to 180%?
> > 
> > _______________________________________________
> > Xen-api mailing list
> > Xen-api@xxxxxxxxxxxxx
> > http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
> 
> _______________________________________________
> Xen-api mailing list
> Xen-api@xxxxxxxxxxxxx
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api