[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Limiting domU network?



Fajar A. Nugraha wrote:

 > You could employ traffic shaping on the network, that's what we do at work.
 > It's quite easy to nest HTB classes


So your boundary router is Linux, with tc?
Can you share your config / HTB init scripts?
Last time I tried it was ... confusing (to say it politely) and
testing with iperf does not show the desired traffic limit. Then again
it might be because of iperf was short and bursty.

It took me a while to get my head around it all, and it didn't all appear in one go either !

This is the business end of what we have - anonimised for obvious reasons. Like many things, it's not the only way to do it, and there are probably other ways as well, but "it works for us".

In terms of resources required, this runs on a 1GHz Pentium III, and memory requirements aren't high :
cat /proc/meminfo
MemTotal:      1036092 kB
MemFree:        499152 kB
Buffers:        399816 kB
Cached:         101296 kB
I'll correct the statement I made earlier - it generally runs between 97% and 99% idle, it will use more cpu when it's updating the RRDs, and when the frontend is drawing graphs. I share the directory containing the rrd files via NFS.
A typical output from top is :
top - 10:54:31 up 44 days,  8:17,  1 user,  load average: 0.00, 0.01, 0.00
Tasks:  65 total,   2 running,  63 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   1036092k total,   537172k used,   498920k free,   399816k buffers
Swap:  2939884k total,        0k used,  2939884k free,   101296k cached



This is the traffic accounting stuff. We have a full class C at work, so I log traffic for all 254 addresses. Before I put this in place, we had problem with running out of bandwidth, but we had no idea how badly, which servers were consuming it, or whether we could mitigate the problem. As it happens, we were able to mitigate the problem (and make VoIP usable again) by application of traffic shaping, and we put off upgrading the bandwidth for about a year. Since then, it's allowed us to spot "problems", and along with Nagios to monitor stuff, it's often allowed us to fix things before the customers phone us up to say it's broken :-)

The data is stuffed into an RRD database, and from there it is graphed with some custom scripts. Since anyone in the office can call up the graphs, it means a lot of troubleshooting can be done without me getting involved.


accounting :

outside-in:COUNT        -       ethext  -
outside-out:COUNT       -       -       ethext
DONE    outside

# Do acocunting by IP address
account-ip      -       -       -
total-ip-in:COUNT       account-ip      ethext  -
total-ip-out:COUNT      account-ip      -       ethext
DONE total-ip

acc-serv
total-serv-in:COUNT     acc-serv        ethext  -
total-serv-out:COUNT    acc-serv        -       ethext
DONE    total-serv


INCLUDE accounting.ip
INCLUDE accounting.service



accounting.ip :

acc1-in:COUNT   account-ip      ethext  a.b.c.1
acc1-out:COUNT  account-ip      a.b.c.1 ethext
DONE    acc1

acc2-in:COUNT   account-ip      ethext  a.b.c.2
acc2-out:COUNT  account-ip      a.b.c.2 ethext
DONE    acc2

acc3-in:COUNT   account-ip      ethext  a.b.c.3
acc3-out:COUNT  account-ip      a.b.c.3 ethext
DONE    acc3

...

acc252-in:COUNT account-ip      ethext  a.b.c.252
acc252-out:COUNT        account-ip      a.b.c.252       ethext
DONE    acc252

acc253-in:COUNT account-ip      ethext  a.b.c.253
acc253-out:COUNT        account-ip      a.b.c.253       ethext
DONE    acc253

acc254-in:COUNT account-ip      ethext  a.b.c.254
acc254-out:COUNT        account-ip      a.b.c.254       ethext
DONE    acc254


accounting.service is currently empty, it's there to support future plans to add accounting by IP and service - eg so we can see the split between http, https, smtp, ftp etc for an IP address.


Getting the data out is done with a shell script run every few minutes from cron :
#/bin/bash
# Script to extract values from shorewall output

cd /var/rrd

/usr/bin/rrdtool update ip-stats.rrd N:`/sbin/iptables -L account-ip -vxn | \
  /usr/bin/awk 'BEGIN   { getline ; getline }
        { print $2 }' | \
  /usr/bin/tr '
' ':' | /bin/sed -e 's/:$//'`


And the rrd definition for this is :
rrdtool create ip-stats.rrd -s 300 \
  DS:total-in:DERIVE:600:0:U \
  DS:total-out:DERIVE:600:0:U \
  \
  DS:ip1-in:DERIVE:600:0:U \
  DS:ip1-out:DERIVE:600:0:U \
  DS:ip2-in:DERIVE:600:0:U \
  DS:ip2-out:DERIVE:600:0:U \
  DS:ip3-in:DERIVE:600:0:U \
  DS:ip3-out:DERIVE:600:0:U \
...
  DS:ip253-in:DERIVE:600:0:U \
  DS:ip253-out:DERIVE:600:0:U \
  DS:ip254-in:DERIVE:600:0:U \
  DS:ip254-out:DERIVE:600:0:U \
  \
  RRA:AVERAGE:0.5:1:576 \
  RRA:MAX:0.5:1:576 \
  RRA:AVERAGE:0.5:6:672 \
  RRA:MAX:0.5:6:672 \
  RRA:AVERAGE:0.5:24:732 \
  RRA:MAX:0.5:24:732 \
  RRA:AVERAGE:0.5:288:730 \
  RRA:MAX:0.5:288:730

# CFs for :
#   1 x 576    48hrx 5m
#   6 x 672    14d x 1/2hr
#  24 x 732    61d x 2hr
# 288 x 730   730d x 24hr


I'll do the traffic shaping in a separate email.

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.