Re: [Xen-users] Limiting domU network?

[Simon Hobson <linux@xxxxxxxxxxxxxxxx>]

Fajar A. Nugraha wrote:

>  > You could employ traffic shaping on the network, that's what we do at work.
>  > It's quite easy to nest HTB classes

> So your boundary router is Linux, with tc?
> Can you share your config / HTB init scripts?
> Last time I tried it was ... confusing (to say it politely) and
> testing with iperf does not show the desired traffic limit. Then again
> it might be because of iperf was short and bursty.
It took me a while to get my head around it all, and it didn't all 
appear in one go either !
This is the business end of what we have - anonimised for obvious 
reasons. Like many things, it's not the only way to do it, and there 
are probably other ways as well, but "it works for us".
In terms of resources required, this runs on a 1GHz Pentium III, and 
memory requirements aren't high :
> cat /proc/meminfo
> MemTotal:      1036092 kB
> MemFree:        499152 kB
> Buffers:        399816 kB
> Cached:         101296 kB
I'll correct the statement I made earlier - it generally runs between 
97% and 99% idle, it will use more cpu when it's updating the RRDs, 
and when the frontend is drawing graphs. I share the directory 
containing the rrd files via NFS.
A typical output from top is :
> top - 10:54:31 up 44 days,  8:17,  1 user,  load average: 0.00, 0.01, 0.00
> Tasks:  65 total,   2 running,  63 sleeping,   0 stopped,   0 zombie
> Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
> Mem:   1036092k total,   537172k used,   498920k free,   399816k buffers
> Swap:  2939884k total,        0k used,  2939884k free,   101296k cached


This is the traffic accounting stuff. We have a full class C at work, 
so I log traffic for all 254 addresses. Before I put this in place, 
we had problem with running out of bandwidth, but we had no idea how 
badly, which servers were consuming it, or whether we could mitigate 
the problem. As it happens, we were able to mitigate the problem (and 
make VoIP usable again) by application of traffic shaping, and we put 
off upgrading the bandwidth for about a year. Since then, it's 
allowed us to spot "problems", and along with Nagios to monitor 
stuff, it's often allowed us to fix things before the customers phone 
us up to say it's broken :-)
The data is stuffed into an RRD database, and from there it is 
graphed with some custom scripts. Since anyone in the office can call 
up the graphs, it means a lot of troubleshooting can be done without 
me getting involved.

accounting :

> outside-in:COUNT        -       ethext  -
> outside-out:COUNT       -       -       ethext
> DONE    outside
>
> # Do acocunting by IP address
> account-ip      -       -       -
> total-ip-in:COUNT       account-ip      ethext  -
> total-ip-out:COUNT      account-ip      -       ethext
> DONE total-ip
>
> acc-serv
> total-serv-in:COUNT     acc-serv        ethext  -
> total-serv-out:COUNT    acc-serv        -       ethext
> DONE    total-serv
>
>
> INCLUDE accounting.ip
> INCLUDE accounting.service


accounting.ip :

> acc1-in:COUNT   account-ip      ethext  a.b.c.1
> acc1-out:COUNT  account-ip      a.b.c.1 ethext
> DONE    acc1
>
> acc2-in:COUNT   account-ip      ethext  a.b.c.2
> acc2-out:COUNT  account-ip      a.b.c.2 ethext
> DONE    acc2
>
> acc3-in:COUNT   account-ip      ethext  a.b.c.3
> acc3-out:COUNT  account-ip      a.b.c.3 ethext
> DONE    acc3
>
> ...
>
> acc252-in:COUNT account-ip      ethext  a.b.c.252
> acc252-out:COUNT        account-ip      a.b.c.252       ethext
> DONE    acc252
>
> acc253-in:COUNT account-ip      ethext  a.b.c.253
> acc253-out:COUNT        account-ip      a.b.c.253       ethext
> DONE    acc253
>
> acc254-in:COUNT account-ip      ethext  a.b.c.254
> acc254-out:COUNT        account-ip      a.b.c.254       ethext
> DONE    acc254

accounting.service is currently empty, it's there to support future 
plans to add accounting by IP and service - eg so we can see the 
split between http, https, smtp, ftp etc for an IP address.

Getting the data out is done with a shell script run every few 
minutes from cron :
> #/bin/bash
> # Script to extract values from shorewall output
>
> cd /var/rrd
>
> /usr/bin/rrdtool update ip-stats.rrd N:`/sbin/iptables -L account-ip -vxn | \
>   /usr/bin/awk 'BEGIN   { getline ; getline }
>         { print $2 }' | \
>   /usr/bin/tr '
> ' ':' | /bin/sed -e 's/:$//'`

And the rrd definition for this is :
> rrdtool create ip-stats.rrd -s 300 \
>   DS:total-in:DERIVE:600:0:U \
>   DS:total-out:DERIVE:600:0:U \
>   \
>   DS:ip1-in:DERIVE:600:0:U \
>   DS:ip1-out:DERIVE:600:0:U \
>   DS:ip2-in:DERIVE:600:0:U \
>   DS:ip2-out:DERIVE:600:0:U \
>   DS:ip3-in:DERIVE:600:0:U \
>   DS:ip3-out:DERIVE:600:0:U \
> ...
>   DS:ip253-in:DERIVE:600:0:U \
>   DS:ip253-out:DERIVE:600:0:U \
>   DS:ip254-in:DERIVE:600:0:U \
>   DS:ip254-out:DERIVE:600:0:U \
>   \
>   RRA:AVERAGE:0.5:1:576 \
>   RRA:MAX:0.5:1:576 \
>   RRA:AVERAGE:0.5:6:672 \
>   RRA:MAX:0.5:6:672 \
>   RRA:AVERAGE:0.5:24:732 \
>   RRA:MAX:0.5:24:732 \
>   RRA:AVERAGE:0.5:288:730 \
>   RRA:MAX:0.5:288:730
>
> # CFs for :
> #   1 x 576    48hrx 5m
> #   6 x 672    14d x 1/2hr
> #  24 x 732    61d x 2hr
> # 288 x 730   730d x 24hr

I'll do the traffic shaping in a separate email.

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users