[Xen-users] Terrible network performance

[Jeff Noxon <jeff@xxxxxxxxxxxxxx>]

I upgraded one of my servers today from Xen 2.0.6 to Xen 3.

With Xen 2, I experienced network dropouts from time to time, usually
once or twice a day, where all domU network activity would stop for up
to 15 minutes.

With Xen 3, I am experiencing horrible latency almost all of the time.
CPU utilization (from 'xm top') is as close to zero as it gets, yet I
get erratic ping results like this between domU and dom0 or any other
networked machine.  See a 30-second snapshot:

$ ping 192.168.1.144
PING 192.168.1.144 (192.168.1.144) 56(84) bytes of data.
64 bytes from 192.168.1.144: icmp_seq=1 ttl=64 time=19.6 ms
64 bytes from 192.168.1.144: icmp_seq=2 ttl=64 time=171 ms
64 bytes from 192.168.1.144: icmp_seq=3 ttl=64 time=322 ms
64 bytes from 192.168.1.144: icmp_seq=4 ttl=64 time=482 ms
64 bytes from 192.168.1.144: icmp_seq=5 ttl=64 time=61.9 ms
64 bytes from 192.168.1.144: icmp_seq=6 ttl=64 time=227 ms
64 bytes from 192.168.1.144: icmp_seq=7 ttl=64 time=391 ms
64 bytes from 192.168.1.144: icmp_seq=8 ttl=64 time=547 ms
64 bytes from 192.168.1.144: icmp_seq=9 ttl=64 time=118 ms
64 bytes from 192.168.1.144: icmp_seq=10 ttl=64 time=122 ms
64 bytes from 192.168.1.144: icmp_seq=11 ttl=64 time=293 ms
64 bytes from 192.168.1.144: icmp_seq=12 ttl=64 time=455 ms
64 bytes from 192.168.1.144: icmp_seq=13 ttl=64 time=49.4 ms
64 bytes from 192.168.1.144: icmp_seq=14 ttl=64 time=220 ms
64 bytes from 192.168.1.144: icmp_seq=15 ttl=64 time=567 ms
64 bytes from 192.168.1.144: icmp_seq=16 ttl=64 time=169 ms
64 bytes from 192.168.1.144: icmp_seq=17 ttl=64 time=0.305 ms
64 bytes from 192.168.1.144: icmp_seq=18 ttl=64 time=0.292 ms
64 bytes from 192.168.1.144: icmp_seq=19 ttl=64 time=0.277 ms
64 bytes from 192.168.1.144: icmp_seq=20 ttl=64 time=0.271 ms
64 bytes from 192.168.1.144: icmp_seq=21 ttl=64 time=0.276 ms
64 bytes from 192.168.1.144: icmp_seq=22 ttl=64 time=0.280 ms
64 bytes from 192.168.1.144: icmp_seq=23 ttl=64 time=0.662 ms
64 bytes from 192.168.1.144: icmp_seq=24 ttl=64 time=0.277 ms
64 bytes from 192.168.1.144: icmp_seq=25 ttl=64 time=0.265 ms
64 bytes from 192.168.1.144: icmp_seq=26 ttl=64 time=0.264 ms
64 bytes from 192.168.1.144: icmp_seq=27 ttl=64 time=0.273 ms
64 bytes from 192.168.1.144: icmp_seq=28 ttl=64 time=168 ms
64 bytes from 192.168.1.144: icmp_seq=29 ttl=64 time=93.3 ms
64 bytes from 192.168.1.144: icmp_seq=30 ttl=64 time=379 ms

--- 192.168.1.144 ping statistics ---
31 packets transmitted, 30 received, 3% packet loss, time 30011ms
rtt min/avg/max/mdev = 0.264/162.221/567.720/181.810 ms

Notice that it's fine for 10 seconds and then goes to hell again.
I have seen ping times exceeding 1000ms on a routine basis.

dom0 network performance is absolutely normal.

Any suggestions?  I am pulling my hair out.  I tried running fewer domU's
(normally I run 11) and it seems to make no difference.

I have tried SMP and non-SMP, with no difference.  Hardware is 2 x
Athlon MP 1800+ on a Tyan Tiger MPX motherboard.  3.5G RAM.  3ware disk
controller and 3com 3c59x network.

Thanks

Jeff

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users