[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] losing network packets on heavy load, xen 3.0.3


  • To: xen-users@xxxxxxxxxxxxxxxxxxx
  • From: "Luis Vinay" <luisvinay@xxxxxxxxx>
  • Date: Fri, 24 Nov 2006 13:09:08 -0300
  • Delivery-date: Fri, 24 Nov 2006 08:09:26 -0800
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type; b=A05j30vvpT5LjjWkZhHjYrv1Dklt52RxEzCVw902Q+T5nsbkJsA1zL9ave6+7fZpMOhdlVk5pgXruYwWDNgO+Z6hf4cQIpnT2E305KO6+oE8mqOCRs5EiYWEKfbtHJsmHC0CaXgLeWiWotCxJFS5W1pTpKCmVW07dof7KhH2pvY=
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

I have made this test many times with similar results all times:

Two virtual machines, both with Debian 3.1 r3, the firs is running "iperf -s -d" and the second runs in a infite loop "iperf -c [ip addr of machine 1]  -t 60 -N -P 10 -d" whit this creates 10(+10) parallel bidirectional connections for 60 seconds on the server side I get this:

vm-debian-B:~#  iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
[  4] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39537
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
write2 failed: Broken pipe
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  6] local 0.0.0.0 port 46381 connected with 10.0.41.129 port 5001
[  6]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[  9] local 0.0.0.0 port 46383 connected with 10.0.41.129 port 5001
[  9]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ 13] local 10.0.41.131 port 46387 connected with 10.0.41.129 port 5001
[ 12] local 10.0.41.131 port 46386 connected with 10.0.41.129 port 5001
[ 14] local 10.0.41.131 port 46388 connected with 10.0.41.129 port 5001
[ 10] local 10.0.41.131 port 46384 connected with 10.0.41.129 port 5001
[ 16] local 10.0.41.131 port 46390 connected with 10.0.41.129 port 5001
[ 11] local 10.0.41.131 port 46385 connected with 10.0.41.129 port 5001
[  8] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39538
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  7] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39539
connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
[ 32] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39540
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 35] local 0.0.0.0 port 46411 connected with 10.0.41.129 port 5001
[ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 39] local 0.0.0.0 port 46415 connected with 10.0.41.129 port 5001
[ 39]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
[ 41] local 0.0.0.0 port 46417 connected with 10.0.41.129 port 5001
write2 failed: Broken pipe
[ 41]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ 40] local 0.0.0.0 port 46416 connected with 10.0.41.129 port 5001
[ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
[ 44] local 0.0.0.0 port 46419 connected with 10.0.41.129 port 5001
[ 44]  0.0- 0.1 sec  0.00 Bytes  0.00 bits/sec
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 43] local 0.0.0.0 port 46418 connected with 10.0.41.129 port 5001
[ 43]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
[ 34] local 0.0.0.0 port 46410 connected with 10.0.41.129 port 5001
[ 34]  0.0 - 0.2 sec  0.00 Bytes  0.00 bits/sec
[SUM]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
connect failed: Connection refused
[ 42] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39541
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
connect failed: Connection refused
write2 failed: Broken pipe
[ 30] local 0.0.0.0 port 46407 connected with 10.0.41.129 port 5001
[ 30]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ 31] local 0.0.0.0 port 46408 connected with 10.0.41.129 port 5001
[ 31]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ 22] local 10.0.41.131 port 46399 connected with 10.0.41.129 port 5001
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Connection reset by peer
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 23] local 0.0.0.0 port 46400 connected with 10.0.41.129 port 5001
[ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 27] local 0.0.0.0 port 46404 connected with 10.0.41.129 port 5001
[ 27]  0.0 - 0.0 sec  0.00 Bytes  0.00 bits/sec

write2 failed: Broken pipe
write2 failed: Connection reset by peer
write2 failed: Connection reset by peer
write2 failed: Connection reset by peer
write2 failed: Broken pipe
[ 28] local 0.0.0.0 port 46405 connected with 10.0.41.129 port 5001
[ 28]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 19] local 10.0.41.131 port 46396 connected with 10.0.41.129 port 5001
[ 51] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39542
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 47] local 0.0.0.0 port 46422 connected with 10.0.41.129 port 5001
[ 47]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 48] local 0.0.0.0 port 46423 connected with 10.0.41.129 port 5001
[ 48]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 37] local 0.0.0.0 port 46428 connected with 10.0.41.129 port 5001
[ 37]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 35] local 0.0.0.0 port 46426 connected with 10.0.41.129 port 5001
[ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ 18] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39543
[ 33] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39544
[ 50] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39546
[ 61] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39545
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Connection reset by peer
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
[ 22]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
[SUM]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
write2 failed: Connection reset by peer
[ 19]  0.0- 1.2 sec  11.3 KBytes  78.0 Kbits/sec
connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 54] local 0.0.0.0 port 46459 connected with 10.0.41.129 port 5001
[ 54]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 53] local 0.0.0.0 port 46458 connected with 10.0.41.129 port 5001
[ 53]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 65] local 0.0.0.0 port 46469 connected with 10.0.41.129 port 5001
[ 65]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 64] local 0.0.0.0 port 46468 connected with 10.0.41.129 port 5001
[ 64]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 75] local 0.0.0.0 port 46479 connected with 10.0.41.129 port 5001
[ 75]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 67] local 0.0.0.0 port 46471 connected with 10.0.41.129 port 5001
[ 67]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 68] local 0.0.0.0 port 46472 connected with 10.0.41.129 port 5001
[ 68]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 74] local 0.0.0.0 port 46478 connected with 10.0.41.129 port 5001
[ 74]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
------------------------------------------------------------
Client connecting to 10.0.41.129, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
connect failed: Connection refused
connect failed: Connection refused
connect failed: Connection refused
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 17] local 0.0.0.0 port 46438 connected with 10.0.41.129 port 5001
[ 17]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 40] local 0.0.0.0 port 46432 connected with 10.0.41.129 port 5001
[ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
[ ... ]
[ 23] local 0.0.0.0 port 46437 connected with 10.0.41.129 port 5001
[ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
write2 failed: Broken pipe
write2 failed: Broken pipe
[ 20] local 0.0.0.0 port 46439 connected with 10.0.41.129 port 5001
[ ... ]
[  5]  0.0-57.9 sec  11.1 MBytes  1.61 Mbits/sec

the client side shows this:
Debian31r1-vm1:~# iperf -c 10.0.41.131 -t 60 -N -P 10 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.0.41.131, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  4] local 10.0.41.129 port 39538 connected with 10.0.41.131 port 5001
[  6] local 10.0.41.129 port 39540 connected with 10.0.41.131 port 5001
[  8] local 10.0.41.129 port 39542 connected with 10.0.41.131 port 5001
[  7] local 10.0.41.129 port 39541 connected with 10.0.41.131 port 5001
[  9] local 10.0.41.129 port 39543 connected with 10.0.41.131 port 5001
[ 10] local 10.0.41.129 port 39544 connected with 10.0.41.131 port 5001
[  3] local 10.0.41.129 port 39537 connected with 10.0.41.131 port 5001
[ 13] local 10.0.41.129 port 39546 connected with 10.0.41.131 port 5001
[  5] local 10.0.41.129 port 39539 connected with 10.0.41.131 port 5001
[ 14] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46384
[ 15] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46385
[ 16] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46386
[ 17] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46387
[ 18] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46388
[ 19] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46389
[ 11] local 10.0.41.129 port 39545 connected with 10.0.41.131 port 5001
[ 20] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46391
[ 22] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46392
[ 23] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46393
[ 21] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46390
[  4]  0.0-60.0 sec  11.6 MBytes  1.62 Mbits/sec
[  7]  0.0-60.0 sec  11.8 MBytes  1.65 Mbits/sec
[  6]  0.0-60.0 sec  12.2 MBytes  1.71 Mbits/sec
[ 14]  0.0-60.0 sec  18.1 MBytes  2.53 Mbits/sec
[ 16]  0.0-60.0 sec  14.1 MBytes  1.97 Mbits/sec
[ 18]  0.0-60.0 sec  14.3 MBytes  2.00 Mbits/sec
[ 19]  0.0-60.0 sec  12.5 MBytes  1.75 Mbits/sec
[ 21]  0.0-60.0 sec  13.2 MBytes  1.85 Mbits/sec
[ 15]  0.0-60.0 sec  14.4 MBytes  2.01 Mbits/sec
[ 13]  0.0-60.3 sec  10.1 MBytes  1.40 Mbits/sec
[  9]  0.0-60.3 sec  8.66 MBytes  1.20 Mbits/sec
[ 17]  0.0-60.2 sec  12.2 MBytes  1.71 Mbits/sec
[  8]  0.0-60.5 sec  4.96 MBytes    688 Kbits/sec
[ 11]  0.0-60.5 sec  11.0 MBytes  1.53 Mbits/sec
[ 23]  0.0-60.6 sec  17.6 MBytes  2.44 Mbits/sec
[  5]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
[ 22]  0.0-60.6 sec  14.0 MBytes  1.93 Mbits/sec
[  3]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
[ 10]  0.0-61.1 sec  9.49 MBytes  1.30 Mbits/sec
[SUM]  0.0-61.1 sec    105 MBytes  14.5 Mbits/sec
[ 20]  0.0-61.7 sec  11.1 MBytes  1.52 Mbits/sec
[SUM]  0.0-61.7 sec    142 MBytes  19.3 Mbits/sec

As we can see there is a lot of connection errors and the speed measurement reveals a very poor performance

Test platform detail:

Dom0
   Dell PE1950, 4 Xeon @1.6G, 8Gig memory
   RedHat Enterprise Server 4 update 4
   Xen 3.0.3, kernel 2.6.16.29

Virtual Machines
   1 CPU, 300Mb memory
   Debian 3.1r3
   Kernel 2.6.8-3-386

Both machines are on the same Dom0 and the use bridged networking

Any ideas or comments?
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.