[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] losing network packets on heavy load, xen 3.0.3


  • To: Luis Vinay <luisvinay@xxxxxxxxx>, <xen-users@xxxxxxxxxxxxxxxxxxx>
  • From: Daniel Nielsen <djn@xxxxxxxxxx>
  • Date: Mon, 27 Nov 2006 10:29:21 +0100
  • Delivery-date: Mon, 27 Nov 2006 01:29:36 -0800
  • List-id: Xen user discussion <xen-users.lists.xensource.com>
  • Thread-index: AccSBoTFwyurNn35EduDggAUUWMFRg==
  • Thread-topic: [Xen-users] losing network packets on heavy load, xen 3.0.3

Hi Luis.

It sounds very similar to our issues with Xen and bridged networking. In our
case it is also a problem when NFS servers have high disk I/O (I've posted
to this list about it, but no response yet).

Just to verify that it indeed is a similar issue, then if you monitor the
number of dropped packets on the vif in dom0, does the number of dropped
packets increase when running this test? Just use 'ifconfig vifX.X' to see
the status of a vif.

I really like the way Xen makes us think about server management, but if
these network issues isn't resolved one way or the other, then we might need
to rethink our virtualization strategy.

Are there other options for configuring networking in Xen than using the
bridges? /me goes googling.

/Daniel
Portalen 


> From: Luis Vinay <luisvinay@xxxxxxxxx>
> Date: Fri, 24 Nov 2006 13:09:08 -0300
> To: <xen-users@xxxxxxxxxxxxxxxxxxx>
> Subject: [Xen-users] losing network packets on heavy load, xen 3.0.3
> 
> I have made this test many times with similar results all times:
> 
> Two virtual machines, both with Debian 3.1 r3, the firs is running "iperf -s
> -d" and the second runs in a infite loop "iperf -c [ip addr of machine 1]
> -t 60 -N -P 10 -d" whit this creates 10(+10) parallel bidirectional
> connections for 60 seconds on the server side I get this:
> 
> vm-debian-B:~#  iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [  4] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39537
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> write2 failed: Broken pipe
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  6] local 0.0.0.0 port 46381 connected with 10.0.41.129 port 5001
> [  6]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [  9] local 0.0.0.0 port 46383 connected with 10.0.41.129 port 5001
> [  9]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 13] local 10.0.41.131 port 46387 connected with 10.0.41.129 port 5001
> [ 12] local 10.0.41.131 port 46386 connected with 10.0.41.129 port 5001
> [ 14] local 10.0.41.131 port 46388 connected with 10.0.41.129 port 5001
> [ 10] local 10.0.41.131 port 46384 connected with 10.0.41.129 port 5001
> [ 16] local 10.0.41.131 port 46390 connected with 10.0.41.129 port 5001
> [ 11] local 10.0.41.131 port 46385 connected with 10.0.41.129 port 5001
> [  8] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39538
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  7] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39539
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [ 32] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39540
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 35] local 0.0.0.0 port 46411 connected with 10.0.41.129 port 5001
> [ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 39] local 0.0.0.0 port 46415 connected with 10.0.41.129 port 5001
> [ 39]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> [ 41] local 0.0.0.0 port 46417 connected with 10.0.41.129 port 5001
> write2 failed: Broken pipe
> [ 41]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 40] local 0.0.0.0 port 46416 connected with 10.0.41.129 port 5001
> [ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> [ 44] local 0.0.0.0 port 46419 connected with 10.0.41.129 port 5001
> [ 44]  0.0- 0.1 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 43] local 0.0.0.0 port 46418 connected with 10.0.41.129 port 5001
> [ 43]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> [ 34] local 0.0.0.0 port 46410 connected with 10.0.41.129 port 5001
> [ 34]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> [ 42] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39541
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> connect failed: Connection refused
> write2 failed: Broken pipe
> [ 30] local 0.0.0.0 port 46407 connected with 10.0.41.129 port 5001
> [ 30]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 31] local 0.0.0.0 port 46408 connected with 10.0.41.129 port 5001
> [ 31]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 22] local 10.0.41.131 port 46399 connected with 10.0.41.129 port 5001
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Connection reset by peer
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 23] local 0.0.0.0 port 46400 connected with 10.0.41.129 port 5001
> [ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 27] local 0.0.0.0 port 46404 connected with 10.0.41.129 port 5001
> [ 27]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> 
> write2 failed: Broken pipe
> write2 failed: Connection reset by peer
> write2 failed: Connection reset by peer
> write2 failed: Connection reset by peer
> write2 failed: Broken pipe
> [ 28] local 0.0.0.0 port 46405 connected with 10.0.41.129 port 5001
> [ 28]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 19] local 10.0.41.131 port 46396 connected with 10.0.41.129 port 5001
> [ 51] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39542
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 47] local 0.0.0.0 port 46422 connected with 10.0.41.129 port 5001
> [ 47]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 48] local 0.0.0.0 port 46423 connected with 10.0.41.129 port 5001
> [ 48]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 37] local 0.0.0.0 port 46428 connected with 10.0.41.129 port 5001
> [ 37]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 35] local 0.0.0.0 port 46426 connected with 10.0.41.129 port 5001
> [ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 18] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39543
> [ 33] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39544
> [ 50] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39546
> [ 61] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39545
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Connection reset by peer
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [ 22]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
> [SUM]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
> write2 failed: Connection reset by peer
> [ 19]  0.0- 1.2 sec  11.3 KBytes  78.0 Kbits/sec
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 54] local 0.0.0.0 port 46459 connected with 10.0.41.129 port 5001
> [ 54]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 53] local 0.0.0.0 port 46458 connected with 10.0.41.129 port 5001
> [ 53]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 65] local 0.0.0.0 port 46469 connected with 10.0.41.129 port 5001
> [ 65]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 64] local 0.0.0.0 port 46468 connected with 10.0.41.129 port 5001
> [ 64]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 75] local 0.0.0.0 port 46479 connected with 10.0.41.129 port 5001
> [ 75]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 67] local 0.0.0.0 port 46471 connected with 10.0.41.129 port 5001
> [ 67]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 68] local 0.0.0.0 port 46472 connected with 10.0.41.129 port 5001
> [ 68]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 74] local 0.0.0.0 port 46478 connected with 10.0.41.129 port 5001
> [ 74]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 17] local 0.0.0.0 port 46438 connected with 10.0.41.129 port 5001
> [ 17]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 40] local 0.0.0.0 port 46432 connected with 10.0.41.129 port 5001
> [ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 23] local 0.0.0.0 port 46437 connected with 10.0.41.129 port 5001
> [ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 20] local 0.0.0.0 port 46439 connected with 10.0.41.129 port 5001
> [ ... ]
> [  5]  0.0-57.9 sec  11.1 MBytes  1.61 Mbits/sec
> 
> the client side shows this:
> Debian31r1-vm1:~# iperf -c 10.0.41.131 -t 60 -N -P 10 -d
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to 10.0.41.131, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.0.41.129 port 39538 connected with 10.0.41.131 port 5001
> [  6] local 10.0.41.129 port 39540 connected with 10.0.41.131 port 5001
> [  8] local 10.0.41.129 port 39542 connected with 10.0.41.131 port 5001
> [  7] local 10.0.41.129 port 39541 connected with 10.0.41.131 port 5001
> [  9] local 10.0.41.129 port 39543 connected with 10.0.41.131 port 5001
> [ 10] local 10.0.41.129 port 39544 connected with 10.0.41.131 port 5001
> [  3] local 10.0.41.129 port 39537 connected with 10.0.41.131 port 5001
> [ 13] local 10.0.41.129 port 39546 connected with 10.0.41.131 port 5001
> [  5] local 10.0.41.129 port 39539 connected with 10.0.41.131 port 5001
> [ 14] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46384
> [ 15] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46385
> [ 16] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46386
> [ 17] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46387
> [ 18] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46388
> [ 19] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46389
> [ 11] local 10.0.41.129 port 39545 connected with 10.0.41.131 port 5001
> [ 20] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46391
> [ 22] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46392
> [ 23] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46393
> [ 21] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46390
> [  4]  0.0-60.0 sec  11.6 MBytes  1.62 Mbits/sec
> [  7]  0.0-60.0 sec  11.8 MBytes  1.65 Mbits/sec
> [  6]  0.0-60.0 sec  12.2 MBytes  1.71 Mbits/sec
> [ 14]  0.0-60.0 sec  18.1 MBytes  2.53 Mbits/sec
> [ 16]  0.0-60.0 sec  14.1 MBytes  1.97 Mbits/sec
> [ 18]  0.0-60.0 sec  14.3 MBytes  2.00 Mbits/sec
> [ 19]  0.0-60.0 sec  12.5 MBytes  1.75 Mbits/sec
> [ 21]  0.0-60.0 sec  13.2 MBytes  1.85 Mbits/sec
> [ 15]  0.0-60.0 sec  14.4 MBytes  2.01 Mbits/sec
> [ 13]  0.0-60.3 sec  10.1 MBytes  1.40 Mbits/sec
> [  9]  0.0-60.3 sec  8.66 MBytes  1.20 Mbits/sec
> [ 17]  0.0-60.2 sec  12.2 MBytes  1.71 Mbits/sec
> [  8]  0.0-60.5 sec  4.96 MBytes    688 Kbits/sec
> [ 11]  0.0-60.5 sec  11.0 MBytes  1.53 Mbits/sec
> [ 23]  0.0-60.6 sec  17.6 MBytes  2.44 Mbits/sec
> [  5]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
> [ 22]  0.0-60.6 sec  14.0 MBytes  1.93 Mbits/sec
> [  3]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
> [ 10]  0.0-61.1 sec  9.49 MBytes  1.30 Mbits/sec
> [SUM]  0.0-61.1 sec    105 MBytes  14.5 Mbits/sec
> [ 20]  0.0-61.7 sec  11.1 MBytes  1.52 Mbits/sec
> [SUM]  0.0-61.7 sec    142 MBytes  19.3 Mbits/sec
> 
> As we can see there is a lot of connection errors and the speed measurement
> reveals a very poor performance
> 
> Test platform detail:
> 
> Dom0
>    Dell PE1950, 4 Xeon @1.6G, 8Gig memory
>    RedHat Enterprise Server 4 update 4
>    Xen 3.0.3, kernel 2.6.16.29
> 
> Virtual Machines
>    1 CPU, 300Mb memory
>    Debian 3.1r3
>    Kernel 2.6.8-3-386
> 
> Both machines are on the same Dom0 and the use bridged networking
> 
> Any ideas or comments?
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.