Re: [Xen-users] Release 0.8.0 of GPL PV Drivers for Windows

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Sat, Mar 01, 2008 at 10:25:18PM -0500, jim burns wrote:
> On Saturday 01 March 2008 05:38:55 pm you wrote:
> > What NIC you have?
> 
> The numbers for my physical nic were just included for comparison purposes, 
> to 
> show what the range of possible rates could be. I don't really care about 
> those numbers because all my active vms are on file backed vbds on the xen 
> server's (fc8) disk. All the other, more significant #s were for software 
> nics, with no intermediate hardware nics. However:
> 

Yep. I asked because of the "bad" 80 Mbit/sec result on a 100 Mbit network.. 

> > What driver and version of the driver? 
> > "ethtool -i device"
> 
> On the client side (SuSE 10.3):
> 
> [815] > ethtool -i eth0
> driver: e100
> version: 3.5.17-k4-NAPI
> firmware-version: N/A
> bus-info: 0000:02:08.0
> [816] > lspci|grep 02:08.0
> 02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (LOM) 
> Ethernet Controller (rev 81)
> 
> and on the server side (fc8):
> [717] > ethtool -i peth0
> driver: b44
> version: 1.01
> firmware-version:
> bus-info: 0000:03:00.0
> jimb@Insp6400 03/01/08  9:58PM:~
> [718] > lspci|grep 03:00.0
> 03:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 
> 02)
>

OK. 

e100 NIC is a good one, b44 one is not one of the best NICs out there..
 
> > Did you try disabling checksum offloading? "ethtool -K ethX tx off"
> > Try that on dom0 and/or on domU. Maybe also "ethtool -K ethX tso off"
> 
> Why would I do that? That's not how I operate normally. Doesn't that take 
> checksumming out of the hardware, and put it in software, slowing things 
> down? What are the advantages here? However, my current settings are:
> 

That's because with some version of xen and/or drivers (I'm not sure
actually) it was a known fact that performance got bad when you had hw checksum
calculations turned on..

So just to see if that's the case here.. I guess this was mostly for domU.. 

> SuSE:
> [817] > ethtool -k eth0
> Offload parameters for eth0:
> Cannot get device rx csum settings: Operation not supported
> Cannot get device tx csum settings: Operation not supported
> Cannot get device scatter-gather settings: Operation not supported
> Cannot get device tcp segmentation offload settings: Operation not supported
> Cannot get device udp large send offload settings: Operation not supported
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
>

Maybe try turning on offloading/checksumming settings here? 
 

> fc8:
> [720] > ethtool -k peth0
> Offload parameters for peth0:
> Cannot get device rx csum settings: Operation not supported
> Cannot get device tx csum settings: Operation not supported
> Cannot get device scatter-gather settings: Operation not supported
> Cannot get device tcp segmentation offload settings: Operation not supported
> Cannot get device udp large send offload settings: Operation not supported
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: off
>

OK. Maybe try turning on here too, at least for testing with the physical suse
box.. just to see if it has any effect?
 
> > Does your ethX interface have errors? Check with "ifconfig ethX".
> >
> > Do you have tcp retransmits? Check with "netstat -s".
> 
> Before a test, on the fc8 side:
> [721] > ifconfig peth0; netstat -s
> peth0     Link encap:Ethernet  HWaddr 00:15:C5:04:7D:4F
>           inet6 addr: fe80::215:c5ff:fe04:7d4f/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1492  Metric:1
>           RX packets:10566824 errors:0 dropped:219 overruns:0 frame:0
>           TX packets:12540392 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:4070940466 (3.7 GiB)  TX bytes:3443253043 (3.2 GiB)
>           Interrupt:22
> 
> Tcp:
>     6370 active connections openings
>     278 passive connection openings
>     5616 failed connection attempts
>     19 connection resets received
>     9 connections established
>     15612021 segments received
>     16291429 segments send out
>     296 segments retransmited
>     0 bad segments received.
>     5834 resets sent
> 
> and after:
> [722] > iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 192.168.1.100 port 5001 connected with 192.168.1.101 port 26433
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-60.1 sec   539 MBytes  75.3 Mbits/sec
> jimb@Insp6400 03/01/08 10:08PM:~
> [723] > ifconfig peth0; netstat -s
> peth0     Link encap:Ethernet  HWaddr 00:15:C5:04:7D:4F
>           inet6 addr: fe80::215:c5ff:fe04:7d4f/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1492  Metric:1
>           RX packets:10962754 errors:0 dropped:237 overruns:0 frame:0
>           TX packets:12715673 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:369386256 (352.2 MiB)  TX bytes:3459549868 (3.2 GiB)
>           Interrupt:22
> 

Ok, so a bit more drops.. no errors at least. 


> Tcp:
>     6382 active connections openings
>     279 passive connection openings
>     5628 failed connection attempts
>     19 connection resets received
>     9 connections established
>     16008836 segments received
>     16467614 segments send out
>     296 segments retransmited
>     0 bad segments received.
>     5846 resets sent


But no more retransmits.. 

> 
> Which shows a modest increase in drops in ifconfig, and no real significant 
> differences in netstat, except for the TcpExt: section.
> 

Yep..

> > I think it might be a good idea to "force" good/big tcp window size to get
> > comparable results..
> 
> I did a couple of 32k window size tests, with not much significant difference.
> 

I usually use at least 256k window sizes :) 

Did you try with multiple threads at the same time? Did it have any effect? 

> > Some things to check:
> >
> > - txqueuelen of ethX device. I guess 1000 is the default nowadays.. try
> > with bigger values too. This applies to dom0 and to linux domU.
> >
> > - txqueuelen of vifX.Y devices on dom0. Default has been really small, so
> >   make sure to configure that bigger too.. This applies to both linux
> >   and windows vm's.
> 
> I guess this is done on ifconfig, since it appears in an ifconfig output. Is 
> it done in ipconfig for windows?
> 

"ifconfig eth0 txqueuelen <value>" on linux.. I don't know how to do that in
windows. 

But it's important to do that on dom0 for vifX.Y devices.. those are the dom0
sides of the virtual machine virtual NICs. 

> > - Check sysctl net.core.netdev_max_backlog setting.. it should be at least
> >   1000, possibly even more.. this applies to dom0 and linux domU.
> 
> Where is this set, and what do I have to restart to make it 
> effective? /etc/sysctl.conf?
> 

Yep, modify /etc/sysctl.conf and run "sysctl -p /etc/sysctl.conf".

> In general, are there any downsides in changing these values?
> 

http://kb.pert.geant2.net/PERTKB/InterfaceQueueLength

There's something about these settings

btw. what was the CPU usage for dom0 and for domU when you did these iperf
tests? 

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users