raw netif performance enhancement

[Charalampos Rotsos <cr409@xxxxxxxxxxxx>]

Hi all, 

I have been working the last few days on the openflow switch in mirage and I
have some observations which would be interesting to get some feedback from the
rest of the mirage developers. 

The setup is as follows: I have two linux VMs each connected to a bridge (xapi1
and xapi2) which run iperf to measure network performance. In order to bridge
the two VMs I am running a third VM, which can be interchangably a VM running
openvswitch or a Mirage Switch, which is allocated a vif attached to each of the
vifs. The switch VM simply bridges traffic between the two VMs.

Comparing the mirage switch with the ovs I have observed the folloiwng
behaviour. When I am running an openvswitch DumU switch I am able to switch
between two vms approx. 1G of udp traffic using iperf. 

In order to simulate a similar behaviour, I have written a small mirage app
which bridges traffic between 2 vifs. During the Manager.create callback, for
each vif I register a promiscuous callback for each vif using the 
set_promiscuous
Manager method. The callback receives a packet for one of the ports and forwards
it to the other port, using the inject packet method of the Manager. These
methods hook on the OS.Ethif module and listen or write for packets directly on
the OS.Netif entity. 

A first thing I observed with Mirage was that beyond 700Mbps the VM crashes
because it runs out of memory. This behaviour I think is related to the logic of
the listen method which spawns a new thread for each packet received. If the 
thread
creation rate is lower than the packet processing rate, then the VM cannot
fulfil the memory requirements and dies. This is easily solvable
using a capped Lwt_stream which functions as a NIC rx queue. Using this trick I
managed to solve the memory sortage problem. 

Now my next problem is that the VM cannot switch more that 700Mbps, while the
CPU utilisation is maximum 60% during execution. I inserted some counter in the
code and noticed that the tx ring tends to drop some packets (I compare the
number of packet I inject to the vif with the number of packets observed on the
end host of the iperf test). I have verified using tc and ifconfig that these
packets are not lost in between the vif during the switching process (you need
to do a bit of configuration in the txqueuelen in order to ensure that packets
are not dropped). My guess so far is that the performance bottleneck is on the
TX ring of the vif. Are there any hints on how I could improve the performance
or any ideas on the bottleneck point? 


-- 
Charalampos Rotsos
PhD student
The University of Cambridge
Computer Laboratory
William Gates Building
JJ Thomson Avenue
Cambridge
CB3 0FD
Phone: +44-(0) 1223 767032
Email: cr409@xxxxxxxxxxxx