[Xen-devel] Dom0 crash at 40Mbps Iperf traffic with only 80% CPU utilization ??

[Kaushik Bhandankar <kaushikb@xxxxxxxxxxxxx>]

Hello

I have a 3 node experiment as detailed below to estimate the bridged 
networking performance in para-virtualized Xen 3.0 on Emulab 
(https://www.emulab.net/)
Here is how the 3-node topology looks (This topology is specified by 
means of an NS-2 file)
topology:

                 _______________________________
                |                               |
Node0:eth0 ------|Node1:eth0           Node1:eth1|------ Node2:eth0
                |_______________________________|

Node0 and Node2 run some standard 2.6.* kernel whereas Node1 runs 
para-virtualized Xen 3.0 using LVM-created root & swap partitions for 
DomU (2.6.12-xenU)
*******************************************************************
Traffic flow:

            _____________________________________
           |                Node1               |
           |   ______________________________   |
           |  |            DomU              |  |
           |  |                              |  |
           |  |    eth0 ---------> eth1      |  |
           |  |_____|_______________|________|  |
           |        |               |           |
           |      vif1.0          vif1.1        |
           |        |               |           |
           |        |               |           |
           |      xenbr0          xenbr1        |
           |        |      Dom0     |           |
           |        |               |           |
Node0:eth0--|----> eth0            eth1 ---------|-----> Node2:eth0
           |____________________________________|


Node0:eth0, Node1:DomU:eth0 are in 10.1.1/24 subnet whereas 
Node1:DomU:eth1 and Node2:eth0 are in 10.1.2/24 subnet (Since Node1:eth0 
and Node1:eth1 are attached to bridges, they do not have IP addresses on 
them and are not ping'able)
At Node0:eth0, I set the next hop of all traffic as Node1:DomU:eth0 and 
at Node2, I set the next hop of all traffic as Node1:DomU:eth1.
Now, I try to run an Iperf server at Node2 for UDP traffic (const 
64-byte packets size):
iperf -s -i 2 -l 64 -u

and an Iperf client at Node0 for UDP traffic to node2:eth0 (const 64-bit 
packet size, vary the Bandwidth from 10Mbps till possibly 80Mbps)
iperf -c <IP address of Node2:eth0> -u -l 64 -t 90 -i 2 -w 256k -b 10M

Where 10M is the bandwidth of iperf traffic between Node0 and Node2 (in 
Mbps).
I also monitor the CPU usage at Node1 Dom0 using "vmstat 1" at Dom0.

When B/W is 36.6M (Mbps), the CPU utilization at Dom0 is about 70-80% 
but when the B/W is increased to 39.4M (Mbps), the Dom0 crashes in 
typically 5-15 seconds (M SSH session to Dom0 'hangs') but before it 
crashes, the CPU utiliation is till 75-85%. At this point, I have to 
again boot Dom0 and the DomU on top of it.
Does anybody know why this happens ?? and how can I debug this situation 
(if at all) ? I am new to the Xen source code so please be as detailed 
as possible
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel