RE: [Xen-users] High network latency on first packet

["Russ Purinton" <rpurinton@xxxxxxxxxxxxxxxxxxxxxxx>]

Try simultaneous tcpdumps on dom0 and domU while trying this test.  You
can then see the ARP traffic and maybe be able to determine where the
delay is introduced.  I guess it's possible that ARP resolution time
could be incorporated into the first RTT result.  I just tested ARP
response times on my physical and xen servers and saw resolution usually
occurring between 0.02 ms and 0.2 ms.  

If ARP isn't your problem, tcpdumps should show this too.
The command I used

tcpdump -w file.pcap -nei eth0
                
                
        Russ Purinton
Director of Network Operations
Office   207.591.6900   
Direct:  207.591.6908   
Fax:     207.591.6919   
rpurinton@xxxxxxxxxxxxxxxxxxxxxxx
www.voipnettechnologies.com
                

-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Luca Corti
Sent: Tuesday, April 27, 2010 5:47 AM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] High network latency on first packet

Hi all,

My setup is Debian testing dom0/domUs with a 2.6.32 pvops kernel from
Debian unstable. Hypervisor is 3.4.2 from Debian testing.
I use network- and vif- route with a default route in domU pointing to
the nic because multiple IPs with bridge would trigger port shutdown on
the switch (only one mac-address allowed per port). A subnet is routed
to the dom0, which then knows which addresses the domUs have.

The problem is that every first packet between a domU and a host not in
the arp cache has a very high delay. This is protocol independent
(happens for http and dns too) but I'll provide ping as an example:

Route in domU:
--------------

$ ip route
default dev eth0  scope link 


Ping from domU to remote host:
------------------------------

$ ping 1.2.3.4
PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=304 ms
64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.7 ms
64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=22.7 ms
^C
--- 1.2.3.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 22.702/116.691/304.574/132.853 ms

Immediately replying remote host on second try.

$ ping 1.2.3.4
PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=23.0 ms
64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.8 ms
64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=23.1 ms
^C
--- 1.2.3.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 22.895/23.010/23.103/0.151 ms



Ping from remote host to domU:
------------------------------

Initial ping to domU from remote host.

$ ping 5.6.7.8
PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data.
64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=458 ms
64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=24.9 ms
64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.3 ms
^C
--- 5.6.7.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 24.936/169.512/458.272/204.184 ms

Immediately replying remote host on second try.

$ ping 5.6.7.8
PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data.
64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=24.6 ms
64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=25.3 ms
64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.2 ms
^C
--- 5.6.7.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 24.612/25.068/25.300/0.371 ms


Initial time goes up as high as 800ms. My guess is that this has
something to do with arp. Since when a host is in the arp cache, the
delay disappears, which is why the second ping gets an immediate reply.

I tried specifing mac addresses in the domU cfg files and playing with
domU routing flags without apparent benefit.

Is this an issue with Xen? Is this related to my setup?

thanks

Luca


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users