[Xen-users] Packets on the bridge are disappearing

[Brad Plant <bplant@xxxxxxxxxxxx>]

Hi list,

I have discovered an issue with random packets disappearing after they enter 
the bridge on dom0 hosts. It doesn't seem to matter what the source and 
destination hosts/ports are, but between maybe once and 10 times a day, some 
packets will disappear. I will show the communication between host 10.10.20.1 
(connected via vif_priv_db1) and 10.10.60.1 (connected via vif_priv_dir1) on 
port 9307 to illustrate the problem. Both these virtual hosts are located on 
the same dom0, but the problem also occurs when the source (10.10.60.1) is on a 
different dom0.

The following shows all the traffic on the bridge (br_private) for hosts 
1010.20.1 and 10.10.60.1 on port 9307

09:01:59.122168 IP 10.10.60.1.58663 > 10.10.20.1.9307: S 
2189940749:2189940749(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122289 IP 10.10.60.1.58664 > 10.10.20.1.9307: S 
2201205083:2201205083(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122498 IP 10.10.20.1.9307 > 10.10.60.1.58663: S 
2085426483:2085426483(0) ack 2189940750 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122498 IP 10.10.20.1.9307 > 10.10.60.1.58664: S 
2096234353:2096234353(0) ack 2201205084 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122606 IP 10.10.60.1.58663 > 10.10.20.1.9307: . ack 1 win 46
09:01:59.122607 IP 10.10.60.1.58664 > 10.10.20.1.9307: . ack 1 win 46
09:01:59.122677 IP 10.10.60.1.58663 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:01:59.122714 IP 10.10.60.1.58664 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:00.126949 IP 10.10.60.1.58681 > 10.10.20.1.9307: S 
2198990274:2198990274(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:00.126949 IP 10.10.60.1.58682 > 10.10.20.1.9307: S 
2194428786:2194428786(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110326 IP 10.10.60.1.58681 > 10.10.20.1.9307: S 
2198990274:2198990274(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110402 IP 10.10.60.1.58682 > 10.10.20.1.9307: S 
2194428786:2194428786(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110494 IP 10.10.20.1.9307 > 10.10.60.1.58681: S 
2103435422:2103435422(0) ack 2198990275 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110530 IP 10.10.20.1.9307 > 10.10.60.1.58682: S 
2089908768:2089908768(0) ack 2194428787 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110574 IP 10.10.60.1.58681 > 10.10.20.1.9307: . ack 1 win 46
09:02:03.110608 IP 10.10.60.1.58682 > 10.10.20.1.9307: . ack 1 win 46
09:02:03.110845 IP 10.10.60.1.58681 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:03.110956 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:03.110985 IP 10.10.20.1.9307 > 10.10.60.1.58682: P 1:109(108) ack 1 win 46
09:02:03.110986 IP 10.10.20.1.9307 > 10.10.60.1.58682: F 109:109(0) ack 1 win 46
09:02:03.111050 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 
2194428787:2194428787(0) win 0
09:02:03.111074 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 
2194428787:2194428787(0) win 0

The following shows the traffic on the virtual interface for the domU 
(vif_priv_db1) for host 10.10.60.1 on port 9307

09:01:59.122214 IP 10.10.60.1.58663 > 10.10.20.1.9307: S 
2189940749:2189940749(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122306 IP 10.10.60.1.58664 > 10.10.20.1.9307: S 
2201205083:2201205083(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122498 IP 10.10.20.1.9307 > 10.10.60.1.58663: S 
2085426483:2085426483(0) ack 2189940750 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122498 IP 10.10.20.1.9307 > 10.10.60.1.58664: S 
2096234353:2096234353(0) ack 2201205084 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:01:59.122629 IP 10.10.60.1.58663 > 10.10.20.1.9307: . ack 1 win 46
09:01:59.122637 IP 10.10.60.1.58664 > 10.10.20.1.9307: . ack 1 win 46
09:01:59.122699 IP 10.10.60.1.58663 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:01:59.122724 IP 10.10.60.1.58664 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:03.110372 IP 10.10.60.1.58681 > 10.10.20.1.9307: S 
2198990274:2198990274(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110416 IP 10.10.60.1.58682 > 10.10.20.1.9307: S 
2194428786:2194428786(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110494 IP 10.10.20.1.9307 > 10.10.60.1.58681: S 
2103435422:2103435422(0) ack 2198990275 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110530 IP 10.10.20.1.9307 > 10.10.60.1.58682: S 
2089908768:2089908768(0) ack 2194428787 win 5840 <mss 
1460,nop,nop,sackOK,nop,wscale 7>
09:02:03.110592 IP 10.10.60.1.58681 > 10.10.20.1.9307: . ack 1 win 46
09:02:03.110619 IP 10.10.60.1.58682 > 10.10.20.1.9307: . ack 1 win 46
09:02:03.110857 IP 10.10.60.1.58681 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:03.110969 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 1:1(0) ack 1 win 46
09:02:03.110985 IP 10.10.20.1.9307 > 10.10.60.1.58682: P 1:109(108) ack 1 win 46
09:02:03.110986 IP 10.10.20.1.9307 > 10.10.60.1.58682: F 109:109(0) ack 1 win 46
09:02:03.111062 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 
2194428787:2194428787(0) win 0
09:02:03.111083 IP 10.10.60.1.58682 > 10.10.20.1.9307: R 
2194428787:2194428787(0) win 0

Notice the 2 missing packets at 09:02:00.126949...?

I have also noticed that when this problem occurs, it isn't restricted to only 
that port. At a similar time, there are packets for port 9306 sent from/to the 
same hosts which also magically disappear. I suspect that the problem would 
actually affect any host attempting to communicate with 10.10.20.1 at this 
time, i.e. it affects the interface (vif_priv_db1) and has nothing to do with 
the source host, but I cannot confirm this.

The problem is very short lived; it doesn't seem to exist for more than 1 
second.

I am using Gentoo and I believe the kernel is based on xen 3.1.2. I have tried 
both 2.6.18 kernels and 2.6.20 (from fedora), but the problem still persists. I 
plan on upgrading the kernel patch to 3.2.1 when it is released (I see it is in 
rc5 atm) to see if that has any effect. In the meantime though, has anyone else 
experienced this problem or know of a possible cause?

Thanks in advance,

Cheers,

Brad

Attachment: signature.asc
Description: PGP signature

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users