Xen project Mailing List

[Xen-devel] Conntrack checksumming problem with pv_ops dom0

From: Dmitriy Novikov <dimetrios@xxxxxxxxx>

Date: Wed, 28 Apr 2010 16:08:24 +0300

Delivery-date: Wed, 28 Apr 2010 06:09:29 -0700

Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=RUft2SySRPvR+6szYNq/2TfS6akajP4oT1qkLMJYhVU4Pney+qVXeWPdiStUuQzXAD zv0an600IhSKkgQrvDtJeDlaD+1WkZv7OvJnd5mhQq4DRRrAksB6vV9lfZABRwrPJ1m6 dwog81ko0DwkveRsY0izIfXu3xIxG/8k/7Sag=

List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Hello. I have 2.6.32.11 pv_ops dom0 kernel, xen 3.4.3. dom0 has 2 x Broadcom NICs: 02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) On second interface 4,6 tagged vlan configured and connected to corresponding bridges: br4 8000.0030485a308f no eth1.4 vif1.1 br6 8000.0030485a308f no eth1.6 vif1.0 domU configured with 2 interfeces (WAN,LAN) which connected to corresponding bridges: vif = ['mac=00:16:3e:00:06:11,bridge=br6','mac=00:16:3e:00:04:11,bridge=br4'] domU used for OpenVPN. OpenVPN runs over tun1 interface. OpenVPN traffic comes from eth0 and forwarded to local network via eth1 interface. All checksumming disabled on xen-netfront interfaces: # ethtool -k eth0 Offload parameters for eth0: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off All checksumming disabled on real and vif/dom0 interfaces: # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off # ethtool -k vif1.1 Offload parameters for vif1.1: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off # ethtool -k vif1.0 Offload parameters for vif1.0: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off Broadcom offloading firmware removed from /lib/firmware: mv /lib/firmware/2.6.32.11-xen-dom0/tigon ~/fw_backup Apr 27 18:26:09 xenon kernel: [ 0.562660] tg3.c:v3.102 (September 1, 2009) Apr 27 18:26:09 xenon kernel: [ 0.562800] xen: registering gsi 26 triggering 0 polarity 1 Apr 27 18:26:09 xenon kernel: [ 0.562817] alloc irq_desc for 26 on node 0 Apr 27 18:26:09 xenon kernel: [ 0.562821] alloc kstat_irqs on node 0 Apr 27 18:26:09 xenon kernel: [ 0.562831] xen: --> irq=26 Apr 27 18:26:09 xenon kernel: [ 0.562841] tg3 0000:02:05.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26 ... Apr 27 18:26:09 xenon kernel: [ 0.607550] eth0: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address 00:30:48:5a:30:8e Apr 27 18:26:09 xenon kernel: [ 0.607708] eth0: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1]) Apr 27 18:26:09 xenon kernel: [ 0.607856] eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[0] <---------------RXcsums[1] ??? Apr 27 18:26:09 xenon kernel: [ 0.607965] eth0: dma_rwctrl[769f4000] dma_mask[64-bit] Apr 27 18:26:09 xenon kernel: [ 0.608111] xen: registering gsi 19 triggering 0 polarity 1 Apr 27 18:26:09 xenon kernel: [ 0.608128] alloc irq_desc for 19 on node 0 Apr 27 18:26:09 xenon kernel: [ 0.608132] alloc kstat_irqs on node 0 Apr 27 18:26:09 xenon kernel: [ 0.608142] xen: --> irq=19 ... Apr 27 18:26:09 xenon kernel: [ 0.615909] tg3 0000:02:05.1: PCI INT B -> GSI 27 (level, low) -> IRQ 27 Apr 27 18:26:09 xenon kernel: [ 0.636546] eth1: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address 00:30:48:5a:30:8f Apr 27 18:26:09 xenon kernel: [ 0.636704] eth1: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1]) Apr 27 18:26:09 xenon kernel: [ 0.636851] eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1] Apr 27 18:26:09 xenon kernel: [ 0.636961] eth1: dma_rwctrl[769f4000] dma_mask[64-bit] ... Apr 27 18:26:09 xenon kernel: [ 7.340398] tg3 0000:02:05.1: firmware: requesting tigon/tg3_tso.bin Apr 27 18:26:09 xenon kernel: [ 7.369999] eth1: Failed to load firmware "tigon/tg3_tso.bin" Apr 27 18:26:09 xenon kernel: [ 7.370123] eth1: TSO capability disabled. Before modprobing nf_conntrack_ipv4 forwarding works fine: # tcpdump -ni tun1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes 15:13:19.525889 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 17, length 64 15:13:19.540332 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 17, length 64 15:13:20.526379 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 18, length 64 15:13:20.540641 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 18, length 64 # tcpdump -ni eth1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 15:13:41.550283 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 39, length 64 15:13:41.563788 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 39, length 64 15:13:42.551962 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 40, length 64 15:13:42.565110 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 40, length 64 # tcpdump -ni vif1.1 icmp tcpdump: WARNING: vif1.1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes 15:14:23.823167 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 81, length 64 15:14:23.836664 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 81, length 64 15:14:24.823795 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 82, length 64 15:14:24.836965 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 82, length 64 But after modprobing nf_conntrack_ipv4 traffic destinated to local network disappears on vif1.1 while traffic is visible on domU's lan interface. A lot of "Attempting to checksum a non-TCP/UDP packet, dropping a protocol 1 packet" messages seen on dom0. # tcpdump -ni eth1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 15:15:25.603809 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 143, length 64 15:15:25.618697 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 143, length 64 15:15:26.610258 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 144, length 64 15:15:26.626101 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 144, length 64 # tcpdump -ni tun1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes 15:15:41.680256 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 159, length 64 15:15:41.694291 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 159, length 64 15:15:42.681677 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 160, length 64 15:15:42.697395 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id 2401, seq 160, length 64 # tcpdump -ni vif1.1 icmp tcpdump: WARNING: vif1.1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes 15:15:58.036979 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 175, length 64 15:15:59.043942 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 176, length 64 15:16:00.052912 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 177, length 64 15:16:01.053850 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id 2401, seq 178, length 64 BUT when I set 'sysctl -w net.netfilter.nf_conntrack_checksum=0' traffic starts to run again. Any ideas? _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.