[Xen-users] RE: [Ebtables-user] complex bridge and nat problem

["Eric Low" <elow@xxxxxxxxxxxx>]

Steffen,

Do you have the proper rules in place to make the firewall stateful?  For 
example:

 iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

Also, can you see the reply packets with a logging rule on the Gateway Bridge?  
Someplace early... like the mangle prerouting or nat prerouting chain.  If I 
recall, the packets should still appear to have the translated address (as the 
destination on the reply) all the way through the chains.  But don't quote me 
on it.  Anyhow, if iptables can see the packets there, you can narrow down 
where to look.  Put in logging rules on each chain and see where it dies.

Here's what I think is happening... I think that because you're double 
bridging, your Gateway Bridge does not know where to route the packets back to. 
 I know that doesn't make a lot of sense with a bridge, but you're obviously 
not doing straight bridging when you SNAT.  From experience, it seems that 
iptables will just drop the packet in certain NAT'ing circumstances if it 
doesn't have either an ARP entry or a ROUTE... or possibly both.

Try the following things:
1) Double SNAT - SNAT on the first bridge, then SNAT again on the second.
2) Add an alias from the client subnet to xenbr0 (xenbr0:0)
3) Check your routing table and try adding routes to the appropriate subnets (a 
route to the client subnet through vif0.0).

Hope this helps.  I've tried to do a lot of weird NAT'ing like this in the 
past, but it's been a while.

Eric


-----Original Message-----
From:   ebtables-user-admin@xxxxxxxxxxxxxxxxxxxxx on behalf of Steffen Heil
Sent:   Mon 4/10/2006 8:10 AM
To:     netfilter@xxxxxxxxxxxxxxxxxxx; ebtables-user@xxxxxxxxxxxxxxxxxxxxx; 
xen-users@xxxxxxxxxxxxxxxxxxx
Cc:     
Subject:        [Ebtables-user] complex bridge and nat problem

Hi

I have problems with nat.
My rather complex setup is as follows:

I use a server (running xen), which has two bridges in on linux kernel:
physical eth0 (renamed to peth0) is connected to the first bridge (xenbr0). 
veth0 (renamed to eth0) is connceted to the first bridge (xenbr0).
I call this first bridge the gateway-bridge, it has no ip address.

I call this domain (=VM) my gateway. It has a public ip and is connected to
xenbr0, which is connected to the physical ethernet card, which is connected
to my isps router...

Then I have another bridge (xenbr1), which has a ip-address on it's own in
my $clientnet.
The client domain (=VM) has a virtual network interface which is connected
to that bridge.

So my bridge setup is as:

bridge name     bridge id               STP enabled
- xenbr0          8000.feffffffffff       no
interfaces
- peth0 == physical interface
- vif0.0 == gateways eth0

bridge name     bridge id               STP enabled
- xenbr1          8000.feffffffffff       no
interfaces
- vif22.0 == clients eth0

Finally I have the following SNAT rule (ip4_forward is enabled.)

Chain POSTROUTING (policy ACCEPT 113K packets, 18M bytes)
 pkts bytes target     prot opt in     out     source
destination
   23  1380 SNAT       all  --  *      eth0    $clientnet/24
0.0.0.0/0           to:$gateway

This somehow works:
If I try to reach my outside ssh server (by ip), I get:

GI = Gateway Interface [tcpdump -nni peth0 host $server]
GB = Gateway Bridge [tcpdump -nni xenbr0 host $server]
CI = Client Interface [tcpdump -nni vif22.0 host $server]
CB = Client Bridge [tcpdump -nni xenbr1 host $server]
SI = Server [tcpdump -nni eth0 host $gateway]

CI> 11:40:00.770005 IP $client.2958 > $server.22: S 3227338208:3227338208(0)
win 5840 <mss 1460,sackOK,timestamp 12927453 0,nop,wscale 2>
CB> 11:40:00.770250 IP $client.2958 > $server.22: S 3227338208:3227338208(0)
win 5840 <mss 1460,sackOK,timestamp 12927453 0,nop,wscale 2>
GB> 11:40:00.770416 IP $gateway.2958 > $server.22: S
3227338208:3227338208(0) win 5840 <mss 1460,sackOK,timestamp 12927453
0,nop,wscale 2>
GI> 11:40:00.770571 IP $gateway.2958 > $server.22: S
3227338208:3227338208(0) win 5840 <mss 1460,sackOK,timestamp 12927453
0,nop,wscale 2>
SI> 13:40:01.108827 IP $gateway.2958 > $server.22: S
3227338208:3227338208(0) win 5840 <mss 1460,sackOK,timestamp 12927453
0,nop,wscale 2>
SI> 13:40:01.108863 IP $server.22 > $gateway.2958: S
1070006580:1070006580(0) ack 3227338209 win 5792 <mss 1460,sackOK,timestamp
4101418364 12927453,nop,wscale 2>
GI> 11:40:00.779428 IP $server.22 > $gateway.2958: S
1070006580:1070006580(0) ack 3227338209 win 5792 <mss 1460,sackOK,timestamp
4101418364 12927453,nop,wscale 2>

So the client sends the SYN, the client bridge passes this to the gateway,
the gateway does SNAT and forwards it to the gateway bridge, the gateway
bridge sends this through the physical interface.
The ssh-Server responds with SYN ACK, and this arrives at the gateway's
physical interface. However, it doesn't make it till the first bridge.

I would expect it at leat to reach the gateway bridge and then the gateway.
I even hoped SNAT would do it's job, rewrite the address and forward it to
the client over the client bridge.
But it DOES NEVER reach the gateway bridge.

Can someone tell me, why packets that where SNATed earlier don't make it up
to the gateway bridge?
The gateway itself (and other domains connected to the gateway bridge work
just fine.

Regards,
  Steffen


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users