Re: [Xen-users] network-bridge breaks networking when eth0:1 is added

[Kai Schaetzl <maillists@xxxxxxxxxxxxx>]

 Nugraha wrote on Tue, 28 Apr 2009 19:58:26 +0700:

> Seriously?

Yes. I would love to give you all the details. The problem is that what 
I'm working with is not the final setup but some "semblance". I had to 
improvise to get something that *should* behave similar than the final 
setup.
The final setup will be machine a4 with three NICs attached to two other 
machines (say a1 and a2, both with more than one NIC) via crossover cable 
and a4 pulling from the domUs on a1 and a2 data. I want to go thru the 
crossover link and not the switch, via a private network where all 
additional NICs and the domUs are on.
The problem is that a1 and a2 are not here in the office, but in 
production in the datacenter. So, I have a4 for testing and maybe can 
attach another box if need be. But most boxes I have here are in use and 
have only one NIC. So I can use them only for a short time and have to 
reconfigure them each time. I did this once and tested from another box 
(name it a3) to domUs on a4 with the network setup I mentioned in my first 
posting, so everythign done by the standard network-bridge script. I was 
happy with that as long as I didn't reboot and noticed the initial running 
of network-bridge breaks the eth0/peth0 bridge.
Now, as I cannot always attach another machine for testing (and I don't 
have one with two NICs anyway) my base assumption is that what works from 
a3 -> a4 -> domU should also work without a3. e.g. I don't ping from a3 
but from a4 dom0 -> a4 domUs, but using the interface/IP number that is on 
the private subnet ("ping 192.168.2.10" for instance, or "ping -I xenbr1 
192.168.2.10" which would ping from the xenbr1 interface which is bridging 
to eth1, where the external machine comes in).
I'm using this simplified test scenario for testing. ping the private IP 
on domU from dom0. If it works, fine. If it doesn't, use -I xenbr1 for the 
ping. If that works, fine, but not so comfortable. If that doesn't work: 
bad :-(

Does this make sense so far?

Now I created two bridges as you suggested on a4 and two NICs on the dom0. 
That works but only in one way (dom0 -> domU) and only if I attach the 
ping to xenbr1. I just played around with routes and have found that 
adding a static route (on dom0) solves this so I can ping from either side 
and without specifying an interface. This solution is much more 
comfortable. The private subnet in question is 192.168.2.0/27.
The output of this setup is now as follows:

dom0:
brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.001ec9fefbab       no              eth0
                                                        vif14.0
xenbr1          8000.001ec9fefbac       no              eth1
                                                        vif14.1

ip addr list | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 192.168.2.4/27 brd 192.168.2.31 scope global eth2
    inet 192.168.1.24/24 brd 192.168.1.255 scope global xenbr0
    inet 192.168.2.3/27 brd 192.168.2.31 scope global xenbr1

ip route
192.168.2.10 via 192.168.2.3 dev xenbr1  scope link
192.168.2.0/27 dev eth2  proto kernel  scope link  src 192.168.2.4
192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
192.168.1.0/24 dev xenbr0  proto kernel  scope link  src 192.168.1.24
default via 192.168.1.1 dev xenbr0

domU is straight forward:
no bridge

ip addr list | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 212.202.99.237/28 brd 212.202.99.239 scope global eth0
    inet 192.168.1.237/24 brd 192.168.1.255 scope global eth0:1
    inet 192.168.2.10/27 brd 192.168.2.31 scope global eth1

ip route
212.202.99.224/28 dev eth0  proto kernel  scope link  src 212.202.99.237
192.168.2.0/27 dev eth1  proto kernel  scope link  src 192.168.2.10
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.237
default via 192.168.1.1 dev eth0  src 192.168.1.237
default via 212.202.99.225 dev eth0

I had forwarding enabled in the meantime, but it works without it as well, 
so I disabled that again.

So, just to make it clear, *this* setup with the additional route is 
working now in all directions. I now realize that my best option is 
probably to use a different subnet each for eth1 and eth2 and different 
subnets on the two machines it goes out to (remember a4 will not be the 
target, but the source). If I do that I can use a route for each subnet, 
otherwise I have to use a route per single IP address.

Oh, and I just find that using different nets for eth1 and eth2 solves the 
problem, anyway, without a static route. Like so:
ip route
192.168.3.0/27 dev eth2  proto kernel  scope link  src 192.168.3.1
192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
192.168.1.0/24 dev xenbr0  proto kernel  scope link  src 192.168.1.24
default via 192.168.1.1 dev xenbr0

The big problem obviously was eth2 being on the same subnet as eth1/xenbr1 
and thus catching the packets to nowhere.

Good. I had just tried to add the static route with sysconfig/route-xenbr1 
and all variations I tried failed with obscure errors. I'm going to 
follow-up on this on the CentOS list.

Thanks for your answers, they were all really very helpful!

I think I have found a solution now, although I still would like to know 
why the original method works on some machines but not this one. 
Interestingly, one of the errors I got when trying to use route-xenbr1 was 
the same as when using the original setup and the network-bridge script.

If you are still interested in the original data I can revert this machine 
to the original setup and grab the data from there or from the other 
machine I mentioned where it works fine.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users