[Xen-users] cross-machine + cross-domain routing

["Zhan Xiaoying" <xyzhan@xxxxxxxxxxxxx>]

Hi,

I'm setting up *** cross-machine + cross-domain ***" routing, what I
want to achieve is: 
say I have nsmc15, nsmc17, nsmc19 running xen. I want to use nsmc19 as
the "router" to connect the two "clouds of subnets" running on nsmc15,
nsmc17 respectively. That is, if I have 10.15.0.4 as dom1 on nsmc15, and
10.17.0.4 as dom1 on nsmc17, I hope they can ping each other via the
router in the middle (dom1 on nsmc19).

What I have done is:
On nsmc19
1. keep xend configuration as bridging.
2. Generate dom1 with the configuration:
kernel = "/boot/vmlinuz-2.6.11-xenU"
ramdisk = "/boot/initrd-2.6.11.10-xenU.img"
memory = 256
name = "router"
nics=2
vif=['ip=10.15.0.1/16,mac=00:00:0a:0f:00:01',
'ip=10.17.0.1/16,mac=00:00:0a:11:00:01']
disk = [ 'phy:vg/router2,sda1,w' ]
root = "/dev/sda1 ro"
extra = "3"
3. By running "brctl show",and "ifconfig", it seems bothe xen-br0,
vif5.0, vif5.1 is "up" and working.
----------------------------------------------------------------
On nsmc17
1. keep xend configuration as routing
2. generate the dom1 with configuration kernel =
"/boot/vmlinuz-2.6.11-xenU"
ramdisk = "/boot/initrd-2.6.11.10-xenU.img" 
memory = 256
name = "worker174"
nics=1
vif = ['ip=10.17.0.4']
disk = [ 'phy:vg/worker1,sda1,w' ]
root = "/dev/sda1 ro"
ip="10.17.0.4" 
netmask="255.255.0.0"
gateway="130.309.251.17"
hostname="worker174"
extra = "3"
3. add this entry to routing:
route add -host 10.17.0.1 gw 130.209.251.19 eth0
---------------------------------------------------------------------
On nsmc15
1. keep xend configuration as routing
2. generate the dom1 with configuration 
kernel = "/boot/vmlinuz-2.6.11-xenU"
ramdisk = "/boot/initrd-2.6.11.10-xenU.img" 
memory = 256
name = "worker154"
nics=1
vif=['ip=10.15.0.4']
disk = [ 'phy:vg/worker1,sda1,w' ]
root = "/dev/sda1 ro"
ip="10.15.0.4"
netmask="255.255.0.0"
gateway="130.209.251.15"
hostname="worker154"
extra = "3"
3. add this entry to routing:
route add -host 10.15.0.1 gw 130.209.251.19 eth0
----------------------------------------------------------
so far all routing and ping results are:
1.From nsmc19, (130.209.251.19), it *** CAN NOT *** ping 10.15.0.1,
neither 10.17.0.1, although they are on same box.
Kernel IP routing table (nsmc19)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
130.209.240.0   0.0.0.0         255.255.240.0   U     0      0        0
xen-br0
10.17.0.0       0.0.0.0         255.255.0.0     U     0      0        0
xen-br0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
eth0
10.15.0.0       0.0.0.0         255.255.0.0     U     0      0        0
xen-br0
0.0.0.0         130.209.240.48  0.0.0.0         UG    0      0        0
xen-br0

Kernel IP routing table (dom1 on nsmc19, acting as router)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
10.17.0.0       0.0.0.0         255.255.0.0     U     0      0        0
eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
eth1
10.15.0.0       0.0.0.0         255.255.0.0     U     0      0        0
eth0
2.From nsmc17 (130.209.251.17),it can ping 10.17.0.1
But from dom1 on nsmc17 (10.17.0.4) it still*** CAN NOT ***  ping
10.17.0.1
Kernel IP routing table (nsmc17)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
10.17.0.4       0.0.0.0         255.255.255.255 UH    0      0        0
vif6.0
10.17.0.1       130.209.251.19  255.255.255.255 UGH   0      0        0
eth0
130.209.240.0   0.0.0.0         255.255.240.0   U     0      0        0
eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
eth0
0.0.0.0         130.209.240.48  0.0.0.0         UG    0      0        0
eth0

Kernel IP routing table (dom1 on nsmc17)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
10.17.0.0       0.0.0.0         255.255.0.0     U     0      0        0
eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
lo
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0
eth0
3.From nsmc15 (130.209.251.15),it can ping 10.15.0.1
But from dom1 on nsmc15 (10.15.0.4) it still*** CAN NOT ***  ping
10.15.0.1
Kernel IP routing table (nsmc15)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
10.15.0.1       130.209.251.19  255.255.255.255 UGH   0      0        0
eth0
10.15.0.4       0.0.0.0         255.255.255.255 UH    0      0        0
vif12.0
130.209.240.0   0.0.0.0         255.255.240.0   U     0      0        0
eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
eth0
0.0.0.0         130.209.240.48  0.0.0.0         UG    0      0        0
eth0

Kernel IP routing table(dom1 on nsmc15)
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0
lo
10.15.0.0       0.0.0.0         255.255.0.0     U     0      0        0
eth0
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0
eth0

4. of couse, 10.15.0.4 and 10.17.0.4 can not ping each other.

I have tried everything I can remember. I once put everything on one
machine (two end hosts + one router in the middle, using routing rather
than bridging), it works. But when I put it across multiple machines, it
fails. Am I missing something? Could anyone give some help? 

Thanks indeed for any help.
Zhan 


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users