[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] dual vif per domain troubles



Hi,

I'm trying to run a domU with 2 nics. So i created 2 bridges and a dummy device to let my new bridge talk to. The whole point is to do administrative tasks on the domU without doing extra traffic for my users on the public nic.

When i'm running with just one vif per domU everything runs fine, until i add a second bridge. All traffic is halted on the returnpath from xen-br0 to eth0 (see tcpdump, below)

So traffic from the outside only works with 1 vif per domU
When running with 2 vifs per domain i can ping from my dom0, but not
really usefull, because those domU's won't make sense then.

Maybe the solution is really simple, but since i worked on this for some time now, i've lost my patience :) Ow and all public ip's are masked
just because i don't like my IP be listed in all kind of groups :)

Without further ado, i give you my dumps and such

----------------------------------------------------------------------
The setup.
----------------------------------------------------------------------

--------- | winxp | --------- | /------ dom0 name : atlas / eth0 : 192.168.0.180 | / dummy0: x.x.123.4 ---------/ | atlas | | xen | \ --------- ------- domU name : baseinstall eth0 : x.x.123.9 eth1 : 192.168.0.181
----------------------------------------------------------------------
All info i can give right now with domU running only one vif
This works fine. Pinging from the win-box gives replies
basic config on the domU part. (based on xmexample1)
----------------------------------------------------------------------

atlas:~# ip route
x.x.123.0/26 dev dummy0  proto kernel  scope link  src x.x.123.4
192.168.0.0/24 dev xen-br0  proto kernel  scope link  src 192.168.0.180
default via 192.168.0.50 dev xen-br0

atlas:~# ip addr
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
2: dummy0: <BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue
   link/ether 9e:59:f0:3c:81:8f brd ff:ff:ff:ff:ff:ff
   inet x.x.123.4/26 brd x.x.123.63 scope global dummy0
3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
   link/ether 00:50:bf:d6:57:38 brd ff:ff:ff:ff:ff:ff
   inet 192.168.0.180/24 brd 192.168.0.255 scope global eth0
4: xen-br0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether 00:50:bf:d6:57:38 brd ff:ff:ff:ff:ff:ff
   inet 192.168.0.180/32 brd 192.168.0.255 scope global xen-br0
5: vif1.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

atlas:~# brctl show
bridge name     bridge id               STP enabled     interfaces
xen-br0         8000.0050bfd65738       no              eth0
                                                       vif1.0

atlas:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
x.x.123.0    0.0.0.0         255.255.255.192 U     0      0        0 dummy0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 xen-br0
0.0.0.0         192.168.0.50    0.0.0.0         UG    0      0        0 xen-br0

atlas:~# brctl showmacs xen-br0
port no mac addr                is local?       ageing timer
 1     00:0b:6a:ad:02:c7       no               128.44
 1     00:0c:76:56:b4:c7       no               162.03
 1     00:0e:50:3d:9d:17       no               185.88
 1     00:0e:a6:23:b9:ad       no                 0.00
 1     00:0e:a6:23:d4:4f       no               138.53
 1     00:10:60:38:81:fc       no                38.61
 1     00:50:bf:d6:57:38       yes                0.00
 2     aa:00:00:45:f7:1b       no               185.88
 2     fe:ff:ff:ff:ff:ff       yes                0.00

atlas:~# ifconfig
dummy0    Link encap:Ethernet  HWaddr 9E:59:F0:3C:81:8F
         inet addr:x.x.123.4  Bcast:x.x.123.63  Mask:255.255.255.192
         UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

eth0      Link encap:Ethernet  HWaddr 00:50:BF:D6:57:38
         inet addr:192.168.0.180  Bcast:192.168.0.255  Mask:255.255.255.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:356 errors:0 dropped:0 overruns:0 frame:0
         TX packets:437 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:38274 (37.3 KiB)  TX bytes:41424 (40.4 KiB)
         Interrupt:9 Base address:0xa400

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:684 errors:0 dropped:0 overruns:0 frame:0
         TX packets:684 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:39203 (38.2 KiB)  TX bytes:39203 (38.2 KiB)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:26 errors:0 dropped:0 overruns:0 frame:0
         TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:3567 (3.4 KiB)  TX bytes:5964 (5.8 KiB)

xen-br0   Link encap:Ethernet  HWaddr 00:50:BF:D6:57:38
         inet addr:192.168.0.180  Bcast:192.168.0.255  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:325 errors:0 dropped:0 overruns:0 frame:0
         TX packets:403 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:29849 (29.1 KiB)  TX bytes:35669 (34.8 KiB)


atlas:~# ip maddr
2:      dummy0
       link  01:00:5e:00:00:01
3:      eth0
       link  01:00:5e:00:00:01
4:      xen-br0
       link  01:00:5e:00:00:01


----------------------------------------------------------------------
All info with domU running with 2 vifs. Also note the 2 bridges
and the dummy0 interface. when i try to ping the domU from the winbox. It just failes.
----------------------------------------------------------------------

I modified my domU config file like so: (only relevant parts)

nics=2
vif = ['bridge=xen-loc', 'bridge=xen-br0' ]

I modified the network script like so, to support the 2 bridges
The excisting script no resides as network.xen (which is called from this file)
I copied this example from: http://julien.danjou.info/xen.html

atlas:~# cat /etc/xen/scripts/network
#!/bin/sh
set -e

OP=$1
shift

script=/etc/xen/scripts/network.xen

case ${OP} in
 start)
       $script start bridge=xen-br0 netdev=eth0
       $script start bridge=xen-loc netdev=dummy0
       ;;
 stop)
       $script start bridge=xen-br0 netdev=eth0
       $script start bridge=xen-loc netdev=dummy0
       ;;
 *)
       echo 'Unknown command: ' ${OP}
       echo 'Valid commands are: start, stop'
       exit 1
esac


atlas:~# ip route
x.x.123.0/26 dev xen-loc  proto kernel  scope link  src x.x.123.4
192.168.0.0/24 dev xen-br0  proto kernel  scope link  src 192.168.0.180
default via 192.168.0.50 dev xen-br0

atlas:~# ip addr
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
2: dummy0: <BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue
   link/ether 3e:8a:bf:84:29:10 brd ff:ff:ff:ff:ff:ff
   inet x.x.123.4/26 brd x.x.123.63 scope global dummy0
3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
   link/ether 00:50:bf:d6:57:38 brd ff:ff:ff:ff:ff:ff
   inet 192.168.0.180/24 brd 192.168.0.255 scope global eth0
4: xen-br0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether 00:50:bf:d6:57:38 brd ff:ff:ff:ff:ff:ff
   inet 192.168.0.180/32 brd 192.168.0.255 scope global xen-br0
5: xen-loc: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether 3e:8a:bf:84:29:10 brd ff:ff:ff:ff:ff:ff
   inet x.x.123.4/32 brd x.x.123.63 scope global xen-loc
6: vif1.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
7: vif1.1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
   link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

atlas:~# brctl show
bridge name     bridge id               STP enabled     interfaces
xen-br0         8000.0050bfd65738       no              eth0
                                                       vif1.1
xen-loc         8000.3e8abf842910       no              dummy0
                                                       vif1.0

atlas:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
x.x.123.0    0.0.0.0         255.255.255.192 U     0      0        0 xen-loc
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 xen-br0
0.0.0.0         192.168.0.50    0.0.0.0         UG    0      0        0 xen-br0

atlas:~# brctl showmacs xen-br0
port no mac addr                is local?       ageing timer
 1     00:0e:50:3d:9d:17       no                87.86
 1     00:0e:a6:23:b9:ad       no                 0.00
 1     00:0e:a6:23:d4:4f       no               112.76
 1     00:50:bf:d6:57:38       yes                0.00
 2     aa:00:00:64:e1:d7       no               108.50
 2     fe:ff:ff:ff:ff:ff       yes                0.00

atlas:~# brctl showmacs xen-loc
port no mac addr                is local?       ageing timer
 1     3e:8a:bf:84:29:10       yes                0.00
 2     aa:00:00:64:e1:d7       no                78.24
 2     fe:ff:ff:ff:ff:ff       yes                0.00


atlas:~# ifconfig
dummy0    Link encap:Ethernet  HWaddr 3E:8A:BF:84:29:10
         inet addr:x.x.123.4  Bcast:x.x.123.63  Mask:255.255.255.192
         UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 b)  TX bytes:1876 (1.8 KiB)

eth0      Link encap:Ethernet  HWaddr 00:50:BF:D6:57:38
         inet addr:192.168.0.180  Bcast:192.168.0.255  Mask:255.255.255.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:159 errors:0 dropped:0 overruns:0 frame:0
         TX packets:207 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:16966 (16.5 KiB)  TX bytes:19507 (19.0 KiB)
         Interrupt:9 Base address:0xa400

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:828 errors:0 dropped:0 overruns:0 frame:0
         TX packets:828 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:47336 (46.2 KiB)  TX bytes:47336 (46.2 KiB)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:42 errors:0 dropped:0 overruns:0 frame:0
         TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:1876 (1.8 KiB)  TX bytes:280 (280.0 b)

vif1.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3 errors:0 dropped:0 overruns:0 frame:0
         TX packets:6 errors:0 dropped:1 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:182 (182.0 b)  TX bytes:326 (326.0 b)

xen-br0   Link encap:Ethernet  HWaddr 00:50:BF:D6:57:38
         inet addr:192.168.0.180  Bcast:192.168.0.255  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:149 errors:0 dropped:0 overruns:0 frame:0
         TX packets:199 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:13668 (13.3 KiB)  TX bytes:18129 (17.7 KiB)

xen-loc   Link encap:Ethernet  HWaddr 3E:8A:BF:84:29:10
         inet addr:x.x.123.4  Bcast:x.x.123.63  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:42 errors:0 dropped:0 overruns:0 frame:0
         TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:1288 (1.2 KiB)  TX bytes:280 (280.0 b)

atlas:~# ip maddr
2:      dummy0
       link  01:00:5e:00:00:01
3:      eth0
       link  01:00:5e:00:00:01
4:      xen-br0
       link  01:00:5e:00:00:01
5:      xen-loc
       link  01:00:5e:00:00:01

----------------------------------------------------------------------
tcpdumps with 2 running vifs. All packets travel the right way
except on the way back from xen-br0 to eth0. There it failes.
----------------------------------------------------------------------

running ping 192.168.0.181 from external box (winxp)

atlas:~# tcpdump 'icmp or arp' -n -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
18:48:28.875891 arp who-has 192.168.0.181 tell 192.168.0.52
18:48:28.877460 arp reply 192.168.0.181 is-at aa:00:00:64:e1:d7
18:48:28.877506 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 22272
18:48:34.117703 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 22528
18:48:34.117955 arp who-has 192.168.0.52 tell 192.168.0.181
18:48:34.118032 arp reply 192.168.0.52 is-at 00:0e:a6:23:b9:ad
18:48:39.586215 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 22784
18:48:45.039124 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 23040


atlas:~# tcpdump 'icmp or arp' -n -i xen-br0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xen-br0, link-type EN10MB (Ethernet), capture size 96 bytes
18:49:41.755115 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 23296
18:49:41.755388 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 23296
18:49:44.308907 arp who-has 192.168.0.50 tell 192.168.0.53
18:49:46.754424 arp who-has 192.168.0.52 tell 192.168.0.181
18:49:46.754548 arp reply 192.168.0.52 is-at 00:0e:a6:23:b9:ad
18:49:47.005364 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 23552
18:49:47.005502 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 23552
18:49:52.442648 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 23808
18:49:52.442881 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 23808
18:49:57.911188 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 24064
18:49:57.911416 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 24064
18:50:01.313560 arp who-has 192.168.0.29 tell 192.168.0.29


tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
18:50:32.891190 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 24576
18:50:32.891465 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 24576
18:50:38.222152 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 24832
18:50:38.222344 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 24832
18:50:43.721882 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 25088
18:50:43.722039 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 25088
18:50:49.174790 IP 192.168.0.52 > 192.168.0.181: icmp 40: echo request seq 25344
18:50:49.174960 IP 192.168.0.181 > 192.168.0.52: icmp 40: echo reply seq 25344


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.