I've finally got to spend some time looking further into this.
I now believe the underlaying problem is that Open vSwitch is unable to connect the brib0 bridge interface to the ib0 physical interface. I suspect the cause of this to be the long MAC address of the Infiniband NICs, but so far I have not found a workaround for the issue.
These are the relevant devices for my setup:
[root@hypoxcp1 ~]# ifconfig
brib0 Link encap:Ethernet HWaddr 80:00:00:48:FE:80
inet addr:10.1.2.2 Bcast:10.1.2.255 Mask:255.255.255.0
inet6 addr: fe80::8200:ff:fe48:fe80/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:720 (720.0 b)
eth0 Link encap:Ethernet HWaddr 00:30:48:CC:5C:A4
inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:31755 errors:0 dropped:0 overruns:0 frame:0
TX packets:10544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4284224 (4.0 MiB) TX bytes:1433336 (1.3 MiB)
ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:10.1.2.102 Bcast:10.1.2.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:2044 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:128
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
xenbr0 Link encap:Ethernet HWaddr 00:30:48:CC:5C:A4
inet addr:10.1.1.2 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:25892 errors:0 dropped:0 overruns:0 frame:0
TX packets:10538 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3586527 (3.4 MiB) TX bytes:1432868 (1.3 MiB)
The ifconfig command reports the wrong (or truncated) MAC address for the ib0 device. The real address can be found using other commands:
[root@hypoxcp1 ~]# cat /sys/class/net/ib0/address
80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
[root@hypoxcp1 ~]# ip link show ib0
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 128
link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
As mentioned earlier in this thread I've had issues with duplicate MAC addresses in /etc/ovs-vswitchd.conf, but a clean install somehow fixed that issue, so the proper MAC address is now added to the file:
[root@hypoxcp1 ~]# cat /etc/ovs-vswitchd.conf
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.port=brib0
bridge.brib0.port=ib0
bridge.brib0.port=vif1.2
bridge.brib0.xs-network-uuids=6455dd7f-4a61-43b8-a49d-656f749c4ac6
bridge.xenbr0.mac=00:30:48:cc:5c:a4
bridge.xenbr0.port=eth0
bridge.xenbr0.port=vif1.1
bridge.xenbr0.port=xenbr0
bridge.xenbr0.xs-network-uuids=528d85a4-f582-c181-54eb-acf09ac7dcf4
bridge.xenbr1.mac=00:30:48:cc:5c:a5
bridge.xenbr1.port=eth1
bridge.xenbr1.port=vif1.0
bridge.xenbr1.port=xenbr1
bridge.xenbr1.xs-network-uuids=4f033ff5-5a56-629c-1c27-0765ba7c03bb
I'm no expert on XCP and Open vSwitch, but I believe it works something like this:
- XAPI writes /etc/ovs-vswitchd.conf based on the XCP DB
- XAPI starts up Open vSwitch
- Open vSwitch creates the interfaces defined in /etc/ovs-vswitchd.conf
To me it seems like the MAC address for the brib0 interface is truncated, and I believe this causes Open vSwitch to not bind brib0 and ib0 together:
[root@hypoxcp1 ~]# ovs-ofctl show brib0
Apr 14 15:38:31|00001|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt
features_reply (xid=0x6bb27f3f): ver:0x97, dpid:32f493d6e290
n_tables:2, n_buffers:256
features: capabilities:0x17, actions:0x3ff
LOCAL(brib0): addr:80:00:00:48:fe:80, config: 0, state:0
Apr 14 15:38:31|00002|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt
get_config_reply (xid=0x9b99aaf1): miss_send_len=0
[root@hypoxcp1 ~]# ovs-ofctl show xenbr0
Apr 14 15:38:19|00001|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt
features_reply (xid=0x836b0867): ver:0x97, dpid:f68bde598f51
n_tables:2, n_buffers:256
features: capabilities:0x17, actions:0x3ff
1(eth0): addr:00:30:48:cc:5c:a4, config: 0, state:0
current: 1GB-FD COPPER AUTO_NEG
advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG
supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG
LOCAL(xenbr0): addr:00:30:48:cc:5c:a4, config: 0, state:0
Apr 14 15:38:19|00002|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt
get_config_reply (xid=0x2b665ea): miss_send_len=0
As you see the binding to ib0 is missing, and the MAC of brib0 is different from that in /etc/ovs-vswitchd.conf.
As previously stated I can communicate between XCP hosts on both brib0 and ib0 using this setup. The problem is that VIFs on the brib0 network are not reachable. I have the following IB interfaces on a single host:
From within the VM that uses vif1.3 I try to ping brib0 and ib0 and watch the traffic on the XCP host:
[root@hypoxcp1 ~]# tcpdump -i vif1.2
tcpdump: WARNING: vif1.2: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.2, link-type EN10MB (Ethernet), capture size 96 bytes
15:54:59.948660 arp who-has 10.1.2.102 tell 10.1.2.202
15:55:00.948643 arp who-has 10.1.2.102 tell 10.1.2.202
15:55:01.948645 arp who-has 10.1.2.102 tell 10.1.2.202
[root@hypoxcp1 ~]# tcpdump -i brib0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on brib0, link-type EN10MB (Ethernet), capture size 96 bytes
15:54:22.612723 arp who-has 10.1.2.102 tell 10.1.2.202
15:54:23.612643 arp who-has 10.1.2.102 tell 10.1.2.202
15:54:24.612642 arp who-has 10.1.2.102 tell 10.1.2.202
[root@hypoxcp1 ~]# tcpdump -i ib0
tcpdump: WARNING: arptype 32 not supported by libpcap - falling back to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ib0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
The packets never reach ib0.
This setup adds the follow IP routes to the XCP host:
[root@hypoxcp1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr0
10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0
10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 brib0
0.0.0.0 10.1.1.1 0.0.0.0 UG 0 0 0 xenbr0
If I remove the ib0 route I can talk to brib0 and ib0 from vif1.3, but only on the same physical machine. Inter-host and inter-vm over network communication breaks without that route.
I also tried using "bridge" networking instead of "vswitch", but the system behaves the same way AFAICT, though the configuration is of course different.
I'm not sure what to try next. I could use the IB network for the management interface and not run any VMs on it, but please let me know if you have any idea what's wrong.
Thanks!
Trygve
On Thu, Apr 8, 2010 at 11:40 AM, Trygve Sanne Hardersen
<trygve@xxxxxxxxxxxxx> wrote:
Hi
Yes, I believe the packets are lost between brib0 and ib0, so they are never sent across the network but it works on a single host.
I'll do some more testing and let you know what I find.
Thanks!
TrygveOn Thu, Apr 8, 2010 at 10:40 AM, Dave Scott
<Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi,
Is it true that you have managed to get VM <-> Host
connectivity working but not VM <-> VM (across host) connectivity
working?
If so then it would be interesting to use something like tcpdump
to find out where the packets are going missing. If they’re entering the
vswitch and then getting lost then it would be worth talking about this on the
openvswitch mailing list.
Another possibility is to revert to non-vswitch based networking
in dom0: try writing “bridge” to /etc/xensource/network.conf and
rebooting.
Cheers,
Dave
Hello,
I have been playing with the XCP for a while now, and must
say I'm very exited about the technology. I had no prior experience with Xen so
it has taken me a while to understand the concepts, but now I feel
most important issues are solved and I've purchased some hardware to build my
(tiny) cloud on.
The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet
and 1 x Infiniband (IB) NICs per node. I want to use the IB NIC to provide fast
connectivity between the domUs, while the Ethernet NICs will be used for the
XCP management interface and ISP connectivity.
I have successfully built OFED 1.5.1 in the XCP DDK VM and
installed OFED in the XPC 0.1.1 dom0. From there I can bring up the
IB network, but I'm having problems getting this to work properly within XCP
virtual machines. This is what happens:
Starting out I have 2 nodes in a pool; both are clean with
only lo, eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands
to add the IB NICs to the pool:
xe pif-scan host-uuid=NODE1
xe pif-plug uuid=NODE1_IB0
xe pif-scan host-uuid=NODE2
xe pif-plug uuid=NODE2_IB0
As expected this adds ib0/brib0 on both nodes and a single
pool-wide network, but there is no connectivity between the hosts after I give
brib0 an IP:
xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2
netmask=255.255.255.0 mode=static
xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3
netmask=255.255.255.0 mode=static
ping 10.1.2.3 --> destination host unavailable
However if I also give ib0 an IP and use this as gateway for
brib0, connectivity is achieved:
ifconfig ib0 10.1.2.22 netmask 255.255.255.0
xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2
netmask=255.255.255.0 gateway=10.1.2.22 mode=static
ifconfig ib0 10.1.2.33 netmask 255.255.255.0
xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3
netmask=255.255.255.0 gateway=10.1.2.33 mode=static
This is very well, but when I add a VIF on the IB network to
a VM it is not able to communicate through it:
xe vif-create device=2 mac=random network-uuid=IB_NET
vm-uuid=NODE1_IBVM
ifconfig eth2 10.1.2.122 netmask 255.255.255.0
xe vif-create device=2 mac=random network-uuid=IB_NET
vm-uuid=NODE2_IBVM
ifconfig eth2 10.1.2.133 netmask 255.255.255.0
ping 10.1.2.122 --> reply
ping 10.1.2.22 --> destination host unavailable
ping 10.1.2.2 --> destination host unavailable
ping 10.1.2.133 --> destination host
unavailable
ping 10.1.2.33 --> destination host unavailable
ping 10.1.2.3 --> destination host unavailable
I believe that the problem lies somewhere in the routing
table configuration. This setup gives the following routing table:
10.1.2.0 0.0.0.0
255.255.255.0 U 0 0
0 ib0
10.1.2.0 0.0.0.0
255.255.255.0 U 0 0
0 brib0
If I delete and then add the brib0 route, the route order is
changed:
10.1.2.0 0.0.0.0
255.255.255.0 U 0 0
0 brib0
10.1.2.0 0.0.0.0
255.255.255.0 U 0 0
0 ib0
Using this the VM can talk to the host (and visa versa), but
hot across the network. Connectivity between ib0/brib0 over the network is also
broken.
I've also noticed that the same MAC is added to
/etc/ovs-vswitchd.conf multiple times for brib0:
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
I've tried removing some of these but that does not seem to
have any effect. My experience with IP routing and especially vswitch is
limited and I'm not sure what to try from here. I've tried various
configurations but no luck so far.
Note that I'm testing with 2 XCP nodes configured in a pool.
I've also checked that the PIFs are in the same order on both nodes (the
reference mentions this). The MTU (1500) of brib0 differs from that of ib0
(2044), but changing this does not solve the problem.
Any help is much appreciated. Thanks!
--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway
hypobytes.com
+47 40 55 30 25
--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway
hypobytes.com+47 40 55 30 25
--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway
hypobytes.com+47 40 55 30 25
