|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] using ipoib with xcp
Hello, I have been playing with the XCP for a while now, and must say I'm very exited about the technology. I had no prior experience with Xen so it has taken me a while to understand the concepts, but now I feel most important issues are solved and I've purchased some hardware to build my (tiny) cloud on.
The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet and 1 x Infiniband (IB) NICs per node. I want to use the IB NIC to provide fast connectivity between the domUs, while the Ethernet NICs will be used for the XCP management interface and ISP connectivity.
I have successfully built OFED 1.5.1 in the XCP DDK VM and installed OFED in the XPC 0.1.1 dom0. From there I can bring up the IB network, but I'm having problems getting this to work properly within XCP virtual machines. This is what happens:
Starting out I have 2 nodes in a pool; both are clean with only lo, eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands to add the IB NICs to the pool: xe pif-scan host-uuid=NODE1
xe pif-plug uuid=NODE1_IB0 xe pif-scan host-uuid=NODE2 xe pif-plug uuid=NODE2_IB0 As expected this adds ib0/brib0 on both nodes and a single pool-wide network, but there is no connectivity between the hosts after I give brib0 an IP:
xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 mode=static xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 mode=static ping 10.1.2.2 --> reply
ping 10.1.2.3 --> destination host unavailable However if I also give ib0 an IP and use this as gateway for brib0, connectivity is achieved: ifconfig ib0 10.1.2.22 netmask 255.255.255.0
xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 gateway=10.1.2.22 mode=static ifconfig ib0 10.1.2.33 netmask 255.255.255.0 xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 gateway=10.1.2.33 mode=static
ping 10.1.2.2 --> reply ping 10.1.2.3 --> reply ping 10.1.2.22 --> reply ping 10.1.2.33 --> reply This is very well, but when I add a VIF on the IB network to a VM it is not able to communicate through it:
xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE1_IBVM ifconfig eth2 10.1.2.122 netmask 255.255.255.0 xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE2_IBVM
ifconfig eth2 10.1.2.133 netmask 255.255.255.0 ping 10.1.2.122 --> reply ping 10.1.2.22 --> destination host unavailable ping 10.1.2.2 --> destination host unavailable
ping 10.1.2.133 --> destination host unavailable ping 10.1.2.33 --> destination host unavailable ping 10.1.2.3 --> destination host unavailable
I believe that the problem lies somewhere in the routing table configuration. This setup gives the following routing table: 10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0
10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0 If I delete and then add the brib0 route, the route order is changed: 10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0
10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0 Using this the VM can talk to the host (and visa versa), but hot across the network. Connectivity between ib0/brib0 over the network is also broken.
I've also noticed that the same MAC is added to /etc/ovs-vswitchd.conf multiple times for brib0: bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25 bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25 I've tried removing some of these but that does not seem to have any effect. My experience with IP routing and especially vswitch is limited and I'm not sure what to try from here. I've tried various configurations but no luck so far.
Note that I'm testing with 2 XCP nodes configured in a pool. I've also checked that the PIFs are in the same order on both nodes (the reference mentions this). The MTU (1500) of brib0 differs from that of ib0 (2044), but changing this does not solve the problem.
Any help is much appreciated. Thanks! Trygve _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |