|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] Infiniband support (eIPoIB) in Xen 4.4 / Debian Jessie 8.2
Good Morning, We are experimenting some issues setting up Infiniband (eIPoIB) on Debian 8.2 hypervisor with Xen 4.4. The system is able to load the kernel module and to create the bridge as suggested by Debian. The VM is able to start and load the network configuration, but it's not able to communicate with other systems in that network. Can you see anything wrong in this configuration? Is there any bug that we are not aware of? Thanks! Jordi -------------------------------------------------------------------------------------------------- Software packages from Mellanox installed in the hypervisor : root@snow01(bcam):~# dpkg -l | grep -i MLNX ii dapl2-utils 2.1.5mlnx-OFED.3.0.0.3.3 amd64 Utilities for use with the DAPL libraries ii ibacm 1.0.9mlnx1-OFED.3.0.10.ga105e8b amd64 The ibacm daemon helps reduce the load of managing path record lookups on ii ibacm-dev 1.0.9mlnx1-OFED.3.0.10.ga105e8b amd64 Most applications do not need to know how to talk directly to the ibacm ii ibverbs-utils 1.1.8mlnx1-OFED.3.1.1.0.0 amd64 Examples for the libibverbs library ii infiniband-diags 1.6.5.MLNX20150902.0e83419 amd64 OpenFabrics Alliance InfiniBand Diagnostic Tools ii infiniband-diags-compat 1.6.5.MLNX20150902.0e83419 amd64 OpenFabrics Alliance InfiniBand Diagnostic Tools ii knem 1.1.2.90mlnx-OFED.3.1.0.0.7.1.g4b084fc amd64 KNEM is a Linux kernel module enabling high-performance intra-node MPI communication for large messages. KNEM offers support for asynchronous and vectorial data transfers as well as offloading memory copies on to Intel I/OAT hardware. See http://runtime.bordeaux.inria.fr/knem/ for details. ii knem-dkms 1.1.2.90mlnx-OFED.3.1.0.0.7.1.g4b084fc all DKMS support for mlnx-ofed kernel modules ii libdapl-dev 2.1.5mlnx-OFED.3.0.0.3.3 amd64 Development files for the DAPL libraries ii libdapl2 2.1.5mlnx-OFED.3.0.0.3.3 amd64 The Direct Access Programming Library (DAPL) ii libibcm-dev 1.0.5mlnx2-OFED.3.0.11.gd7d485d amd64 Development files for the libibcm library ii libibcm1 1.0.5mlnx2-OFED.3.0.11.gd7d485d amd64 Userspace InfiniBand Communication Manager. ii libibmad 1.3.11.MLNX20150329.b2cb00f amd64 OpenFabrics Alliance InfiniBand MAD library ii libibmad-devel 1.3.11.MLNX20150329.b2cb00f amd64 Development files for the libibmad library ii libibmad-static 1.3.11.MLNX20150329.b2cb00f amd64 Static version of the libibmad library ii libibumad 1.3.10.2.MLNX20150406.966500d amd64 OpenFabrics Alliance InfiniBand umad (user MAD) library ii libibumad-devel 1.3.10.2.MLNX20150406.966500d amd64 Development files for the libibumad library ii libibumad-static 1.3.10.2.MLNX20150406.966500d amd64 Static version of the libibumad library ii libibverbs-dev 1.1.8mlnx1-OFED.3.1.1.0.0 amd64 Development files for the libibverbs library ii libibverbs1 1.1.8mlnx1-OFED.3.1.1.0.0 amd64 Library for direct userspace use of RDMA (InfiniBand/iWARP) ii libibverbs1-dbg 1.1.8mlnx1-OFED.3.1.1.0.0 amd64 Debugging symbols for the libibverbs library ii libmlx4-1 1.0.6mlnx1-OFED.3.1.1.0.0 amd64 Userspace driver for Mellanox ConnectX InfiniBand HCAs ii libmlx4-1-dbg 1.0.6mlnx1-OFED.3.1.1.0.0 amd64 Debugging symbols for the libmlx4 driver ii libmlx4-dev 1.0.6mlnx1-OFED.3.1.1.0.0 amd64 Development files for the libmlx4 driver ii libmlx5-1 1.0.2mlnx1-OFED.3.1.1.0.3 amd64 Userspace driver for Mellanox ConnectX InfiniBand HCAs ii libmlx5-1-dbg 1.0.2mlnx1-OFED.3.1.1.0.3 amd64 Debugging symbols for the libmlx5 driver ii libmlx5-dev 1.0.2mlnx1-OFED.3.1.1.0.3 amd64 Development files for the libmlx5 driver ii libopensm 4.6.0.MLNX20150830.c69ebab amd64 Infiniband subnet manager libraries ii libopensm-devel 4.6.0.MLNX20150830.c69ebab amd64 Developement files for OpenSM ii librdmacm-dev 1.0.21mlnx-OFED.3.0.1.5.2 amd64 Development files for the librdmacm library ii librdmacm-utils 1.0.21mlnx-OFED.3.0.1.5.2 amd64 Examples for the librdmacm library ii librdmacm1 1.0.21mlnx-OFED.3.0.1.5.2 amd64 Userspace RDMA Connection Manager ii mlnx-ethtool 3.16-OFED.3.0.0.0.1 amd64 This utility allows querying and changing settings such as speed, ii mlnx-fw-updater 3.1-1.0.3 amd64 Mellanox Technologies Firmware update package ii mlnx-ofed-kernel-dkms 3.1-OFED.3.1.1.0.3.1.g9032737 all DKMS support for mlnx-ofed kernel modules ii mlnx-ofed-kernel-utils 3.1-OFED.3.1.1.0.3.1.g9032737 amd64 Userspace tools to restart and tune mlnx-ofed kernel modules ii ofed-scripts 3.1-OFED.3.1.1.0.3 amd64 MLNX_OFED utilities ii opensm 4.6.0.MLNX20150830.c69ebab amd64 An Infiniband subnet manager ii opensm-doc 4.6.0.MLNX20150830.c69ebab amd64 Documentation for opensm Content of openib.conf related with (e)IPoIB root@snow01:~# cat /etc/infiniband/openib.conf | grep -i ipoib # Load IPoIB IPOIB_LOAD=yes # Enable IPoIB Connected Mode SET_IPOIB_CM=auto # Load E_IPoIB E_IPOIB_LOAD=yes kernel modules loaded root@snow01:~# lsmod | grep eth_ipoib eth_ipoib 48004 0 mlx_compat 12495 18 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,eth_ipoib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib Status of the IB HCA root@snow01:~# ibstat |grep State State: Active Information regarding the virtual ethernet root@snow01:~# ethtool -i eth4 driver: eth_ipoib version: 3.1-1.0.3 (29 Sep 2015) firmware-version: 1 bus-info: ib0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no list of the bridges we are working with root@snow01(bcam):~# brctl show bridge name bridge id STP enabled interfaces xenbr7 8000.40f2e9bddf23 no eth3 vif1.0 vif2.0 vif4.0 vif5.0 vif8.0 xib0 8000.e41d2db361d0 no eth4 xpub0 8000.40f2e9bddf20 no bond0 vif8.1 Content of /etc/network/interfaces related with Infiniband. auto ib0 allow-hotplug ib0 iface ib0 inet manual auto eth4 allow-hotplug eth4 iface eth4 inet manual auto xib0 iface xib0 inet static bridge_ports eth4 address 10.10.0.1 netmask 255.255.255.0 network 10.10.0.0 broadcast 10.10.0.255 Configuration file of the VM kernel = '/boot/vmlinuz-3.16.0-4-amd64' extra = 'elevator=noop' ramdisk = '/boot/initrd.img-3.16.0-4-amd64' vcpus = '4' memory = '8192' root = '/dev/xvda2 ro' disk = [ 'phy:/dev/snow_vg/login01-disk,xvda2,w', 'phy:/dev/snow_vg/login01-swap,xvda1,w', ] name = 'login01' vif = [ 'ip=192.168.7.5, mac=76:fd:31:9e:05:01, bridge=xenbr7', 'mac=76:fd:31:9e:05:02,bridge=xpub0', 'mac=76:fd:31:9e:05:03,bridge=xib0', ] _on_poweroff_ = 'destroy' on_reboot = 'restart' on_crash = 'restart' _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |