[Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU directly

[Alexander Petry <petry@xxxxxxxxxxxxxxxxxx>]

Hi list!

I would like to access my InfiniBand HCA from a domU, so I first tried 
the smart-io.hg Xen tree but it did not build right away. As a temporary 
workaround I tried to let the domU directly access the HCA. The steps I 
have taken have been:
1. In dom0:

# lspci -v

03:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx 
HCA] (rev a0)
       Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
       Flags: fast devsel, IRQ 16
       Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
       Memory at f0000000 (64-bit, prefetchable) [size=8M]
       Capabilities: [40] Power Management version 2
       Capabilities: [48] Vital Product Data
       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/5 Enable-
       Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
       Capabilities: [60] Express Endpoint IRQ 0

With that information I edited my grub config to exclude this particular 
PCI device:
module /boot/vmlinuz-2.6.19-4-server root=/dev/sda1 ro console=tty0 
console=ttyS0,9600n8 max_loop=64 pciback.hide=(0000:03:00.0)
2. After a reboot, I edited the domU config so that it includes the line:

pci = ['0000:03:00.0']

The output of lspci -v executed in the user domain is:

0000:00:00.0 InfiniBand: Mellanox Technology: Unknown device 6274 (rev a0)
       Subsystem: Mellanox Technology: Unknown device 6274
       Flags: fast devsel, IRQ 16
       Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
       Memory at f0000000 (64-bit, prefetchable) [size=8M]
       Capabilities: [40] Power Management version 2
       Capabilities: [48] Vital Product Data
       Capabilities: [90] Message Signalled Interrupts: 64bit+ 
Queue=0/5 Enable-
       Capabilities: [84] #11 [001f]
       Capabilities: [60] #10 [0001]

The module cannot be loaded and prints the following error:

xen-domU0:~# modprobe ib_mthca
[90558.241638] ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February 
14, 2006)
[90558.241651] ib_mthca: Initializing 0000:00:00.0
[90559.245730] ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
[90559.245792] ib_mthca: probe of 0000:00:00.0 failed with error -11


The output of 'xm info' is:

host                   : xen-o-matic
release                : 2.6.19-4-server
version                : #2 SMP Thu Apr 5 06:07:34 UTC 2007
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2009
hw_caps                : 
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory           : 2030
free_memory            : 1367
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-0
xen_caps               : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by          : buildd
cc_compile_domain      : buildd
cc_compile_date        : Sat Mar 24 16:01:08 UTC 2007
xend_config_format     : 2


Does anybody have an idea?

Best regards,
   Alexander Petry

--
===========================================
| Dipl. Inf. Alexander Petry              |
| Head Competence Center for HPC          |
| Institut fuer Techno- und               |
| Wirtschaftsmathematik (ITWM)            |
| Fraunhofer-Platz 1                      |
|                                         |
| D-67663 Kaiserslautern                  |
===========================================
| E-Mail:   petry@xxxxxxxxxxx             |
| Internet: http://www.itwm.fhg.de        |
===========================================



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users