Hi,

starting with your switches, I am not sure what SX5012 is. Maybe you were referring to SX6012?

Connecting things with the HCA you were thinking of using, keep in mind that the InfiniBand technology is currently at FDR rates (56Gb/sec). the HCA you specified is cable of this speed so you probably need an FDR switch to complete the set.

to your specific questions:

1. Are there any other components required to do the above?

Nop. basic network would need NICs, cables and switches. everything else is SW

2. Can the NICs be used to connect servers peer to peer without the switch?

Yes. Back2Back works.

3. Is there a limit on how many NIC can be placed in a server(besides physical)

No. as many PCI-e slots you can have. Keep in mind that the PCI slot you need is a high-speed one. typically PCI-e Gen3 x8 or x16

4. If the cables are not included can point me to the correct cables

Cables - Mellanox Technologies

5. Can the SX5012 Switches be linked together via on board ports?

No on-board ports. for connecting between two or more switches you can use any switch port.

6. Is the management software included with the switch or does this need to be purchased separately?

Mellanox carries Managed switches which include all the network management necessary tools (included in the price of the switch) and some other switches are none-managed (have no on-board CPU) and therefore not cables of manage the network but can be managed using an external Subnet Manager.

I hope i could help. anyway, I recommend you touch base with a reseller or Mellanox sales rep for the final touch ups.

Good luck..

Hi Phil,

You can get all BGP routes with the "show ip bgp" command

Hi,

To use ibdump on ROCE, flow steering must be enabled :

in /etc/modprobe.d/mlnx.conf :

mlx4_core log_num_mgm_entry_size=-1

restart driver

After you set that I recommend you upgrade your driver to the latest 2.3-2. if that doesn't help, please try the version I attached.

I think tcpdump is also a good option, depends on what you're trying to filter.

I might as well continue this adventure.

So after looking in the RPMS dir of the installer I ran:

[root@<blah> RPMS]# rpm -ivh opensm-libs-4.2.5.MLNX20140828.7f05469-0.1.x86_64.rpm

warning: opensm-libs-4.2.5.MLNX20140828.7f05469-0.1.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY

Preparing...                ########################################### [100%]

   1:opensm-libs            ########################################### [100%]

Preparing...                ########################################### [100%]

   1:opensm-devel           ########################################### [100%]

But this did not bring the joy I had hoped for:

error: Failed build dependencies:

        opensm-devel is needed by ibutils-1.5.7.1-0.10.g385f871.x86_64

        libibumad-devel is needed by ibutils-1.5.7.1-0.10.g385f871.x86_64

Installing /root/mellanox/MLNX_OFED_LINUX-2.3-2.0.0-rhel6.6-x86_64/src/MLNX_OFED_SRC-2.3-2.0.0/SRPMS/ibutils-1.5.7.1-0.10.g385f871.src.rpm

Everytime......

So I went back to the factory method for installing. Like so:

# ./mlnxofedinstall

Logs dir: /tmp/MLNX_OFED_LINUX-2.3-2.0.0.26272.logs

Error: One or more required packages for installing MLNX_OFED are missing.

Please install the missing packages using your Linux distribution Package Management tool.

Run:

yum install gcc-gfortran

Looks like I missed someting above

[root@<blah> MLNX_OFED_LINUX-2.3-2.0.0-rhel6.6-x86_64]# yum install gcc-gfortran

Loaded plugins: dellsysid, product-id, rhnplugin, security, subscription-manager

This system is receiving updates from RHN Classic or RHN Satellite.

Setting up Install Process

dell-omsa-indep                                                                                                                                                             | 1.9 kB     00:00

dell-omsa-specific                                                                                                                                                          | 1.9 kB     00:00

rhel-x86_64-server-6                                                                                                                                                        | 1.8 kB     00:00

rhel-x86_64-server-extras-6                                                                                                                                                 | 1.6 kB     00:00

rhel-x86_64-server-optional-6                                                                                                                                               | 1.8 kB     00:00

rhel-x86_64-server-rh-common-6                                                                                                                                              | 1.6 kB     00:00

rhel-x86_64-server-supplementary-6                                                                                                                                          | 1.8 kB     00:00

Resolving Dependencies

--> Running transaction check

---> Package gcc-gfortran.x86_64 0:4.4.7-11.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================

Package                                       Arch                                    Version                                         Repository                                             Size

===================================================================================================================================================================================================

Installing:

gcc-gfortran                                  x86_64                                  4.4.7-11.el6                                    rhel-x86_64-server-6                                  4.7 M

Transaction Summary

===================================================================================================================================================================================================

Install       1 Package(s)

Total download size: 4.7 M

Installed size: 0

Is this ok [y/N]: y

Downloading Packages:

gcc-gfortran-4.4.7-11.el6.x86_64.rpm                                                                                                                                        | 4.7 MB     00:07

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing : gcc-gfortran-4.4.7-11.el6.x86_64                                                                                                                                                1/1

  Verifying  : gcc-gfortran-4.4.7-11.el6.x86_64                                                                                                                                                1/1

Installed:

  gcc-gfortran.x86_64 0:4.4.7-11.el6

Complete!

Now to try again!

[root@<blah> MLNX_OFED_LINUX-2.3-2.0.0-rhel6.6-x86_64]# ./mlnxofedinstall

Logs dir: /tmp/MLNX_OFED_LINUX-2.3-2.0.0.27854.logs

This program will install the MLNX_OFED_LINUX package on your machine.

Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.

Do you want to continue?[y/N]:y

Starting MLNX_OFED_LINUX-2.3-2.0.0 installation ...

Installing mlnx-ofa_kernel RPM

Preparing...                ##################################################

mlnx-ofa_kernel             ##################################################

Installing kmod-mlnx-ofa_kernel 2.3 RPM

Preparing...                ##################################################

kmod-mlnx-ofa_kernel        ##################################################

Installing mlnx-ofa_kernel-devel RPM

Preparing...                ##################################################

mlnx-ofa_kernel-devel       ##################################################

Installing kmod-kernel-mft-mlnx 3.7.1 RPM

Preparing...                ##################################################

kmod-kernel-mft-mlnx        ##################################################

Installing knem-mlnx RPM

Preparing...                ##################################################

knem-mlnx                   ##################################################

Installing kmod-knem-mlnx 1.1.1.90mlnx RPM

Preparing...                ##################################################

kmod-knem-mlnx              ##################################################

Installing kmod-iser 1.4.5 RPM

Preparing...                ##################################################

kmod-iser                   ##################################################

Installing kmod-srp 1.3.3 RPM

Preparing...                ##################################################

kmod-srp                    ##################################################

Installing mpi-selector RPM

Preparing...                ##################################################

mpi-selector                ##################################################

Installing user level RPMs:

Preparing...                ##################################################

ofed-scripts                ##################################################

Preparing...                ##################################################

libibverbs                  ##################################################

Preparing...                ##################################################

libibverbs-devel            ##################################################

Preparing...                ##################################################

libibverbs-devel-static     ##################################################

Preparing...                ##################################################

libibverbs-utils            ##################################################

Preparing...                ##################################################

libmlx4                     ##################################################

Preparing...                ##################################################

libmlx4-devel               ##################################################

Preparing...                ##################################################

libmlx5                     ##################################################

Preparing...                ##################################################

libmlx5-devel               ##################################################

Preparing...                ##################################################

libibcm                     ##################################################

Preparing...                ##################################################

libibcm-devel               ##################################################

Preparing...                ##################################################

libibumad                   ##################################################

Preparing...                ##################################################

libibumad-devel             ##################################################

Preparing...                ##################################################

libibumad-static            ##################################################

Preparing...                ##################################################

libibmad                    ##################################################

Preparing...                ##################################################

libibmad-devel              ##################################################

Preparing...                ##################################################

libibmad-static             ##################################################

Preparing...                ##################################################

ibsim                       ##################################################

Preparing...                ##################################################

ibacm                       ##################################################

Preparing...                ##################################################

librdmacm                   ##################################################

Preparing...                ##################################################

librdmacm-utils             ##################################################

Preparing...                ##################################################

librdmacm-devel             ##################################################

Preparing...                ##################################################

opensm-libs                 ##################################################

Preparing...                ##################################################

opensm                      ##################################################

Preparing...                ##################################################

opensm-devel                ##################################################

Preparing...                ##################################################

opensm-static               ##################################################

Preparing...                ##################################################

dapl                        ##################################################

Preparing...                ##################################################

dapl-devel                  ##################################################

Preparing...                ##################################################

dapl-devel-static           ##################################################

Preparing...                ##################################################

dapl-utils                  ##################################################

Preparing...                ##################################################

perftest                    ##################################################

Preparing...                ##################################################

mstflint                    ##################################################

Preparing...                ##################################################

mft                         ##################################################

Preparing...                ##################################################

srptools                    ##################################################

Preparing...                ##################################################

rds-tools                   ##################################################

Preparing...                ##################################################

rds-devel                   ##################################################

Preparing...                ##################################################

ibutils2                    ##################################################

Preparing...                ##################################################

ibutils                     ##################################################

Preparing...                ##################################################

cc_mgr                      ##################################################

Preparing...                ##################################################

dump_pr                     ##################################################

Preparing...                ##################################################

ar_mgr                      ##################################################

Preparing...                ##################################################

ibdump                      ##################################################

Preparing...                ##################################################

infiniband-diags            ##################################################

Preparing...                ##################################################

infiniband-diags-compat     ##################################################

Preparing...                ##################################################

qperf                       ##################################################

Preparing...                ##################################################

fca                         ##################################################

Preparing...                ##################################################

mxm                         ##################################################

Preparing...                ##################################################

openmpi                     ##################################################

Preparing...                ##################################################

bupc                        ##################################################

Preparing...                ##################################################

infinipath-psm              ##################################################

Preparing...                ##################################################

infinipath-psm-devel        ##################################################

Preparing...                ##################################################

mvapich2                    ##################################################

Preparing...                ##################################################

hcoll                       ##################################################

Preparing...                ##################################################

libibprof                   ##################################################

Preparing...                ##################################################

mlnxofed-docs               ##################################################

Preparing...                ##################################################

mpitests_mvapich2__2_0      ##################################################

Preparing...                ##################################################

mpitests_openmpi__1_8_4rc1  ##################################################

Device (42:00.0):

        42:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

        Link Width: 8x

        PCI Link Speed: Unknown

Installation finished successfully.

Attempting to perform Firmware update...

Querying Mellanox devices firmware ...

Device #1:

----------

  Device Type:      ConnectX3

  Part Number:      MCX312A-XCB_A2-A5

  Description:      ConnectX-3 EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6

  PSID:             MT_1080120023

  PCI Device Name:  0000:42:00.0

  Port1 MAC:        f452141fd030

  Port2 MAC:        f452141fd031

  Versions:         Current        Available

     FW             2.11.0500      2.32.5100

     PXE            3.4.0000       3.4.0306

  Status:           Update required

---------

Found 1 device(s) requiring firmware update...

Device #1: Updating FW ... Done

Restart needed for updates to take effect.

Log File: /tmp/MLNX_OFED_LINUX-2.3-2.0.0.27854.logs/fw_update.log

Please reboot your system for the changes to take effect.

Configuring /etc/security/limits.conf.

This looks like a good thing - reboot the machine!

# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... MLNX_OFED_LINUX-2.3-2.0.0 (OFED-2.3-2.0.0): 2.6.32-504.el6.x86_64

Host Driver RPM Check .................. PASS

Firmware on CA #0 NIC .................. v2.32.5100

Firmware Check on CA #0 (NIC) .......... PASS

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 0

Port State of Port #1 on CA #0 (NIC)..... DOWN (Ethernet)

Port State of Port #2 on CA #0 (NIC)..... DOWN (Ethernet)

Error Counter Check on CA #0 (NIC)...... NA (Eth ports)

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (NIC) ............... f4:52:14:03:00:1f:d0:30

------------------ DONE ---------------------

Hello Amirul,

In order to maximize performance I would suggest to do the following:

1. Make sure nodes and switches are running with the latest and greatest FW/driver versions.

2. Tune the system to get a better performance. Please refer to our performance tuning guide:

http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf

In case you still observe a poor BW and you are a contracted customer,  you may open a case with Mellanox support and we will be glad to assist.

Kind Regards,

Einav Ezra

Hi Amirul,

Are the Mellanox adapters located on Pcie Gen3 X8 slot?

Can you provide the output of "lspci -d 15b3: -xxx -vvv" from both adapters?

Okay, figured out myself - this is expected. VF driver won't replace the MS virtual nic. In case a live migration happens and the target machine doesn't have SRIOV, the nic will still function, just without the VF.

Do I need to downgrade the cards from 2.9.1050 to 2.9.1000 for CentOS 6.6?

Thanks in advance.  I have > 80 cards so not having to flash these would help immensely.

gnushell

Hi,

I am trying to install MLNX_OFED_LINUX-2.3-2.0.0 on Ubuntu 14.04 x86_64, kernel 3.13.0-24-generic. The error I receive is:

MLNX_OFED_LINUX-2.3-2.0.0-ubuntu14.04-x86_64$ sudo ./mlnxofedinstall

....

Installing new packages

Installing ofed-scripts-2.3...

Installing mlnx-ofed-kernel-utils-2.3...

Installing mlnx-ofed-kernel-dkms-2.3...

Error: mlnx-ofed-kernel-dkms installation failed!

See:

        /tmp/OFED.18777.logs/mlnx-ofed-kernel-dkms.debinstall.log

MLNX_OFED_LINUX-2.3-2.0.0-ubuntu14.04-x86_64$

There is an error in the log file:

MLNX_OFED_LINUX-2.3-2.0.0-ubuntu14.04-x86_64$ cat /tmp/OFED.18777.logs/mlnx-ofed-kernel-dkms.debinstall.log

.....

Building module:

cleaning build area....

make KERNELRELEASE=3.13.0-24-generic -j12....

Error!  Build of ib_core.ko failed for: 3.13.0-24-generic (x86_64)

Consult the make.log in the build directory

/var/lib/dkms/mlnx-ofed-kernel/2.3/build/ for more information.

See logs attached.

Any idea what is wrong with the installation?

Thanks

I wouldn't recommend, these versions have nothing major that justifies downgrading.

No problem,

it should be there in the Mellanox.com website under Archvies :

http://www.mellanox.com/page/software_overview_ib

Please note you will have to compile it with mlnx_add_kernel_support script

Hi all,

I have been trying to to capture MPI traffic using ibdump for one way latency measurements (at the physical layer).

But all ibdump captures are IPoIB packets interleaved with Subnet monitoring packets such as perfqueries from the UFM/SM sweeps.

Specs:

ibdump, 2.0.0-5, built on Aug 21 2013, 13:09:58. GIT Version: 41a4dd2

I have also tried: ibdump, 2.0.0-8, built on Apr 28 2014, 17:06:51. GIT Version: f8e989d

I have also used --mem-mode and --max-burst to prevent packet dropping but with no luck.

Ofed_info: MLNX_OFED_LINUX-2.0-3.0.0.3 (OFED-2.0-3.0.0)

bash-4.1$ ibv_devinfo

hca_id:    mlx4_0

    transport:            InfiniBand (0)

    fw_ver:                2.30.3200

    node_guid:            0002:c903:0044:e220

    sys_image_guid:            0002:c903:0044:e223

    vendor_id:            0x02c9

    vendor_part_id:            4099

    hw_ver:                0x1

    board_id:            IBM1100110019

    phys_port_cnt:            1

    port:    1

    state:            PORT_ACTIVE (4)

    max_mtu:        4096 (5)

    active_mtu:        4096 (5)

    sm_lid:            1

    port_lid:        15

    port_lmc:        0x00

    link_layer:        InfiniBand

Using tshark -r sniffer.pcap| grep -i infiniband to parse the .pcap files, I typically get this:

cat /etc/modprobe.d/mlnx.conf:

blacklist mlx4_core

blacklist mlx4_en

blacklist mlx5_core

blacklist mlx5_ib

I would highly appreciate anyone who can help understand this?

Thanks,

Fabrice

Thanks!

got 2.2 compiled without any problems!

Hello,

I have a 4036E switch and I am trying to update the software.  Here is the system info:

The health monitor shows:

The update software chassis gives:

with each .fpf being identical to the one on the flash.  The programing finishes successfully and reboots:

during the load everything looks the same as other switches except:

But when it finishes loading it is right back where it started:

Any ideas about what is going on here?

Thank you,

Kevin

Attempting to setup RoCE with windows server 2012. I need to verify the SMB direct/RDMA is working end to end. I am seeing RDMA traffic on windows perf counters, but I am getting Responder QCE errors in Mellanox Diagostic perf counter on both SMB server and SMB client.

SMB Multichannell does not appear to be working.Can anyone shed light on what might be wrong?

Thank you for your time.

It seems like your compilation of the OFED kernel modules failed. Can you provide make.log from /var/lib/dkms/mlnx-ofed-kernel/2.3/build/ ?

Thanks,

--Shachar

Sure, please see attached

Seems like your configure script for the modules were run with incorrect parameters (i.e. ./configure without any parameters, which results in nothing being built). Did you attempt any manual reconfiguring/rebuilding? If not, than we have some issue with the install script. Will ask the relevant engineer about this tomorrow.

no, i did no installation customization or changes to the environment.

I opened tgz and ran "sudo ./mlnxofedinstall".