I was able to get the 5.10 version of the driver to install. Is there any reason we shouldn't just run this older version?

Hello,

I'm installing MLNX_OFED_LINUX-3.1-1.0.3-rhel7.2-x86_64 on CentOS 7.2 machine.

Initially I used MLNX_OFED_LINUX-3.3-1.0.4.0-rhel7.2-x86_64 but it's conflicted with current kernel (3.10.0-327.22.2.el7.x86_64), so I'm using 3.1-1.0.3 version.

(1)

As you can see the following log ((e)&(f)), current firmware version is v2.33.5000 and v2.35.5100 is required.
How can I upgrade firmware of this ConnectX-3 card?

Using "PSID: HP_0280210019", I can't find the proper firmware from the Mellanox website.

Is there any way to get the right firmware for CentOS7.2 (3.10.0-327.22.2.el7.x86_64)?

(2)

I'm installing same OFED on two machines with same environments (same ConnectX-3 card, cables, and OS).

In the (e) log, at least, each VPI is recognized as "UP 1X QDR (Ethernet)".

But, in the (f) log, all VPI are shown as "DOWN (Ethernet)".

Is there anything I should check?

Thank you in advance for any help!

-----

(a)

# cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

# uname -a

Linux xxx 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

(b)

// I'm using only the first port among 2 ports.

# cat /etc/modprobe.d/mlx4_core.conf

options mlx4_core port_type_array=2,2 num_vfs=8,0,0 probe_vf=8,0,0  log_num_mgm_entry_size=-1

(c)

# ./mlnxofedinstall --enable-sriov

...
The 3.10.0-327.22.2.el7.x86_64 kernel is installed, MLNX_OFED does not have drivers available for this kernel.
You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED package with drivers for this kernel.

Installation finished successfully.

Preparing... ################################# [100%]
Updating / installing...
   1:mlnx-fw-updater-3.1-1.0.3 ################################# [100%]

Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf

Attempting to perform Firmware update...
Note: Will burn SR-IOV enabled firmware only for ConnctX3 devices.
To configure SR-IOV for other devices, please use the 'mlxconfig' or 'mstconfig' utility.

The firmware for this device is not distributed inside Mellanox driver: 21:00.0 (PSID: HP_0280210019)
To obtain firmware for this device, please contact your HW vendor.

Failed to update Firmware.
See /tmp/MLNX_OFED_LINUX-3.1-1.0.3.23105.logs/fw_update.log

WARNING: Original /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsave
To load the new driver, run:
/etc/init.d/openibd restart

(d)

# lspci -nn | grep Mella

21:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]

21:00.1 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.2 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.3 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.4 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.5 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.6 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:00.7 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

21:01.0 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

(e)

# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 9

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... MLNX_OFED_LINUX-3.1-1.0.3 (OFED-3.1-1.0.3): modules

Host Driver RPM Check .................. PASS

Firmware on CA #0 VPI .................. v2.33.5000

Firmware Check on CA #0 (VPI) .......... FAIL

    REASON: mismatch CA #0 firmware detected (found v2.33.5000, required v2.35.5100)

Firmware on CA #1 VPI .................. v2.33.5000

Firmware Check on CA #1 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #2 VPI .................. v2.33.5000

Firmware Check on CA #2 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #3 VPI .................. v2.33.5000

Firmware Check on CA #3 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #4 VPI .................. v2.33.5000

Firmware Check on CA #4 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #5 VPI .................. v2.33.5000

Firmware Check on CA #5 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #6 VPI .................. v2.33.5000

Firmware Check on CA #6 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #7 VPI .................. v2.33.5000

Firmware Check on CA #7 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #8 VPI .................. v2.33.5000

Firmware Check on CA #8 (VPI) .......... NA

    REASON: NO required fw version

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 9

Port State of Port #1 on CA #0 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #2 on CA #0 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #1 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #2 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #3 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #4 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #5 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #6 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #7 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #1 on CA #8 (VPI)..... UP 1X QDR (Ethernet)

Error Counter Check on CA #0 (VPI)...... NA (Eth ports)

Error Counter Check on CA #1 (VPI)...... PASS

Error Counter Check on CA #2 (VPI)...... PASS

Error Counter Check on CA #3 (VPI)...... PASS

Error Counter Check on CA #4 (VPI)...... PASS

Error Counter Check on CA #5 (VPI)...... PASS

Error Counter Check on CA #6 (VPI)...... PASS

Error Counter Check on CA #7 (VPI)...... PASS

Error Counter Check on CA #8 (VPI)...... PASS

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (VPI) ............... 00:02:c9:03:00:f6:ef:f0

Node GUID on CA #1 (VPI) ............... 00:14:05:00:d3:a2:8c:35

Node GUID on CA #2 (VPI) ............... 00:14:05:00:e5:c0:e2:a1

Node GUID on CA #3 (VPI) ............... 00:14:05:00:d5:e0:c2:1a

Node GUID on CA #4 (VPI) ............... 00:14:05:00:ca:34:61:56

Node GUID on CA #5 (VPI) ............... 00:14:05:00:08:0d:34:72

Node GUID on CA #6 (VPI) ............... 00:14:05:00:ba:43:bd:b4

Node GUID on CA #7 (VPI) ............... 00:14:05:00:d6:4f:7f:80

Node GUID on CA #8 (VPI) ............... 00:14:05:00:2a:aa:5f:08

------------------ DONE ---------------------

(f)

# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 9

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... MLNX_OFED_LINUX-3.1-1.0.3 (OFED-3.1-1.0.3): modules

Host Driver RPM Check .................. PASS

Firmware on CA #0 VPI .................. v2.33.5000

Firmware Check on CA #0 (VPI) .......... FAIL

    REASON: mismatch CA #0 firmware detected (found v2.33.5000, required v2.35.5100)

Firmware on CA #1 VPI .................. v2.33.5000

Firmware Check on CA #1 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #2 VPI .................. v2.33.5000

Firmware Check on CA #2 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #3 VPI .................. v2.33.5000

Firmware Check on CA #3 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #4 VPI .................. v2.33.5000

Firmware Check on CA #4 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #5 VPI .................. v2.33.5000

Firmware Check on CA #5 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #6 VPI .................. v2.33.5000

Firmware Check on CA #6 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #7 VPI .................. v2.33.5000

Firmware Check on CA #7 (VPI) .......... NA

    REASON: NO required fw version

Firmware on CA #8 VPI .................. v2.33.5000

Firmware Check on CA #8 (VPI) .......... NA

    REASON: NO required fw version

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 1

Port State of Port #1 on CA #0 (VPI)..... UP 1X QDR (Ethernet)

Port State of Port #2 on CA #0 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #1 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #2 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #3 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #4 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #5 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #6 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #7 (VPI)..... DOWN (Ethernet)

Port State of Port #1 on CA #8 (VPI)..... DOWN (Ethernet)

Error Counter Check on CA #0 (VPI)...... NA (Eth ports)

Error Counter Check on CA #1 (VPI)...... PASS

Error Counter Check on CA #2 (VPI)...... PASS

Error Counter Check on CA #3 (VPI)...... PASS

Error Counter Check on CA #4 (VPI)...... PASS

Error Counter Check on CA #5 (VPI)...... PASS

Error Counter Check on CA #6 (VPI)...... PASS

Error Counter Check on CA #7 (VPI)...... PASS

Error Counter Check on CA #8 (VPI)...... PASS

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (VPI) ............... 24:be:05:ff:ff:82:04:70

Node GUID on CA #1 (VPI) ............... 00:14:05:00:74:9d:d3:e8

Node GUID on CA #2 (VPI) ............... 00:14:05:00:5b:d6:c8:5c

Node GUID on CA #3 (VPI) ............... 00:14:05:00:91:15:4c:6f

Node GUID on CA #4 (VPI) ............... 00:14:05:00:f4:f1:03:95

Node GUID on CA #5 (VPI) ............... 00:14:05:00:01:b5:af:22

Node GUID on CA #6 (VPI) ............... 00:14:05:00:d4:12:58:1e

Node GUID on CA #7 (VPI) ............... 00:14:05:00:d4:4c:ca:e8

Node GUID on CA #8 (VPI) ............... 00:14:05:00:72:14:6c:52

------------------ DONE ---------------------

Hi Serge,

They're not exactly the same, libvma is pulling packets by simulating a network stack which offloads traffic from kernel, DPDK accelerates kernel traffic by going directly to the HW (using the PMD driver). in the end both products are utilizing fast access to the ConnectX.HW buffers

Different flows of traffic in an application might be better on one or the other - I suggest you check what is the best flow for you

Hello,

1. The firmware is per PSID and not per OS so if your PSID is HP_0280210019 you should contact HP for the firmware bin file since the card was manufactured by HP.

2. hca_self_test is not the right tool to check the link state and speed, please use ibstat or ibv_devinfo instead.

ethtool <interface> will also work.

You can check the interfaces that were created by the driver by invoking ibdev2netdev.

Best Regards,

Viki

Hi Michael,

You could have a corrupted partition.Try booting to the other partition during the boot process there is an option "Press Enter to boot this image, or 'Ctrl B' for boot menu" this allows manual intervention of the boot process so that you can decide which partition to boot from.

Thanks a lot, Viki!

1. Right. PSID is shown as HP's one. I wanted to check whether there is any compatible OFED with this Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]. If no, let me contact to HP.

2. Actually I tried ibstat also but it didn't show the needed firmware version, so I wanted to show that information using hca_self_test. This ibstat result doesn't complain the firmware version even though hca_self_test described "(found v2.33.5000, required v2.35.5100)". Is there no problem with v2.33.5000?

# ibstat

CA 'mlx4_0'

  CA type: MT4099

  Number of ports: 2

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x0002c90300f6eff0

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x0202c9fffef6eff0

  Link layer: Ethernet

  Port 2:

  State: Down

  Physical state: Disabled

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x0202c9fffef6eff1

  Link layer: Ethernet

CA 'mlx4_1'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500d3a28c35

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0xcc0567fffed08d89

  Link layer: Ethernet

CA 'mlx4_2'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500e5c0e2a1

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x04a4dffffe9554b4

  Link layer: Ethernet

CA 'mlx4_3'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500d5e0c21a

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x78a965fffe2cd6c6

  Link layer: Ethernet

CA 'mlx4_4'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500ca346156

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0xa0550afffeeb939f

  Link layer: Ethernet

CA 'mlx4_5'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500080d3472

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x70e0bffffe624948

  Link layer: Ethernet

CA 'mlx4_6'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500ba43bdb4

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0xd8ff4bfffe554d38

  Link layer: Ethernet

CA 'mlx4_7'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x00140500d64f7f80

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0x70ca2afffe3c7237

  Link layer: Ethernet

CA 'mlx4_8'

  CA type: MT4100

  Number of ports: 1

  Firmware version: 2.33.5000

  Hardware version: 1

  Node GUID: 0x001405002aaa5f08

  System image GUID: 0x0002c90300f6eff3

  Port 1:

  State: Active

  Physical state: LinkUp

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x0c010000

  Port GUID: 0xa4ba1afffe657a61

  Link layer: Ethernet

# ibv_devinfo

hca_id: mlx4_8

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:2aaa:5f08

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_7

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:d64f:7f80

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_6

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:ba43:bdb4

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_5

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:080d:3472

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_4

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:ca34:6156

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_3

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:d5e0:c21a

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_2

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:e5c0:e2a1

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_1

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0014:0500:d3a2:8c35

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4100

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 1

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

hca_id: mlx4_0

  transport: InfiniBand (0)

  fw_ver: 2.33.5000

  node_guid: 0002:c903:00f6:eff0

  sys_image_guid: 0002:c903:00f6:eff3

  vendor_id: 0x02c9

  vendor_part_id: 4099

  hw_ver: 0x1

  board_id: HP_0280210019

  phys_port_cnt: 2

  port: 1

  state: PORT_ACTIVE (4)

  max_mtu: 4096 (5)

  active_mtu: 4096 (5)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

  port: 2

  state: PORT_DOWN (1)

  max_mtu: 4096 (5)

  active_mtu: 1024 (3)

  sm_lid: 0

  port_lid: 0

  port_lmc: 0x00

  link_layer: Ethernet

Tried that, the other partition doesn't boot at all. Same result.

Did you also try a "cold boot" (remove power for at least 1min) and see if it recovers.If not it would need a RMA.

Of course I did. Unfortunately, RMA is not really an option for me, as I bought this switch off eBay. It was working fine before I attempted that flash... Can someone from Mellanox help, please? Maybe something I can flash from the uboot prompt?

Hi Stanley,

The applicable FW depends on the PSID/BoardID of the HCA card.

Have you installed the MFT (Mellanox FW Tool) on the ESXI server?

If so, to get the PSID of the HCA card:

#/opt/mellanox/bin/mst start

#/opt/mellanox/bin/mst status -v

#/opt/mellanox/bin/flint -d <mst device> q

You will see a field label as PSID.

If you could please provide in order to point you to the correct FW.

If you do not have the MFT tool installed, you can download it from the following link:

http://www.mellanox.com/page/management_tools

NOTE: Make sure you download both "mst" & "mft" vibs and choose the accurate kernel (vmk or native).

Sophie.

Hi Atul,

What is the amount of memory on the server & client?

Also, 3.3.1.0.0.0 was removed from our website and replaced with 3.3.1.0.4.0; can you try with this driver version?

If you just validate the RDMA connectivity between the server & client (NFS aside), are you getting the same messages?

Sophie.

Good start for user space application is perftest suite that is opensource project: git.openfabrics.org Git - ~grockah/perftest.git/summary . It is verbs API based application and should work with Infiniband or RoCE . Check ib_write_bw, ib_send_lat, etc.

For kernel space, you can take a look on the iser  - Linux SCSI Target - ISCSI Extensions for RDMA

Hello Kevin,

1. We do not have control over posting of our products on Amazon, eBay...etc.

    You are right.  MCX413A-BCAT is "ConnectX-4 EN network interface card, 40GbE single-port QSFP"

2.  The simple difference: ConnectX4-EN is 100GbE Ethernet Adapter while is ConnectX-4 Lx EN is provides 1/10/25/40/50 gigabit speeds.

If you need to know more technical details please see the product briefs for each cards family.

Product Briefs:

ConnectX-4 EN: http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_EN_Card.pdf

ConnectX-4 Lx EN: http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_Lx-EN_Card.pdf

Cheers,

~R

Try approaching to Microsoft to ask for wdk for win2012r2

I see in their web site that last DDK version was for win2003

and last WDK (windows driver kit) version is for windows 2008 sp1

Windows Driver Kit - Wikipedia, the free encyclopedia 

As fro Network Direct - it is part of Mellanox WinOF driver

Can be downloaded from:

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

Hi Michael,

no way from u-boot.

what exactly did you did to go into this state?

what are the commands that you ran?

Thanks for your response. I am able to write user-space code but there is no direct mapping from user-space verbs to kernel-space verbs. I am stuck on writing kernel space code rdma-read and rdma-write. I was wondering if there is any working example of kernel space verbs on both the remote machine and local machine.

My card supports 64k queue pairs but i will be able to create only 16 connections per RoCE interface in windows 2012R2 OS(limitation of windows).Is there any way to

exhaust maximum number of queue pairs on that adapter in windows.

The last column is the most important as it shows the typical value, and these values are close.  One of the hosts, probably, has something different in the settings  -BIOS, OS tuning, some daemon running, management interrupts and it cause to spikes in some iterations.

Try to run loopback tests.

I haven't done anything special, just attempted to upgrade the firmware via SSH/CLI. It got stuck during the flash process and, after waiting a while, I rebooted it using the reset button and it didn't come back from the reboot. It took me some time to find the right cable to connect to the serial interface and only then I've discovered what happened.

What is strange though that it gets to the U-boot prompt, which after some reading here, I think should have been password protected. So I can see and execute all the U-boot commands, which makes me think there is some hope it can be revived. Can someone, please, tell me what to do? I'm pretty sure I should be able to flash all the necessary bits and pieces, I only need to get them and the exact instructions what to do.

Thanks heaps!

Hi, Sophie, thank you for reply: noticed that you are referring to MOFED 3.3-1.0.4.0 -- while problem what observed on MOFED-3.3-1.0.0.0 -- upgraded the OFED distro and NFSoRDMA works fine on ConnectX-4 Lx now.

Original problem was observed on CentOS 7.2, 3.10.0-327.el7.x86_64, OFED 3.3-1.0.0.0 & ConnectX-4 LX NICs (server & client identical systems).

Thanks & Regards,

   Philip Romanov