Hello-

I have a number of older servers with ConnectX-2 VPI IB cards running a very old OS (Ubuntu 10.04). We want to take them to Ubuntu 18.04 instead. As I understand it, Mellanox is only providing Ubuntu 18.04 installers for OFED v 4.x, however support for ConnectX-2 cards was dropped in this version of OFED. So it seems I need to get OFED 3.x installed on Ubuntu 18. Has anyone had any success with that, or could please offer some advice? Or is there any hope an official installer could be provided? I see OFED 3 is supported on Ubuntu 16, but it'd be great if we could use the newer OS for 2 years more longevity. Thanks for any advice!

-Lewis

Hello everybody,

I have an issue in my NetworkDirect RDMA application when loading the Mellanox NDv2 provider. It seems as the newer WinOF-2 driver for MLX-5 IB adapters (mlx5nd.dll) requires that the connecting process has adminstrator privileges.

Because when running my application with normal user privileges, I get an error0x80070005 (Access denied), whereas this has never been an issue with the older WinOF driver for MLX-4 adapters (mlx4nd.dll).

Here the failing code sequence from my ndhelper.cpp:

static HMODULE      g_hProvider  = NULL;

static IND2Provider g_pIProvider = NULL;

static HRESULT LoadProvider( __in WSAPROTOCOL_INFOW* pProtocol )

{

    WCHAR* pPath = ::GetProviderPath( pProtocol ); // %SystemRoot%\System32\mlx5nd.dll

     g_hProvider = ::LoadLibraryW( pPath );

    ::HeapFree( ::GetProcessHeap(), 0, pPath );

    DLLGETCLASSOBJECT pfnDllGetClassObject = reinterpret_cast<DLLGETCLASSOBJECT>(

        ::GetProcAddress( g_hProvider, "DllGetClassObject" )

    );

    DLLGETCLASSOBJECT pfnDllCanUnloadNow = reinterpret_cast<DLLCANUNLOADNOW>(

        ::GetProcAddress(g_hProvider, "DllCanUnloadNow")

    );

    IClassFactory* pClassFactory;

    HRESULT hr = pfnDllGetClassObject(

        pProtocol->ProviderId,

        IID_IClassFactory,

        reinterpret_cast<void**>(&pClassFactory)

    );

    if (g_pIProvider == NULL) {

        hr = pClassFactory->CreateInstance(

            NULL,

            IID_IND2Provider,

            reinterpret_cast<void**>(&g_pIProvider)

        );

        if (FAILED(hr)) {

            TRACE("ClassFactory->CreateInstance(IID_IND2Provider) failed with error 0x%08X", hr); // Without having admin rights, always ending up here!

            g_pIProvider = NULL;

        }

        pClassFactory->Release();

    }

}

Unfortunately, just giving my process admin privileges is not an option for me. So I would appreciate if someone has an idea how to overcome this issue.

Perhaps some tuning of security configuration via dcomcnfg or the like(?)

BTW: The Mellanox-provided tools nd_read_bw.exe and nd_write_bw.exe have the same behavior:

• Those from WinOF driver 5.35 can run WITH or WITHOUT admin privileges.
• Those from WinOF-2 driver 1.80 can run only WITH admin privileges.

My system configuration:

• HPE DL380Gen10
• MLX-5 EDR Infiniband HCA (HPEIB EDR/EN 100Gb 2P 840QSFP28)
• Windows Server 2012 R2
• Mellanox WinOF-2 version 1.80

Thanks and Regards

Environment:

• Ubuntu 16.04, Kernel 4.15.0-29-generic
• MLNX_OFED_LINUX-4.3-1.0.1.0:
• MCX416A-CCAT

Configured PFC with VLAN 100 and Priority 3. While trying to enable the interface, I get:

root@sm16:~# nmcli con up ens1f0.100

Error: Connection activation failed: Failed to find a compatible device for this connection

Any pointers?

This is how the configuration of the tagged and untagged interfaces look like:

root@sm16:~# nmcli c s

NAME        UUID                                  TYPE            DEVICE

ens1f0      e5251358-dffb-4e07-b72f-b9e93ca6eca8  802-3-ethernet  --

ens1f0.100  10c8d003-164a-4a8d-a21f-ff6bb712a090  vlan            --

# nmcli con s ens1f0.100 | less

connection.id:                          ens1f0.100

connection.uuid:                        10c8d003-164a-4a8d-a21f-ff6bb712a090

connection.interface-name:              ens1f0

connection.type:                        vlan

connection.autoconnect:                 yes

connection.autoconnect-priority:        0

connection.timestamp:                   0

connection.read-only:                   no

connection.permissions:

connection.zone:                        --

connection.master:                      --

connection.slave-type:                  --

connection.autoconnect-slaves:          -1 (default)

connection.secondaries:

connection.gateway-ping-timeout:        0

connection.metered:                     unknown

connection.lldp:                        -1 (default)

802-3-ethernet.port:                    --

802-3-ethernet.speed:                   0

802-3-ethernet.duplex:                  --

802-3-ethernet.auto-negotiate:          yes

802-3-ethernet.mac-address:             --

802-3-ethernet.cloned-mac-address:      --

802-3-ethernet.mac-address-blacklist:

802-3-ethernet.mtu:                     4200

802-3-ethernet.s390-subchannels:

802-3-ethernet.s390-nettype:            --

802-3-ethernet.s390-options:

802-3-ethernet.wake-on-lan:             1 (default)

802-3-ethernet.wake-on-lan-password:    --

ipv4.method:                            manual

ipv4.dns:

ipv4.dns-search:

ipv4.dns-options:                       (default)

ipv4.dns-priority:                      0

ipv4.addresses:                         10.0.20.75/24

ipv4.gateway:                           --

ipv4.routes:

ipv4.route-metric:                      -1

ipv4.ignore-auto-routes:                no

ipv4.ignore-auto-dns:                   no

ipv4.dhcp-client-id:                    --

ipv4.dhcp-timeout:                      0

ipv4.dhcp-send-hostname:                yes

ipv4.dhcp-hostname:                     --

ipv4.dhcp-fqdn:                         --

ipv4.never-default:                     no

ipv4.never-default:                     no

ipv4.may-fail:                          yes

ipv4.dad-timeout:                       -1 (default)

ipv6.method:                            auto

ipv6.dns:

ipv6.dns-search:

ipv6.dns-options:                       (default)

ipv6.dns-priority:                      0

ipv6.addresses:

ipv6.gateway:                           --

ipv6.routes:

ipv6.route-metric:                      -1

ipv6.ignore-auto-routes:                no

ipv6.ignore-auto-dns:                   no

ipv6.never-default:                     no

ipv6.may-fail:                          yes

ipv6.ip6-privacy:                       -1 (unknown)

ipv6.addr-gen-mode:                     stable-privacy

ipv6.dhcp-send-hostname:                yes

ipv6.dhcp-hostname:                     --

vlan.parent:                            ens1f0

vlan.id:                                100

vlan.flags:                             1 (REORDER_HEADERS)

vlan.ingress-priority-map:

vlan.egress-priority-map:               0:3,1:3,2:3,3:3,4:3,5:3,6:3,7:3

# nmcli con s ens1f0| less

connection.id:                          ens1f0

connection.uuid:                        e5251358-dffb-4e07-b72f-b9e93ca6eca8

connection.interface-name:              ens1f0

connection.type:                        802-3-ethernet

connection.autoconnect:                 yes

connection.autoconnect-priority:        0

connection.timestamp:                   0

connection.read-only:                   no

connection.permissions:

connection.zone:                        --

connection.master:                      --

connection.slave-type:                  --

connection.autoconnect-slaves:          -1 (default)

connection.secondaries:

connection.gateway-ping-timeout:        0

connection.metered:                     unknown

connection.lldp:                        -1 (default)

802-3-ethernet.port:                    --

802-3-ethernet.speed:                   0

802-3-ethernet.duplex:                  --

802-3-ethernet.auto-negotiate:          yes

802-3-ethernet.mac-address:             --

802-3-ethernet.cloned-mac-address:      --

802-3-ethernet.mac-address-blacklist:

802-3-ethernet.mtu:                     4200

802-3-ethernet.s390-subchannels:

802-3-ethernet.s390-nettype:            --

802-3-ethernet.s390-options:

802-3-ethernet.wake-on-lan:             1 (default)

802-3-ethernet.wake-on-lan-password:    --

ipv4.method:                            auto

ipv4.dns:

ipv4.dns-search:

ipv4.dns-options:                       (default)

ipv4.dns-priority:                      0

ipv4.addresses:

ipv4.gateway:                           --

ipv4.routes:

ipv4.route-metric:                      -1

ipv4.ignore-auto-routes:                no

ipv4.ignore-auto-dns:                   no

ipv4.dhcp-client-id:                    --

ipv4.dhcp-timeout:                      0

ipv4.dhcp-send-hostname:                yes

ipv4.dhcp-hostname:                     --

ipv4.dhcp-fqdn:                         --

ipv4.never-default:                     no

ipv4.may-fail:                          yes

ipv4.dad-timeout:                       -1 (default)

ipv6.method:                            auto

ipv6.dns:

ipv6.dns-search:

ipv6.dns-options:                       (default)

ipv6.dns-priority:                      0

ipv6.addresses:

ipv6.gateway:                           --

ipv6.routes:

ipv6.route-metric:                      -1

ipv6.ignore-auto-routes:                no

ipv6.ignore-auto-dns:                   no

ipv6.never-default:                     no

ipv6.may-fail:                          yes

ipv6.ip6-privacy:                       -1 (unknown)

ipv6.addr-gen-mode:                     stable-privacy

ipv6.dhcp-send-hostname:                yes

ipv6.dhcp-hostname:                     --

Hi Ryan,

I am getting the same end result with RHEL 7.5, Kernel 4.9, ethtool version 4.8 for Connectx-4 & Connectx-3.

I am however able to retrieve the information on a Connectx-5. 

You can download the MFT package and use the "mlxcables" utility.

http://www.mellanox.com/page/management_tools

Note: see section mst cable add first.

Note: Latest FW for that HCA card is 12.23.1020.

Sophie.

Hi Suraj,

Thank you for posting your question on the Mellanox Community.

Based on your information provided (We also noticed your posting on https://mailman.stanford.edu/pipermail/mininet-discuss/2018-August/008031.html), currently we do hot have support for running Soft-RoCE on a Mininet topology. Currently we only provided a method of running from a nic-2-nic.

Also please note that Soft-RoCE is still in BETA

We would recommend to post this question on the Mininet Mailings-list as you already did. Maybe somebody from the mailings-list has a solution for this as they need to implement this into the Mininet Framework

Thanks and regards,
~Mellanox Technical Support

Hi Kenneth,

Thank you for posting your question on the Mellanox Community.

We have noticed that you also opened a Mellanox Support case regarding this issue and that we provided you access to the PRM's of the ConnectX-5

If you need anything, please do not hesitate to open a new support case by sending an email to support@mellanox.com

Thanks and regards,
~Mellanox Technical Support

Is there any error messages? Does ibtracert work (#ibtracert <src lid> <dst lid>?

Hi Brian,

When using virtualization, GRH (global routing header) must be present in the packet. For ibping, --dgid <GID> parameter need to be used (see man ibping).

To get GIDs, on the server run 'show_gids' and use the output on the client side

Server

#show_gids

DEV     PORT    INDEX   GID                                     IPv4            VER     DEV

---     ----    -----   ---                                     ------------    ---     ---

mlx5_1  1       0       fe80:0000:0000:0000:248a:0703:009c:01a7                 v1

Client

#ibping --dgid  fe80:0000:0000:0000:248a:0703:009c:01a7 18

If you like to check RDMA connectivity between VMs, use utilities from perftest package (ib_read_bw, ib_write_bw, etc) with -R parameter.

Could you tell me the detail for your application ? why you need such feature ?

Hello.

We have a plan to connect Cisco Catalyst 3850 switch's 1Gbps SFP port to  Mellanox SN2100-BB2F  QSFP port.

For this connection, we are going to connect them using NAM1Q00A-QSA and  MC3208011-SX( SFP transiever).

I am not sure whether it will work fine or not.

I have read about QSA article and it says as follow, But I don't know the meaning of "to plug another SFP+ to SFP into the QSA".

       " It is less common but possible to plug another SFP+ to SFP (10GbE to 1GbE) into the QSA and to reduce the port speed even to 1GbE.

          Again, if it is a Mellanox Ethernet switch port, you must configure that manually.

          For example, assuming the QSA was plugged into Ethernet switch port 1/1 and another SFP+ to SFP adapter was plugged into the QSA to reach 1GbE, run:

             switch (config) # interface ethernet 1/1 speed 1000

             switch (config) # configuration write

 

 

Question)  If we plug MC3208011-SX into  NAM1Q00A-QSA and configure "speed 1000" at that interface ,  will it works fine ? ( = we don't need something needed to be purchased more.)

                     or do we need to purchase something like a speed convert adapter also ?

Thank you for your attention.

Hi,

I use connect-3-pro cards on hp machines to boot via ipxe but getting attached errors. Apparently, the iPXE does not support connect-3-pro cards. just wondering is there any chance to add ipxe support for the cards.

Thanks,

Bal

If using HPC-X or OpenMPI from the Mellanox OFED, you might add '-mca btl self -mca pml ucx'

Hello

Recently delivered sb7800 to our customers.

The customer requested that the SB7800 OS be installed with the most stable version of OS and wants to be recommended by the vendor.

Please advise me of the most stable os of the sb7800 devices that have come to date.

Thank you.

It is found that when configuring the adapter as RoCEv2 by “options mlx4_core roce_mode=0000:81:00.0-1,0000:84:00.0-1”, after OS rebooting, one MT4103 can not work well. But there is nothing wrong for another one with the same firmware.

The port state always shows “Disabled" as below:

# ibstat mlx4_1

CA ‘mlx4_1’

  CA type: MT4103

  Number of ports: 1

  Firmware version: 2.42.5000

  Hardware version: 0

  Node GUID: 0xe41d2d030029ecb0

  System image GUID: 0xe41d2d030029ecb0

  Port 1:

  State: Down

  Physical state: Disabled

  Rate: 10

  Base lid: 0

  LMC: 0

  SM lid: 0

  Capability mask: 0x04010000

  Port GUID: 0xe61d2dfffe29ecb0

  Link layer: Ethernet

And related ethnet interface’s link can not be detectable, but actually the cable is connected.

# ethtool ens259

Settings for ens259:

  Supported ports: [ FIBRE ]

  Supported link modes:   1000baseKX/Full

  Supported pause frame use: Symmetric Receive-only

  Supports auto-negotiation: No

  Advertised link modes:  1000baseKX/Full

  Advertised pause frame use: Symmetric

  Advertised auto-negotiation: No

  Speed: Unknown!

  Duplex: Unknown! (255)

  Port: FIBRE

  PHYAD: 0

  Transceiver: internal

  Auto-negotiation: off

  Supports Wake-on: d

  Wake-on: d

  Current message level: 0x00000014 (20)

         link ifdown

  Link detected: no

I have tried OFED driver of MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.4-x86_64 and MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.4-x86_64. Neither can work.

The switch is Dell 4048-on.

So, what is the problem with one MT4103? How can I enable the physical port?

SFPcables.com is an official store of 10Gtek company. It offers one-stop Datacenter solution and products, including compatible Transceivers, DAC cables, AOC cables, Fiber Optical, Patch-cord, CWDM/DWDM passive/active systems, Network Adapters and more. As a commitment to offering high-quality, reliable and customized products, SFPcables.com develops and tests new products to meet the growing Data Centers and High Performance Computing market needs. We guarantee only premium materials and quality engineering are used in the designing and manufacturing for the fiber and cooper products of SFPcables.com and the whole production process are under the highest quality control.
In addition to our extensive product line, our customer service has earned us a reputation of trust that is unparalleled in the industry. SFPcables.com holds thousands of cables and transceivers in stock to ensure 24-hour shipping for all orders.

 

SFPcables.com invests heavily in technology, with a Compatibility Test Lab full of the latest brands of equipment, SFPcables.com ensures its precise programming for various brands of switches, servers and routers.
We help clients navigate the complexities of their hardware architecture to guarantee compliance throughout the network.
Our service success is proven by the loyalty and support of some of the largest data centers in the world, including:telecommunications, corporations, government agencies and reputable distributors.

we have a feedback for our customer’s support. as you see on the pic.

up to $200 off !!!

Take more save more, sfpcables is a shop store of 10Gtek. we always have exhabition all over the world, all our products are with Certification. 100% high quality guaranteed.

 

Our special sell:

100G QSFP28 (EDR) DAC Cable

FDR 56GBASE-SR4 QSFP+ Transceiver

56Gb/s QSFP+ FDR DAC Cable

Just some due diligence here.

We put our ConnectX5 cards in our 3 host vmware 6.5 stack, and did not get it to work with host_chaining. We ended up contacting support about it, and the reply we got wasn't optimistic.

"Host-chaining is currently not supported as it is not GA for ESXi."

So my previous post was a grain of salt, and marked out accordingly.

I have yet to see *any* documentation on host_chaining specifically; which is really sad, since As far as I know, my post above is the best available.

You're welcome!

I'm glad I helped someone after all the headache I went through for it.

I have no hard experience with VMWare, and so take all of this with a grain of salt.

First thought is vlan tags. I was told that VMWare tags by default.

From my (limited) understanding and thoughts, host chaining inside VMware is not a good idea.

If you setup a virtual switch (on the vmware side) and put both ports of the card on the switch, give that switch an IP, that would allow for vmotion and such over the link at close to line speed. Letting the switch (analogous to openvswitch) do all of the routing, and fast pathing.

Thoughts - If there was host chaining:

Vmware still sees both ports (we can't assign IPs to raw port interfaces to start with.)

It doesn't really know which port to send out, so it could take the extra hop before it gets to the destination.

Three node, desired going from A -> B might take the path of A -> C -> B

Where I can talk is non-chaining speed.

We did try using openswitch and the cards with chaining off. So long as the stp stuff is turned on; we got nearly line speed.

We opened a support ticket for our problems with MTU. It took a while, but we found the problem.

They have a nice little utility (sysinfo-snapshot) for seeing the card internals and OS config options which helped us (by looking through it.)

See my post below. Host_chaining is not supported on ESXi at this time.

How can I enable "packet pacing" on the connectX-5 ? With the command ibv_devinfo -v I get Packet_pacing_caps: qp_rate_limits_max: 0kbps I expect here the wire rate of the card.

Thanks Patrick

Thank you for responding quickly.

I am able to ibping to the gid on first dev but not on the second one:

SERVER:

-----------------------

# show_gids

DEV     PORT    INDEX   GID                                     IPv4            VER     DEV

---     ----    -----   ---                                     ------------    ---     ---

mlx5_0  1       0       fe80:0000:0000:0000:248a:0703:0014:f9ac                 v1

mlx5_1  1       0       fe80:0000:0000:0000:248a:0703:0014:f850                 v1

n_gids_found=2

CLIENT:

name@server:/etc/infiniband$ ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f9ac 8

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.109 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.095 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.139 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.174 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.159 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.190 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.169 ms

Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.163 ms

^Z[6]   Killed                  ibping 8

[7]   Killed                  ibping -S

[8]+  Stopped                 ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f9ac 8

name@server:/etc/infiniband$ ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f850 8

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)

^Z

[9]+  Stopped                 ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f850 8

How can ibping the other gids?

Thanks

Brian