Hello,

My fiber ISP uses 2.5G SFP modules and I was wondering if there are any Mellanox ConnectX cards that support syncing at this speed over fiber?

Thanks.

Hi Mattias,

Thank you for posting your question on the Mellanox Community.

Based on the information provided, unfortunately we do not support the cable in the current line of adapters. From a previous post, we noticed that you are using a ConnectX-2, also this adapter is not supported anymore as it is EOL and EOS.

For supported cables, please refer to the firmware RN of the ConnectX-5 EN -> http://www.mellanox.com/pdf/firmware/ConnectX5-FW-16_23_1020-release_notes.pdf

Thanks and regards,
~Mellanox Technical Support

hi

I have exactly the same problem with my ConnectX3 cards that I got second hand. They report (on Centos 7.5):

[root@lair ~]# mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:

----------

Device type:    ConnectX3      

Device:         /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot

-E- Failed to query device current configuration

[root@lair ~]# flint -d /dev/mst/mt4099_pciconf0 q

Image type:            FS2

FW Version:            2.42.5000

FW Release Date:       5.9.2017

Product Version:       02.42.50.00

Rom Info:              type=PXE version=3.4.752

Device ID:             4099

Description:           Node             Port1            Port2            Sys image

GUIDs:                 0012312312312345 0012312312312346 0012312312312347 0012312312312348

MACs:                                       0002c9a166c0     0002c9a166c1

VSD:                  

PSID:                  MT_1170110023

while  mlxconfig reports:

[root@lair ~]# mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:

----------

Device type:    ConnectX3      

Device:         /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot

-E- Failed to query device current configuration

The cards work great, but I would like to activate sr-iov on some of them.

I have reflashed the cards and tried to reset the cards:

[root@lair ~]# mlxconfig -d /dev/mst/mt4099_pciconf0 reset

Reset configuration for device /dev/mst/mt4099_pciconf0?  ? (y/n) [n] : y

Applying... Done!

-I- Please reboot machine to load new configurations.

Nothing seems to allow me to read/write the configuration.

Is there anything I have missed?

On CentOS 7.5 + Mellanox ConnectX-4 LX having troubs to configure interface.

I have Mellanox ConnectX-4 LX ethernet adapter:

~]# lspci | grep -i Mella

10003:01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

10003:01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

But when mlx5_core module loads there is an error:

mlx5_core 10003:01:00.0: PCI INT A: no GSI

mlx5_core 10003:01:00.0: Missing registers BAR, aborting

mlx5_core 10003:01:00.0: error requesting BARs, aborting

mlx5_core 10003:01:00.0: mlx5_pci_init failed with error code -19

mlx5_core 10003:01:00.1: PCI INT B: no GSI

mlx5_core 10003:01:00.1: Missing registers BAR, aborting

mlx5_core 10003:01:00.1: error requesting BARs, aborting

mlx5_core 10003:01:00.1: mlx5_pci_init failed with error code -19

Always receiving error while trying to read device configuration or open device - it tries to open wrong device (0003:01:00.0 and 0003:01:00.1 instead of 10003:01:00.0 and 10003:01:00.1):

~]# mstflint -d 10003:01:00.0 q

-E- Cannot open Device: 10003:01:00.0. No such file or directory. MFE_CR_ERROR

~]# mst status

MST modules:

------------

    MST PCI module loaded

    MST PCI configuration module loaded

MST devices:

------------

/dev/mst/mt4117_pciconf0         - PCI configuration cycles access.

                                   domain:bus:dev.fn=10003:01:00.0 addr.reg=88 data.reg=92

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

                                   Chip revision is: 00

/dev/mst/mt4117_pci_cr0          - PCI direct access.

                                   domain:bus:dev.fn=10003:01:00.0 bar=0x00000000 size=0x0

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

mopen: Invalid argument

                                   Chip revision is:

~]# flint -d /dev/mst/mt4117_pciconf0 query

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.1/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Failed to open (/sys/bus/pci/devices/0003:01:00.0/config) for reading: No such file or directory

Image type:            FS3

FW Version:            14.21.1000

FW Release Date:       29.10.2017

Product Version:       rel-14_21_1000

Rom Info:              type=UEFI version=14.14.22 cpu=AMD64

                       type=PXE version=3.5.305 cpu=AMD64

Description:           UID                GuidsNumber

Base GUID:             ac1f6b15d92d2f68        4

Base MAC:              ac1f6b2d2f68            4

Image VSD:             N/A

Device VSD:            N/A

PSID:                  SM_2001000001034

Security Attributes:   N/A

Paths /sys/bus/pci/devices/10003:01:00.1/config and /sys/bus/pci/devices/0003:01:00.0/config exist, but mellanox tools try to open wrong one. This error appears everywhere - query device/update firmware/reset device - tools just cut first digit of pci domain number.

The same stuff on Debian works great, but pci device was in 0000 domain (0000:86:00.0 and 0000:86:00.1). This error repeats on RHEL7 too - the same situation.

Looks like problem with 5-digit pci domain. Maybe someone faced and solved the similar problem?

Thanks!

We're using WINOF 5.5 version and still the same problem.

Hi Munesh,

Thank you for posting your question on the Mellanox Community.

Based on the information provided, it shows that the RDMA connection is working else, you will not get any output.

As you are running, Oracle Linux provided MLNX drivers, we recommend to install the MLNX_OFED driver for OL, which you can find through the following link ->http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers.

In our lab, we tested with the MLNX_OFED installed as that is what we support. It comes with all the latest 'perftest' utils.

Attached is a screenshot which I took from our lab system, it show 'ib_write_bw' bi-directional for a 50Gb/s link. And as it is bidirectional it will add both results together.

If after installation of our MLNX_OFED driver, you still experiencing issues, please open a Mellanox Support Ticket, by sending an email to support@mellanox.com

Thanks and regards,

~Mellanox Technical Support

Hi Munesh,

Thank you for posting your question on the Mellanox Community.

Based on the information provided, it shows that the RDMA connection is working else, you will not get any output.

As you are running, Oracle Linux provided MLNX drivers, we recommend to install the MLNX_OFED driver for OL, which you can find through the following link ->http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers.

In our lab, we tested with the MLNX_OFED installed as that is what we support. It comes with all the latest 'perftest' utils.

Attached is a screenshot which I took from our lab system, it show 'ib_write_bw' bi-directional for a 50Gb/s link. And as it is bidirectional it will add both results together.

If after installation of our MLNX_OFED driver, you still experiencing issues, please open a Mellanox Support Ticket, by sending an email to support@mellanox.com

Thanks and regards,

~Mellanox Technical Support

Hi Daniel,

I went through all those steps, but still the HOST_CHAINING isn't working for me. Any additional ideas I can go for?

What I noticed is: Sending a ping from A to B looks the following. ICMP Request is sent correctly from A toB, but Bs

arp request before sending the ICMP answer moves down the line from B to C and C discards the answer.

For me it looks like the HOST_CHAIN is still not working. But on the same page, I have no glue what to do next.

From what I gather, You might not have host_chaining enabled on C; or you might be using VMWare.

Host chaining is all done on-card, and so the host kernels are not aware of it.

Since chaining works based off of the destination mac; if C doesn't have chaining on; C will see that the packet wasn't meant for it, and not bother replying/rejecting/dropping/forwarding the packet.

With chaining on; the ASIC on the card for C will forward it without sending it to the kernel. The host won't even know that there was a packet to start with.

Something else that I might look at is the arp tables. Could it be possible that with other tests, the table is poisoned? I haven't seen it, but host_chaining is something else...

No, it's turned on and I'm not running ESXi, I'm running Debian 9.5. Here's my setup:

Node PVE1

Port1: 172.31.31.11/24 - connected to PVE2 Port2

Port2: 172.31.31.21/24 - connected to PVE4 Port1

root@pve1:~# mlxconfig q | grep HOST_C

HOST_CHAINING_MODE                              BASIC(1)

HOST_CHAINING_DESCRIPTORS                Array[0..7]

HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

Node PVE2

Port1: 172.31.31.12/24 - connected to PVE3 Port2

Port2: 172.31.31.22/24 - connected to PVE1 Port1

root@pve2:~# mlxconfig q | grep HOST_C

HOST_CHAINING_MODE                              BASIC(1)

HOST_CHAINING_DESCRIPTORS                Array[0..7]

HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

Node PVE3

Port1: 172.31.31.13/24 - connected to PVE4 Port2

Port2: 172.31.31.23/24 - connected to PVE2 Port1

root@pve3:~# mlxconfig q | grep HOST_C

HOST_CHAINING_MODE                              BASIC(1)

HOST_CHAINING_DESCRIPTORS                Array[0..7]

HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

Node PVE4

Port1: 172.31.31.14/24 - connected to PVE1 Port2

Port2: 172.31.31.24/24 - connected to PVE3 Port1

root@pve4:~# mlxconfig q | grep HOST_C

HOST_CHAINING_MODE                              BASIC(1)

HOST_CHAINING_DESCRIPTORS                Array[0..7]

HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]

Any ideas what I can look into?

Ah; that diagram looks right, all on the same subnet, and all connected in a correct ring.

If I had to take a guess, lower the MTU back to 1500 on all the nodes (both interfaces) `ifconfig ib0 mtu 1500 ; ifconfig ib1 mtu 1500`

We had issues with high MTU throwing host_chaining into a weird packet drop situation; which looks like what might be happening here. They said that it was fixed in a newer FW, but I wasn't able to fully test and make sure it was fixed.

If that doesn't work, I'm out of ideas. Support will give you a script to run on all the nodes; and that's'd be my next action. They have a lot of useful information in that report; so it is worth a look before you send it off.

I've been disappointed with Mellanox with regards to documentation on *any* of this feature.

Disappointment also on my side :-(

But thank you so much for your help.

Hi,

Look at this thread, it just says that 5 digit for PCI ID domain is not standard compliant.

flint truncating long PCI addresses, device file cannot be opened · Issue #22 · Mellanox/mstflint · GitHub

Marc

We are a little bit confuse about MLAG, PFC and RDMA (RoCe) on a stretch.

We have 4 x SN2010-Spectrum-Switches which we want to connect redundantly with our two Hyper-V-Cluster in a stretch-modus with ConnectX-4 -Cards each Server.

2 switches of each site should run in a HA Modus, therefore we need MLAG of each pair.

As Mellanox-Docs like https://community.mellanox.com/docs/DOC-1434 we have to enable PFC globally. OK

For Hyper-V with RDMA and switch-embedded-teaming under Windows 2016 we need PFC too, but as MLX-Docs https://community.mellanox.com/docs/DOC-2483 with priority (e.g.) 4 desired on two ports over the switch-pair.

Now, do we have to configure the priority of each MLAG-Switch-Pair for PFC via IPL?

How we have to configure PFC-Priority in the trunk ( in our below scenario) between the two MLAGs-Domain ?

Thanks for your help in advance

you can use Esxi6.5 as an initiator (not as a target) for discover, mount and use LUNs over NVMEoF Storage protocol, for instance, when using nvmetcli

use the 2x articles that will guide you through the configuration

How to Monitor and Manage NVMe Devices Using the Command Line Interface -- Virtualization Review

HowTo Configure NVMe over Fabrics (NVMe-oF) Target Offload

delayed_cache_work_func

Detected that the function executes millions of times per second 

Is there any problem in the function that can't be quit.

static void __cache_work_func(struct mlx5_cache_ent *ent)

{

        struct mlx5_ib_dev *dev = ent->dev;

        struct mlx5_mr_cache *cache = &dev->cache;

        int i = order2idx(dev, ent->order);

        int err;

        s64 dtime；

        if (cache->stopped)

                return;

        ent = &dev->cache.ent[i];

        if (ent->cur < 2 * ent->limit && !dev->fill_delay) {

                err = add_keys(dev, i, 1);

                if (ent->cur < 2 * ent->limit) {

                        if (err == -EAGAIN) {

                                mlx5_ib_dbg(dev, "returned eagain, order %d\n",

                                            i + 2);

                                cancel_delayed_work(&ent->dwork);

                                queue_delayed_work(cache->wq, &ent->dwork,

                                                   msecs_to_jiffies(3));

                        } else if (err) {

                                mlx5_ib_warn(dev, "command failed order %d, err %d\n",

                                             i + 2, err);

                                cancel_delayed_work(&ent->dwork);

                                queue_delayed_work(cache->wq, &ent->dwork,

                                                   msecs_to_jiffies(1000));

                        } else {

                                queue_work(cache->wq, &ent->work);

                        }

                }

        } else if (ent->cur > 2 * ent->limit) {

                /*

                 * The remove_keys() logic is performed as garbage collection

                 * task. Such task is intended to be run when no other active

                 * processes are running.

                 *

                 * The need_resched() will return TRUE if there are user tasks

                 * to be activated in near future.

                 *

                 * In such case, we don't execute remove_keys() and postpone

                 * the garbage collection work to try to run in next cycle,

                 * in order to free CPU resources to other tasks.

                 */

                dtime = (cache->last_add + (s64)cache->rel_timeout * HZ) -

                        jiffies;

                if (cache->rel_imm ||

                    (cache->rel_timeout >= 0 && !someone_adding(cache) &&

                     dtime <= 0)) {

                        remove_keys(dev, i, 1);

                        if (ent->cur > ent->limit)

                                queue_work(cache->wq, &ent->work);

                } else if (cache->rel_timeout >= 0) {

                        dtime = max_t(s64, dtime, 0);

                        dtime = min_t(s64, dtime, (MAX_MR_RELEASE_TIMEOUT * HZ));

                        cancel_delayed_work(&ent->dwork);

                        queue_delayed_work(cache->wq, &ent->dwork, dtime);

                }

        } else if (cache->rel_imm && !someone_releasing(cache)) {

                cache->rel_imm = 0;

        }

}

static void delayed_cache_work_func(struct work_struct *work)

{

        struct mlx5_cache_ent *ent;

        ent = container_of(work, struct mlx5_cache_ent, dwork.work);

        __cache_work_func(ent);

}

delayed_cache_work_func

Hi,

I am currently trying to install OFED drivers for FC27. I had a FC 26 install (kernel 4.16.11-100.fc26.x86_64), but upgraded to FC 27 based on documentation. As part of the upgrade I upgraded the kernel to the following 4.18.15-100.fc27.x86_64. I then went to install, and had to build kernel support

./MLNX_OFED_LINUX-4.4-1.0.0.0-fc27-x86_64/mlnxofedinstall --add-kernel-support

Unfortunately the version is the kernel isn't compatible with this version of OFED, as I was getting the following 2 compile errors

OFED EXPECTS THIS BE RETURNING AN INT - /lib/modules/4.18.15-100.fc27.x86_64/build/include/linux/mm.h

void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, unsigned long size);

OFED DOES NOT EXPECT "const struct sk_buff *skb" parameter in API in /lib/modules/4.18.15-100.fc27.x86_64/build/include/net/ip6_route.h

struct rt6_info *rt6_lookup(struct net *net, const struct in6_addr *daddr, const struct in6_addr *saddr, int oif, const struct sk_buff *skb, int flags);

After examination, it would seem that the older kernel version (4.16.11-100.fc26.x86_64), has the correct form of the above APIs and the drivers can be build against this. So I went back to a FC26 install, however the install fails immediately now as it expects a FC27 install.

At this point I am unclear how I should proceed. Should I upgrade from 26-27 and not update the kernel, I believe this will work. I want to do the recommended solution. Any guidance on this matter would be greatly appreciated.

Regards

Peter McCarthy

I don't know if I should be asking this here on on the vmware forums, so forgive me if I am in the wrong place. I have two connectx-5 cards connected directly with a QSFP28 cable. One side is on a vsphere 6.7 host, the other on a CentOS 7.5 storage appliance. The basic functionality is working just fine. I am trying to set up an iSER initiator on the vsphere host, but it doesn't seem to be working. One of the vmware blog posts I found gave directions for creating a software iSCSI initiator, and enabling it for iSER, but that doesn't seem to work (yet I don't get any errors.) I ask this here only because there may be a possibility I am configuring something wrong. The cards are connected back to back (no switch), so I haven't bothered with PFC (dunno if that could be relevant.) I also have SR-IOV enabled, with 1 VF attached to a windows server 2012 guest configured as a VEEAM backup proxy (dunno if SR-IOV could be in play either?) Any thoughts, hints or tips appreciated

Hello Peter,

Thank you for posting your question on the Mellanox Community.

Based on the information provided and an internal discussion, we noticed that you opened a Mellanox Support case.

We will continue to assist and update you through that case.

Thanks and regards,
~Mellanox Technical Support

If this is a job offer, then you are not very good at finding new employees in your project. Usually people write requirements, tasks, give contacts, announce the amount of remuneration. Then potential employees will be able to assess the situation and respond to the announcement. I even ordered a resume from topresume reviews for my last job, because the conditions were so cool that I couldn't miss this opportunity. I think you should learn to do the same if you want the workers to compete for your offer.