compatibility issues between Veritas LLT on RedHat 6.8 and Mellanox

January 6, 2017, 8:14 am

≫ Next: Re: Multiple MTU in routed vlans

≪ Previous: Re: HCA extended port counters

Dear All,

Are you aware of any compatibility issues between Veritas LLT on RedHat 6.8 and Mellanox? I see suspicious messages during system boot (no issues with the functioning of LLT have been noticed).

LLT package is VRTSllt-6.2.1.500-RHEL6.x86_64 on Red Hat 6.8, kernel 2.6.32-642.4.2.el6.x86_64 (mlx4_en.ko 2.2-1 came with the kernel)

[nep179@prdctlscthdb01-20161206]$ sudo egrep "Nov 23(.)*kernel: llt" messages-20161127

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_create_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_create_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_resolve_addr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_resolve_addr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_dereg_mr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_dereg_mr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_reject

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_reject

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_disconnect

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_disconnect

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_resolve_route

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_resolve_route

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_bind_addr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_bind_addr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_create_qp

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_create_qp

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_destroy_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_destroy_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_create_id

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_create_id

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_listen

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_listen

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_destroy_qp

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_destroy_qp

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_get_dma_mr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_get_dma_mr

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_alloc_pd

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_alloc_pd

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_connect

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_connect

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_destroy_id

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_destroy_id

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_resize_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_resize_cq

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol rdma_accept

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol rdma_accept

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: disagrees about version of symbol ib_dealloc_pd

Nov 23 15:09:36 prdctlscthdb01 kernel: llt: Unknown symbol ib_dealloc_pd

[nep179@prdctlscthdb01-20161206]$

[nep179@prdctlscthdb01-20161206]$ sudo egrep "Nov 23(.)*kernel: llt" messages-20161127 | wc -l

[nep179@prdctlscthdb01-20161206]$

Thank you,

Aleksandr

↧

Re: Multiple MTU in routed vlans

January 6, 2017, 8:33 am

≫ Next: Re: Multiple MTU in routed vlans

≪ Previous: compatibility issues between Veritas LLT on RedHat 6.8 and Mellanox

Hi,

After rechecking this issue, I figured out that we don't support fragmentation on the switches, it should be done on the adapter.

packets that will arrived with larger MTU directed to smaller MTU port will be dropped.

There is a DF (don't fragment) flag on the IP header that allow of forbid packet fragments. your problem is when running 9K packets from the Storage to some of the 1500 MTU servers.

Re: Multiple MTU in routed vlans

January 6, 2017, 8:36 am

≫ Next: Re: HCA extended port counters

≪ Previous: Re: Multiple MTU in routed vlans

I deleted the old reply to avoid confusion.

Fragmentation is not supported on the switch. So there is no issue with latency.

↧

Re: HCA extended port counters

January 6, 2017, 3:43 pm

≫ Next: Re: MLNX OFED error for RHEL 7.3 RT

≪ Previous: Re: Multiple MTU in routed vlans

We are using CentOS 6.8 with kernel 2.6.32-642.11.1.el6.x86_64 (latest available), and the CentOS mlx4 kernel modules (I tried using OFED but they wouldn't support NFSoRDMA)

↧

Re: MLNX OFED error for RHEL 7.3 RT

January 6, 2017, 5:34 pm

≫ Next: Re: HCA extended port counters

≪ Previous: Re: HCA extended port counters

Hi Oskar,

Are you able to provide us with the requested log?

Thanks and regards,
~Martijn

↧

Re: HCA extended port counters

January 7, 2017, 4:35 am

≫ Next: -E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

≪ Previous: Re: MLNX OFED error for RHEL 7.3 RT

The changes for this are relatively recent and went into some 4.x kernel.

↧

-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

January 9, 2017, 6:20 am

≫ Next: Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

≪ Previous: Re: HCA extended port counters

Hello,

I want to use SR-IOV function.

so I configure it as the "HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand) " instructed.

But when I run the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4". There is some error happened and I can't find any information about it.

Here is the output:

Device #1:

----------

Device type: ConnectX3

PCI device: /dev/mst/mt4099_pciconf0

Configurations: Current New

SRIOV_EN False(0) True(1)

NUM_OF_VFS 8 4

Apply new Configuration? ? (y/n) [n] : y

Applying... Failed!

-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

I use 12 computers, at first there is two computers that could execute the command correctly. But there is ten computers output that the firmware is unsupported for the command

"mlxconfig -d /dev/mst/mt4099_pciconf0 q", so I update the firmware of all the computer as requested. After that, all the computer could execute the command "mlxconfig -d /dev/mst/mt4099_pciconf0 q". But even the first two computers that could execute the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4" well could no longer running the command any more. It's very strange. I guess something was changed by the firmware update. Any help would be appreciate!!

↧

Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

January 9, 2017, 6:38 am

≫ Next: Re: MLNX OFED error for RHEL 7.3 RT

≪ Previous: -E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

Does the included ESXi 6.5 have RDMA support for the hypervisor back to storage? or is it just for a guest. I really am only using SRP at a HOST to Storage level, nothing inside of the vm is infiniband aware.

Mellanox, if you arent going to release SRP drivers again, would you consider releasing the source for the old ones and we will take it from here..

↧

Re: MLNX OFED error for RHEL 7.3 RT

January 9, 2017, 6:42 am

≫ Next: -E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

≪ Previous: Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

Probably I deleted tjis log but below is different log with for the same error:

Logs dir: /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs

[32m

Below is the list of OFED packages that you have chosen

(some may have been added by the installer due to package dependencies):

[0m

ofed-scripts

mlnx-ofa_kernel

mlnx-ofa_kernel-modules

mlnx-ofa_kernel-devel

knem

kernel-mft

iser

srp

isert

mlnx-sdp

mlnx-rds

Build ofed-scripts 3.4 RPM

Build mlnx-ofa_kernel 3.4 RPM

Running rpmbuild --rebuild --define '_topdir /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/OFED_topdir' --define '_sourcedir %{_topdir}/SOURCES' --define '_specdir %{_topdir}/SPECS' --define '_srcrpmdir %{_topdir}/SRPMS' --define '_rpmdir %{_topdir}/RPMS' --nodeps --define '_dist .rhel7u3' --define 'configure_options --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_vnic-mod --with-mlx5-mod --with-cxgb3-mod --with-cxgb4-mod --with-nes-mod --with-ehca-mod --with-qib-mod --with-ipoib-mod --with-ipath_inf-mod --with-amso1100-mod --with-ocrdma-mod --with-sdp-mod --with-srp-mod --with-rds-mod --with-iser-mod --with-e_ipoib-mod --with-9pnet_rdma-mod --with-9p-mod --with-cxgb3i-mod --with-cxgb4i-mod --with-isert-mod' --define 'KVERSION 3.10.0-514.2.2.rt56.424.el7.x86_64' --define 'K_SRC /lib/modules/3.10.0-514.2.2.rt56.424.el7.x86_64/build' --define '_prefix /usr' /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/MLNX_OFED_SRC-3.4-2.0.0.0/SRPMS/mlnx-ofa_kernel-3.4-OFED.3.4.2.0.0.1.g30039f7.src.rpm

[31mFailed to build mlnx-ofa_kernel 3.4 RPM [0m

Collecting debug info...

[31mSee /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log [0m

↧

-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

January 9, 2017, 7:10 am

≫ Next: Re: What is the recommended firmware updating tool for Mellanox HCAs and/or NICs?

≪ Previous: Re: MLNX OFED error for RHEL 7.3 RT

Hello,

I want to use SR-IOV function.

so I configure it as the "HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand) " instructed.

But when I run the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4". There is some error happened and I can't find any information about it.

Here is the output:

Device #1:

----------

Device type: ConnectX3

PCI device: /dev/mst/mt4099_pciconf0

Configurations: Current New

SRIOV_EN False(0) True(1)

NUM_OF_VFS 8 4

Apply new Configuration? ? (y/n) [n] : y

Applying... Failed!

-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

I use 12 computers, at first there is two computers that could execute the command correctly. But there is ten computers output that the firmware is unsupported for the command

↧

Re: What is the recommended firmware updating tool for Mellanox HCAs and/or NICs?

January 9, 2017, 8:58 am

≫ Next: Re: MLNX OFED error for RHEL 7.3 RT

≪ Previous: -E- Failed to set configuration: failed to set SRIOV params: Message receipt ack

Thanks for your reply. So, my own understanding is in-line with yours.

Regards,

Chin

↧

Re: MLNX OFED error for RHEL 7.3 RT

January 9, 2017, 10:02 am

≫ Next: Re: MLNX OFED error for RHEL 7.3 RT

≪ Previous: Re: What is the recommended firmware updating tool for Mellanox HCAs and/or NICs?

Hi Oskar,

We need the contents of /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log

Thanks and regards,
~Martijn

↧

Re: MLNX OFED error for RHEL 7.3 RT

January 9, 2017, 10:03 am

≫ Next: Re: MLNX OFED error for RHEL 7.3 RT

≪ Previous: Re: MLNX OFED error for RHEL 7.3 RT

Hi Oskar,

We need the contents of /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log

Can you please share that log file.

Thanks and regards,

~Martijn

↧

Re: MLNX OFED error for RHEL 7.3 RT

January 9, 2017, 11:20 am

≫ Next: DPDK testpmd can't access device

≪ Previous: Re: MLNX OFED error for RHEL 7.3 RT

can you give me your email. File is too big to paste here.

↧

DPDK testpmd can't access device

January 9, 2017, 1:46 pm

≫ Next: Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

≪ Previous: Re: MLNX OFED error for RHEL 7.3 RT

EAL: Detected 2 lcore(s)

EAL: Probing VFIO support...

EAL: PCI device 0000:06:00.0 on NUMA socket -1

EAL: probe driver: 15b3:1015 net_mlx5

PMD: mlx5.c:428: mlx5_pci_probe(): cannot access device, is mlx5_ib loaded? [ Yes it is loaded ]

EAL: PCI device 0000:06:00.1 on NUMA socket -1

EAL: probe driver: 15b3:1015 net_mlx5

PMD: mlx5.c:428: mlx5_pci_probe(): cannot access device, is mlx5_ib loaded?

EAL: No probed ethernet devices

PANIC in main():

Empty set of forwarding logical cores - check the core mask supplied in the command parameters

↧

Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?

January 9, 2017, 4:15 pm

≫ Next: Re: CentOS 7 KVM-SR-IOV Performance?

≪ Previous: DPDK testpmd can't access device

Section 4.3 of this article suggests that in-box ESXi drivers are RDMA-aware, which you can also conclude from the very fact one of the modules related to these drivers is called nlmx4(5)_rdma. That doesn't mean though that RDMA is actually going to be effectively used for host-to-storage access. For iSCSI, that would require implementing iSER layer, which the in-box driver clearly doesn't have (after all, it's explicitly called "Software iSCSI Adapter" and it's like 100 years old). And, if VMware NFS could utilise RDMA, I'm pretty sure they'd already told us about it.

So, someone has to do some work here, either Mellanox or VMware or both. But they're not eager to tell us anything, as you can clearly see from 1/2 year lack of response to this thread...

↧