krping is not part of Mellanox OFED. Could you contact 'kpring' developers and check if they can point on any issue with mlx5 driver?
↧
Re: krping (4.7 kernel version) crashing with mlx5_core (CX415A), it is passing with mlx4_core(CX354A)
↧
Re: Change mlx5 to Ethernet mode without mlxconfig
Hi Robert, the Connect-IB HCA only actually supports InfiniBand so that's why you can't configure it to Ethernet. Only the ConnectX cards have dual personalities where you can config them to either IB or Eth. The "X" by the way was supposed to indicate that it can be either. Thanks! Branko.
↧
↧
Re: OpenMPI with MXM 32bit issue
Hello Thomas,
Did you mean to run this test on one host or on two?
If on 2, please add ‘--map-by node’ to the command line and rerun.
I verified that it works:
$/usr/mpi/gcc/openmpi-1.10.3rc4/bin/mpirun -np 2 --map-by node --display-map -mca pml yalla exec
Data for JOB [1162,1] offset 0
======================== JOB MAP ========================
Data for node: vegas27 Num slots: 16 Max slots: 0 Num procs: 1
Process OMPI jobid: [1162,1] App: 0 Process rank: 0
Data for node: vegas28 Num slots: 16 Max slots: 0 Num procs: 1
Process OMPI jobid: [1162,1] App: 0 Process rank: 1
=============================================================
0: ready to run
1: ready to run
1: reached 2147483641 receives
1: reached 2147483642 receives
1: reached 2147483643 receives
1: reached 2147483644 receives
1: reached 2147483645 receives
1: reached 2147483646 receives
1: reached 2147483647 receives
1: reached 2147483648 receives
1: reached 2147483649 receives
1: reached 2147483650 receives
1: reached 2147483651 receives
1: reached 2147483652 receives
1: reached 2147483653 receives
1: reached 2147483654 receives
1: reached 2147483655 receives
1: reached 2147483656 receives
1: reached 2147483657 receives
1: reached 2147483658 receives
1: reached 2147483659 receives
1: reached 2147483660 receives
1: finished
0: reached 2147483641 sends
0: reached 2147483642 sends
0: reached 2147483643 sends
0: reached 2147483644 sends
0: reached 2147483645 sends
0: reached 2147483646 sends
0: reached 2147483647 sends
0: reached 2147483648 sends
0: reached 2147483649 sends
0: reached 2147483650 sends
0: reached 2147483651 sends
0: reached 2147483652 sends
0: reached 2147483653 sends
0: reached 2147483654 sends
0: reached 2147483655 sends
0: reached 2147483656 sends
0: reached 2147483657 sends
0: reached 2147483658 sends
0: reached 2147483659 sends
0: reached 2147483660 sends
0: finished
I'm checking the one host case.
Alina.
↧
Re: OpenMPI with MXM 32bit issue
Hello Alina,
thank you for your response. I meant the case on one host, but I will check the two host case anyway.
One part of the problem is, that although the Infiniband network is not involved in the single host case, the example does not run properly if OpenMPI is compiled with the "--with-mxm" option.
Thomas
↧
Mellanox Connectx-2 shows cable unplugged when connected to switch
I am new to 10gbe, and was able to direct connect 2 test severs using Connectx-2 cards and SPF+ cable successfully, but when connected to the SPF+ port on my 3Com switch, it shows the cable unplugged. This is my test set up.
Server 2012 R2, all updates installed. HP DL380 G7s.
Switches: 3Com 2948- SFP-Plus x4 SFP Connections. Cisco Passive Copper 10gbe Cable, testing working
Mellanox HW Card settings: Eth, all other settings at defaults
Information: Driver 4.80.10388.0 Firmware 2.9.1200 Part# MNPA19-XTR
I understand when using Infiniband it requires a Subnet Manager, but does this also apply in Ethernet mode?
Is a special cable required? or special driver settings?
Thanks, and look forward to playing with these.
Van
↧
↧
Re: OpenMPI with MXM 32bit issue
Well, thats interesting.
The case on two hosts works fine:
$ /opt/openmpi-2.0.1-jessie-mxm-mt/bin/mpirun -np 2 -hostfile hostfile --map-by node --display-map -mca pml yalla openmpi_mxm_freeze
Data for JOB [31717,1] offset 0
======================== JOB MAP ========================
Data for node: intel1 Num slots: 1 Max slots: 0 Num procs: 1
Process OMPI jobid: [31717,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0-1]]:[BB/../../../../../../../../../../..][../../../../../../../../../../../..]
Data for node: intel2 Num slots: 1 Max slots: 0 Num procs: 1
Process OMPI jobid: [31717,1] App: 0 Process rank: 1 Bound: socket 0[core 0[hwt 0-1]]:[BB/../../../../../../../../../../..][../../../../../../../../../../../..]
=============================================================
[1474616276.871628] [intel1:7883 :0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2906.98
[1474616276.903256] [intel2:3181 :0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 3043.73
0: ready to run
1: ready to run
...
0: finished
1: finished
while the one host case does not:
$ /opt/openmpi-2.0.1-jessie-mxm-mt/bin/mpirun -np 2 --map-by node --display-map -mca pml yalla openmpi_mxm_freeze
Data for JOB [31494,1] offset 0
======================== JOB MAP ========================
Data for node: intel1 Num slots: 24 Max slots: 0 Num procs: 2
Process OMPI jobid: [31494,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0-1]]:[BB/../../../../../../../../../../..][../../../../../../../../../../../..]
Process OMPI jobid: [31494,1] App: 0 Process rank: 1 Bound: socket 0[core 1[hwt 0-1]]:[../BB/../../../../../../../../../..][../../../../../../../../../../../..]
=============================================================
[1474615276.877829] [intel1:7723 :0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2971.04
[1474615276.877833] [intel1:7724 :0] sys.c:744 MXM WARN Conflicting CPU frequencies detected, using: 2971.04
0: ready to run
1: ready to run
...
freeze
Since we are normally using a single host and just in extreme cases two or more hosts, a solution for the single host would be appreciated.
↧
Re: OpenMPI with MXM 32bit issue
I see. I will check this and get back to you.
In the meantime can you please check if adding the following to the command line resolves the hang?
-x MXM_TLS=ud
or
-x MXM_TLS=rc
Thanks,
Alina.
↧
Re: Is it possible to switch ConnectX-3 VPI into Ethernet mode in Solaris 11.1?
If you change ETH, ETH mode with mlxconfig, Solaris 11.3 or above support ethernet mode with ConnectX-3 HCA...:)
↧
Re: OpenMPI with MXM 32bit issue
Both options work, while the "ud" option is significantly slower.
↧
↧
MPI startup():ofa fabric is not available and fallback fabric is not exist
Hello All,
I hope you are well. We are facing the above issue whenever we submit second simulation job. The first job goes through smoothly but the second job exits with the above error.
Any ideas what can be causing this and how we can fix this ?
We have Mellanox 56 GBps switch with connext x-3 cards on all nodes on a dell R630 14 node setup.
Thanks
↧
Driverdisk for Xenserver 7?
Hi,
can Mellanox support us with a Xenserver 7 Driver Disk for the MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] (Hewlett-Packard Company HP NC542m Dual Port Flex-10 10GbE BLc Adapter)?
robert
↧
Re: ping is not happening between MCX314A and MCX354A 40G Etherenet slots
Hi!
MCX354A is Infiniband HCA and MCX314A is Ethernet only Adapter.
If you install Infiniband driver and MCX354A, you must change Ethernet mode then you can ping to MCX314A end-to-end connection without switch.
↧
Re: Windows Server 2016 Expected Speeds?
You've not mention your CX-3 speed rate (I assume it's 40Gb/s), neither the actual performance you've achieved, but to try and get the oprtimim perfromance I would suggest to start with the following actions:
1. Follow the performance fine-tuning guidance in the Mellanox WinOF "user manual" to set the proper parameters of driver & fw
2. On Windows platforms - use NTttcp performance tool rather then iperf3
(Windows Server: Using the NTttcp Tool to Troubleshoot Network Performance | Dell Israel
↧
↧
Usage of CX4 to QSFP cable for the Application
I have a hardware using Mellanox ConnectX3 VPI IC. It supports 10GBase-CX4 on infiniband style CX4 connector.
Can i communicate with PC ConnectX3 VPI adapter (Part No.MCX354A-FCBT) using CX4 to QSFP cable? such as http://www.10gtek.com/QSFP+-to-CX4-%28SFF-8470%29-Cable-193 or https://www.cablesondemand.com/pcategory/72/category/QSFP+-+CX4/URvars/Catalog/Library/InfoManage/QSFP_TO_CX4_COPPER_CABLES.htm
Can the cable be used for both Infiniband and 10GigE mode of communication?
Thanks you in advance. Regards@
↧
Dfms high rate is not supported and lots of others
Hi, everyone!
Every time I boot my nodes I see such messages (like on the screenshot). What's that and what should I do to solve the problem???
↧
Re: OpenMPI with MXM 32bit issue
Hi Thomas,
I would like to provide you with an updated version of MXM which should fix the problem.
Can you please tell me what OS and Mellanox-OFED you are using?
Thank you,
Alina.
↧
Re: OpenMPI with MXM 32bit issue
Hi Alina,
we are using Debian Jessie and the MLNX_OFED_LINUX-3.3-1.0.4.0-debian8.3-x86_64 package.
Thomas
↧
↧
Re: Usage of CX4 to QSFP cable for the Application
There shouldn't be an issue communicating with a ConnectX3 VPI.
If your ConnectX3 VPI CX4 design supports switching between Infiniband and Ethernet (as the VPI label implies) - there shouldn't be an issue caused by the medium (CX4 to QSFP), when connecting to a MCX354A-FCBT.
Please note - You need to configure both link partners to work in the same protocol (Eth or IB).
Regards.
↧
Re: OpenMPI with MXM 32bit issue
Hi Thomas,
Here is a link to an updated MXM version:
After installation, MXM will be installed in /opt/mellanox/mxm .
The fix will be part of MXM's January release.
Please let me know if this works well for you.
Thank you,
Alina.
↧
Re: krping (4.7 kernel version) crashing with mlx5_core (CX415A), it is passing with mlx4_core(CX354A)
can you add dmesg to the question.
i might have a solution for this.
↧