Hi all,
Trying to hook up my IB storage device to my Power7 Server running RHEL 7.2 ppc64. Installed the latest OFED, but srp isn't working. Can't find any info anywhere that says it is not supported?
The cards ConnectX 2 VPI cards. Has anyone tried this?
Regards
Mark Guz
IMHO this mlnx_tune Python script has a bug. The logic used in issuing the warning message is incorrect.
1462 pci_width_compatible_to_cpu_ok = not (int(self.actual_pci_width) >= PciDeviceInfo.MIN_LENGTH_CHECK_HSW_COMPATIBLE and cpu_arch == Architecture.HASWELL)
In the same code,
1408 MIN_LENGTH_CHECK_HSW_COMPATIBLE = 16
Now if you follow the logic, the result is:
int(self.actual_pci_width) >= PciDeviceInfo.MIN_LENGTH_CHECK_HSW_COMPATIBLE is 16 >= 16 is True
Architecture.HASWELL is also True
True and True is True
not True is False
pci_width_compatible_to_cpu_ok is false
But this is invalid! EDR needs PCIe gen3 x 16. That's mandatory, so why the script complains? Using lspci -vv, it's easy to see the following:
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
See the above Width x16 ?
So, if the lspci correctly reports an EDR HCA in the system and it's in a PCIe Gen3 x 16 slot, the mlnx_tune issues a warning? This is a bug I am afraid.
I think I have the answer now. It's due to the confusion caused by the prevalent and inconsistent use of MB and MiB out there in different software applications.
When I ran ib_write_bw with the --report_gbits flag, I did see over 50+ Gbps. That got me curious, so I assumed the MB/s output to be actually MiB/s, then 6028MiB/s = 51.02Gbps, as anticipated.
Not sure about PPC64 architecture but, in general, the situation with Mellanox Infiniband and SRP support is this:
Hope this helps.
Cheers!
Hi! Anand
Could you give me a ftp link for my 2 of SX6036G VPI Gateway system?
I received a message another threads that said to me cable detection bug will be resolved in future MLNX-OS release.
My eMail address is here.
jhchoi AT kuwoo DOT co DOT kr
Best Regards.
Hi
Thanks for responding.
I'm not trying to create a target on my PPC64 box, I'm trying to connect
to an external IB target.
I have a ConnectX-2 VPI card that does not work with the inbox drivers on
ppc64
mlx4_core: Initializing 0002:01:00.0
mlx4_core 0002:01:00.0: PCIe link speed is 5.0GT/s, device
supports 5.0GT/s
mlx4_core 0002:01:00.0: PCIe link width is x8, device
supports x8
mlx4_core 0002:01:00.0: Only 64 UAR pages (need more than
128)
mlx4_core 0002:01:00.0: Increase firmware
log2_uar_bar_megabytes?
mlx4_core 0002:01:00.0: Failed to initialize user access
region table, aborting
If I install the OFED the card works but there is no SRP support so I'm
not able to connect to the external target?
Am I missing something?
Cheers
Mark Guz
Senior IT Specialist
Flash Systems & Technology
Office 713-587-1048
Mobile 832-290-8161
Hi,
Can you please run the below and provide the output :
sminfo -P 1
sminfo -P 2
Hi Scott,
Can you copy paste the switches configuration (including the cisco 6509),
please indicate which ports are connected between the SX1016 and the Cisco's
Hi Anand,
EMC² is NOT supportive at all. Their support people even do not know about
Mellanox switches ( !? ), although EMC² use the switches in their ISILON NAS
systems. STRANGE !
I recently devoted some time again to study the extensive Mellanox documentation.
My problem is to put the latest MLNX-OS onto my two MSX6012F-2BFS switches.
These switches have a non standard EMC² firmware, degrading them to unmanaged
switches, local 1Gb management LAN port not operable. I found the solution on how
to do that in appendix D of the MFT user manual version 2.7
What I need is the following:
OEM version of the FLINT program to burn the VPI switches EEPROM the first time.
This version can burn the MLNX-OS to an empty EEPROM. If the EEPROM really needs
to be EMPTY, I would need a program to bulk erase the EEPROM as well.
Latest version of MLNX-OS for the PPC processor, documentation and release notes.
Thank you very much for your great support in the past !
Speedy
We don't support Upgrading from EMC firmware to Mellanox Firmware, you need to contact EMC for that.
Ophir.
Dears,
I need to use Platform MPI 9.1.3 with my application being run in -IBAL mode.
Unfortuantlly no driver released later than WinOF v2.1.2 ( 3.xx , 4.xx , 5.xx ) won't be detected by MPI and it fires error like this :
Rank 0:1: MPI_Init: didn't find active interface/port
Rank 0:1: MPI_Init: Can't initialize RDMA device
Rank 0:1: MPI_Init: Internal Error: Cannot initialize RDMA protocol
if -IBAL switch is removed, TCP will be utilized and the program works but infiniband is not implemented.
Please let me know what's the reason beyond this. Why Mellanox released so many drivers so far which are not compatible with Platform MPI ?
Am I missed a tweak ?
Thanks
6 days ago, I reported that I fixed my IPoIB setup. I just found sometime a while ago to revisit this issue. Indeed, as I suspected, the original, incorrect IPoIB setup was the cause of jitters observed, most likely due to the fact that for 10G Ethernet ports on servers, I have been using some inexpensive third party SFP+ DACs So, problem solved for now.
Hi,
First of all, I'm glad to hear you are using mlnx_tune and I hope you find it at least a little bit helpful.
For the issue above, I agree the behavior isn't perfect but I would like to explain a few points:
I hope this answers your concern. If not, I would be happy to hear your feedback and improve mlnx_tune.
Hi Tal,
Thanks for your response, from the author of mlnx_tune too! Great to e-meet you!
Regarding your point 1: For most (if not all) OS, it is next to impossible to query for the I/O non-posted prefetch value. May I make a suggestion? Given Mellanox's industry-wide weight, and the fact there are only two major COTS BIOS makers: AMI and Phoenix, both publish Windows/Linux based BIOS review tools, so perhaps calling such tools is a way to do more reliable queries for BIOS values, including the I/O no-posted prefetch value?
Regarding your point 2: The issue with I/O non-posted prefetch is only relevant when trying to run traffic at rates higher than 150Gb/s (bidirectional). Indeed, we are using four EDR IB HCAs, with SB7700 IB switch. We are pushing for more than 100Gbps traffic so we are in the territory that you mentioned
Finally, one suggestion: the -p comes with the following options, but AFAICT, the only Mellanox doc that explains somewhat is this HowTo Tune Your Linux Server for Best Performance using mlnx_tune Tool published by the prolific Ophir in Oct 22, 2014. IMHO, there is a lack of actual application examples for each profile. An excellent one for illustration would be again Ophir's HowTo Configure and Test BeeGFS with RDMA.
['HIGH_THROUGHPUT',
'IP_FORWARDING_MULTI_STREAM_THROUGHPUT',
'IP_FORWARDING_MULTI_STREAM_PACKET_RATE',
'IP_FORWARDING_SINGLE_STREAM',
'IP_FORWARDING_SINGLE_STREAM_0_LOSS',
'IP_FORWARDING_SINGLE_STREAM_SINGLE_PORT']
Could you please talk to him about adding this piece of info? Since the naming of these profiles all start with IP, which means they apply to Ethernet mostly (IB traffic doesn't use kernel networking stack!). So, perhaps somewhere there should be more explinations regarding
I look forward to hearing your comments about the above.
Best,
Chin
#1 - constantly in progress. As with any other relevant parameter, I'll add it to mlnx_tune once there's a solution.
Regarding the documentation, next mlnx_tune version should be out by EOM September. I will update the mlnx_tune entry with recent changes and will expand upon the existing profiles.
Until than, mlnx_tune as a static system analyzer is good for both Ethernet and IB. The tuning part is currently only good for Ethernet, mostly because IB has excellent out of box performance.
Again, thanks for your feedback. It is highly appreciated.
Hi Tal,
Look forward to the next mlnx_tune. Thanks for confirming my code-review conclusion. Please talk to Ophir and encourage him to supplement his very useful HowTo Configure and Test BeeGFS with RDMA the use of mlnx_tune. Given the popularity of Mellanox IB products in the HPC space, the addition will be valuable.
Best,
Chin
Hi Tal,
A quick supplement to #1, well-known server vendors may provide their own utilities for set/view BIOS values. Eg. Dell's OpenManage Deployment Kit. Given many such large vendors work closely with Mellanox, that's another option for you and your users?
Best,
Chin
We are aware of such tools, but also would like mlnx_tune to be as independent as possible. I will consider checking if these exist on the system and use them if they are.
