Hi Chris,
have you received a solution for your problem?
I'm quite interested because we just have installed a new cluster in our compagny and, unfortunately, we have the same error messsage than you when we launch some MPI jobs.
It is a very strong problem for us since, up to now, we are not able to start our studies.
Thank you.
Hi,
I'm looking the best value of max_vfs on SR-IOV configuration of virtualization host. I find following statement on ConnectX-5 EN brochure.
SR-IOV: Up to 1K Virtual Functions
SR-IOV: Up to 16 Physical Functions per host
Let's say that the virtualization host is equipped with total of 40 cores (80 threads). Should I use 1024 or 64 (1024/16) for max_vfs? If not, what is the optimal one for that environment? How to utilize all ConnectX-5 EN VF? How to utilize all ConnectX-5 EN PF?
Best regards,
Hi Raphael,
RoCE v2 is a UDP based protocol, and UDP, unfortunately, does not guarantee delivery, ordering or duplicate protection of the packets.
In the case if you have an additional programming questions, I would suggest to ask the question on linux-rdma mailing list.
Hi Pratik,
Oohh I see. Will try this then and will give an update.
Thank you.
Hi,
netdev mailing list - netdev mailing list might be a better place to raise the question.
Hello Gerald,
Many thanks for reaching out to the Mellanox Community.
Unfortunately, the ConnectX-2 adapters you have are EOL/EOS for a while now. Even though the firmware you are running is the latest available, the current versions of the driver do not support the ConnectX-2 adapter anymore.
Also we notice, that you are running on an AMD Ryzen platform, which we do not test. The only AMD platform we test and certify the driver and adapters for is the EPYC platform.
Never the less, you can try if applying the recommendation from the following community post, improves the performance. The link is https://community.mellanox.com/docs/DOC-3086
Our recommendation is to try to get your hands on a pair of ConnectX-3 adapters which are fully supported with our current drivers.
Many Thanks and regards,
~Mellanox Technical Support
Hi Tibor,
Many thanks for reaching out to the Mellanox Community.
Please check if reburning your firmware when the card is booted in the Livefish mode, resolve your issue. If this does not resolve the issue, the card is dead.
For booting you card into Livefish mode, please follow the instructions in the MFT (Mellanox Firmware Tools) User Manual ( http://www.mellanox.com/related-docs/MFT/MFT_user_manual_4_10_0.pdf ), Appendix C: "Booting HCA Device in Livefish Mode".
For correctly burning the firmware in this mode, please use the following command:
# flint -d <device> -i <firmware bin file> -nofs -guid <card guid> burn GUID can be found on the HCA sticker.
Example:
# flint -d /dev/mst/mt26428_pciconf0 -i MNPH29C-XTR.bin -nofs –guid 0002C903000B6D1C burn
After successful burning the firmware, please reboot the card in Normal mode.
Many thanks,
~Mellanox Technical Support
Hi Garegin,
Many thanks for reaching out to the Mellanox Community.
As the ConnectX-4 is using the same mlx5 PMD driver as the ConnectX-5, the ConnectX-4 is supported for using with VPP.
For VPP support, please refer to the VPP Mailings list.
Many thanks.
~Mellanox Technical Support
Hi, Experts:
When deploying VM, I have meet an issue about mlx5_mac_addr_set() to set a new MAC different with the MAC that VMWare Hypervisor generated,
and the unicast traffic (ping) fails, while ARP has learned the new MAC. Since Mellanox NIC is not set anti-spoofing by default,
the VMWare lloks to add some anti-mac-spoofing functionality. The VF is configured through pciPassthru.
I have googled online and some link mentions to set the following parameters in VMX, but it looks NOT working:
pciPassthru0.noPromisc = "false"
pciPassthru0.noForgedSrcAddr = "false"
pciPassthru1.noPromisc = "false"
pciPassthru1.noForgedSrcAddr = "false"
(Note: it is not SR-IOV, and there has no up-link vswitch Port Group there for passthru pciDevice).
Someone else may have asked this question already. I have two questions:
1): Does Mellanox community has finalized the method or document? or no solution yet?
2): How to check (which instruction) whether Mellanox NIC anti-MAC-spoofing status?
Thanks.
-- Edward
Hi Community,
I'd like to understand better a problem we have, which seems to be linked to the fact that DPDK's xstats/ethtool -S shows a lot of "rx-out-of-buffer" packets. I found the performance counter document, but it does not say much about why this could happen, which buffer we're speaking about. I quote "Number of times receive queue had no software buffers allocated for the adapter's incoming traffic.". As rx_nombufs (DPDK stats) is 0 I guess it does not mean that there is not enough software buffers. Are they some internal MLX buffers? What can be done to prevent that?
Thanks,
Tom
Is there any error messages? Does ibtracert work (#ibtracert <src lid> <dst list>?
ibtracert works
We actually have connection but we are only able to ibping to the GUID that is binded on OpenSM but can't ibping to the other GUIDs now.
If your question is if it possible to use different MOFED version for different kernel, then it is not . However, you should be able to install Mellanox OFED on one kernel with --disable-kmp option, reboot to other kernel, recompile same Mellanox OFED using mlnx_add_kernel_support and after that install only kernel modules.
According to you, the issue somewhere in OS (I/O, memory allocation, other) and not in the network. 20Gbps on ConnectX-2 will give you maximum theoretical 16 Gbps because of 8/10 encoding, so 15.6 Gbps is pretty close.
I would suggest to use perf to analyze ssh/rsync behaviour, or maybe 'strace -ttt -T' option in order to see how much time it spends in the system calls
I found one bug in the Mellanox drivers. In the RT kernel on the Centos 7.5 system, the ptp4l or phc2sys processes do not want to start automatically as a service (systemctl). There is no problem with drivers built into the kernel.
Hello Todd -
Please see: http://www.mellanox.com/related-docs/whitepapers/InfiniBandFAQ_FQ_100.pdf and let me know if it helps with your questions.
Specifically: 16 & 17
- InfiniBand supports QoS by creating Virtual Lanes (VL). These VLs are separate logical communication links that share a single physical link.
- InfiniBand, by contrast, uses link-level flow control to ensure that packets are not dropped in the fabric
Many thanks -
~Steve
According to you, the issue somewhere in OS (I/O, memory allocation, other) and not in the network. 20Gbps on ConnectX-2 will give you maximum theoretical 16 Gbps because of 8/10 encoding, so 15.6 Gbps is pretty close.
I would suggest to use perf to analyze ssh/rsync behaviour, or maybe 'strace -ttt -T' option in order to see how much time it spends in the system calls
I've created a puppet resource for interfaces. Most of the interface names on my switch are lowercase with the exception of Ethernet interfaces, so I munged the interface name to hopefully reduce errors in the manifest; e.g.:
manifest:
cisco_interface { 'Ethernet1/1': description => 'foo' } type/cisco_interface.rb:
newparam(:name) do munge { |value| value.downcase } end My provider code also downcases the interface names when I collect the list of interfaces with self.instances.
So this works great when I test with the manifest, but not so great with the puppet resource command which only works when I call it with the name already downcased:
switch# puppet resource cisco_interface 'Ethernet1/1' cisco_interface { 'Ethernet1/1': ensure => 'absent', } switch# puppet resource cisco_interface 'ethernet1/1' cisco_interface { 'ethernet1/1': ensure => 'present', description => 'foo', } The puppet resource command name field seems to just be a simple filter so I think I'm stuck, but I thought I'd seen other resource types munging title values like this.
Is it possible to munge the title values in a way that works for both scenarios? If not then I'm not sure whether it would be better to leave it case-sensitive since that is what users will see in the switch config, or to "help" them avoid errors in the manifest.
it still does not work
What drivers are you using?
Sorry I am new to 10G BASE-SR and I can't seem to find a good resource that can confirm if an SFP supported in a Cisco Nexus 5548UP will be compatible with a new service The new service is described as '10 Gigabit Ethernet LAN PHY IEEE 10G BASE-LR10.3125 Gbps +/- 100 ppm 1310nm'
Ultimately I need to understand if a 'cisco sfp-10g-sr' for which the transmitter wavelength spec is described as 850nm is usable.
Thanks for your patient, i plan to take it and any site recommend?
Anyone talk form sfpcables? 10Gb/s SFP+ SR | SFP-10G-SR | J9150A | SFP+ 10GBase-SR - 10Gtek
5% OFF all items there now, i want to save a lot there.