What is the meaning of this?
switch-b7bc88 (config) # interface ethernet 1/5-1/6 switchport mode hybrid
% interface range is not supported for this command
I’m adapting from instructions whose purpose isn’t always clear, and this error message isn’t either. What I did differently was to use the command “vlan 1” followed by “exit” rather than “vlan 100” or “vlan 200”, since I’m using vlan 1. It’s also not clear what “vlan 1” does, because “vlan 1 ?” just tells me “”.
switch-b7bc88 (config) # interface ethernet 1/5-1/6 switchport mode hybrid
% interface range is not supported for this command
This means that you can configure a range of ports with the switchport attribute,
it means you need to run:
switch-b7bc88 (config) # interface ethernet 1/5 switchport mode hybrid
switch-b7bc88 (config) # interface ethernet 1/6 switchport mode hybrid
vlan 1 ? should show you something, for example:
SX1036-1 [standalone: master] (config) # vlan 1 ?
<cr>
ip Configure ip configuration on vlan
name configures the name of vlan
no Deletes name of vlan
Apparently I’m still “barking up the wrong tree.” It looks like PFC is similar to XON/XOFF control, which is not at all what I’m looking for. What I want, and what is displayed but not explained, is how to direct traffic of priority 4 or higher to one of the lossless queues (or TCs, whatever they are) and anything lower to one of the lossy ones. There is no configuration option described for doing this, which is what my understand of CoS was.
I’m generating my VLAN tags within software using libpcap, and they’re being extracted by the switch prior to sending them out, but they’re apparently being ignored, due to lack of a suitable configuration, which apparently is not the default configuration.
Yes, I tried them separately and got them to work by not following the example OR the help text, which also “permits” a range. The script of this page neglected to created HTML entities, so my “<cr>” came out as “”. In other words, what I was saying is that there is just a validation that you CAN enter “vlan 1”, not what doing so accomplishes, especially when you exit right after that.
Yes, I tried them separately and got them to work by not following the example OR the help text, which also “permits” a range. The script of this page neglected to created HTML entities, so my “<cr>” came out as “”. In other words, what I was saying is that there is just a validation that you CAN enter “vlan 1”, not what doing so accomplishes, especially when you exit right after that.
This basically comes down to the implementation on how LOS is detected on the RJ45 versus the SFP+ module.
In the optical case the LOS can be detected either via the LOS or the LOS of the pin detector, in the RJ45 case LOS can only be detected via the LOS pin which is not connected hence why the the OS does not see the link go down when the physical link goes down between the switch and the NIC.
If you had a GbE SFP optical module the behavior would be the same as the 10GbE SFP+ module.
Hi Bruce
Seems that uou need to remanufacturer.
The way to retrieve and burn an image from Uboot is protected by restricted PW
I recommend to contact mellanox support
B.R
Avi Alkobi
Mellanox application engeneer
The link is not lost even when disconnecting the cable from the RJ-45 connector SFP module.
Use SFP GbE optical module we have no way.
Is there any solution when using an SFP module copper interface rj45?
Hi,
Go over this document as well, it gives good examples of 3 types of networks
End-to-End QoS Configuration for Mellanox Switches and Adapters
This is what we are saying in the copper case (RJ45) when the link goes down the OS will not detect the link lost because the LOS is detected via the LOS pin which is not connected.
This is not the case with optical modules.
Since this is a hardware issue there is no way to fix this.
Hi Serge,
The major and most important element in the interconnect world is the interconnect capabilities to increase applications performance. The ability to offload the CPU, to execute MPI operations in the interconnect hardware, to provide hardware based RDMA and more, basically an offloading architecture, is the critical element in building the most efficient high-performance systems. The reason that Pathscale did not make it with InfiniPath network products, or QLogic with thir TrueScale products is from lack of any offloading capabilities, and their need to use the CPU cycles for anything related to the network. OmniPath is completely the same as TrueScale.
Intel try to move the discussion to number of ports etc. as they don't want users to focus on the most important item of offloading versus non-offloading. Nevertheless, the number of ports you want to have on the switch ASIC is a number that will fit the sweet spot of the technology. The fact that Intel had to find something to "show benefit" and designed a 48 ports ASIC does not mean it is a better device. Intel switch is a higher latency switch versus Mellanox (110ns vs 90ns), it requires you to use special data protection mechanism (LLR) on any port, and any cable distance. Mellanox does not need to use those mechanisms for 2m copper cables and 30m fiber cables (enough for a datacenter design). Moreover, if you saw the Intel switch design (picture on their web site) you can see a very weird front panel - due to signal integrity issues.
The switch ports, for both Mellanox and Intel are consist of 4 lanes - so there is no advantage here for Intel.
As for your second question the answer is no. Intel does not lock their CPUs and I don't think their will ever be. Intel wants to sell CPUs, as this is their primer business. The reason is that they created Omni-Path is to try and have a differentiation versus the other CPU vendors. Omni-Path does not deliver the performance, efficiency as Mellanox InfiniBand, and it is clearly not proven and actually does not yet exist out there. The best option is Intel CPUs and Mellanox interconnect. And Intel knows that too.
Very thanks for thedetailed answer. I have learned a lot of new things
Can Iclarifya fewpoints ?
1) > Mellanox does not need to use those mechanisms for 2m copper cables and 30m fiber cables (enough for a datacenter design)
This is also truefor all othercable lengthsoronly to 2m copper and30m fiber ?
2) And yettheremellanoxplansto increase the numberof ports ona ASIC switch ?
Hi Serge,
Here are the answers:
1. it is for copper cables up to 2m in length and fiber optics up to 30m in length. Beyond that we have of course a superset of more capabilities (FEC and LLR) that being used as needed.
2. Today we have InfiniBand switches with 36-ports at the ASIC level, than can be used to build an unlimited size of a cluster. The newest switch, Switch-IB 2 is the world first smart switch that can manage and execute MPI collective operation on the switch ASIC. We will announce future solutions once they will become available.
Regards,
Gilad
In addition to trying this (or the part that doesn’t involve the egress priority I’m defining myself through an explicit VLAN that includes a DEI bit that’s being ignored), I’ve also managed to stumble across the way to map a given priority to a given TC, but regardless of the “lossless” quality of TC 1 (mapped to priority 4) and 2 (mapped to priority 5), these mappings seem to have have no more effect in determining what is drop-eligible than setting DEI to false for priorities 4 and 5.
In addition, having to run both mlnx_qos on the server as well as configuring through the CLI complicates things to the point where it seems redundant or even contradictory, and in any case, as incomprehensible as reading uncommented code.
Even with the bandwidth setting for TC 0 reduced to 10, TC 1 increased to 30, and TC 3 increased to 60 to accommodate priorities 6 and 7 that were not explicitly assigned there, it turns out to be lower priorities, if any, that manage to get all their sent messages received, based on analysis of a tcpdump on the receiver side.
Also, I tried “man vconfig” and “which vconfig” and came up empty. I guess I could look into “apt-get install” if needed, but since I’m generating the outgoing frames with explicit VLAN tags, I shouldn’t need it for my purposes, should I?
My first impression of layer 2 QoS was that it was relatively straightforward, but that seems not to be the case.
Hi Gerald,
See this
The mlnx_qos command has the following parameter options:
-p : maps priority (UP) to TC, in this example we can map priority i to TCi (for example, priority 1 to TC 1, priority 3 to TC 3 and 5 for TC 5). "-s 0,1,0,3,0,5,0,0" will do the job.
-s: stands for transmission algorithm. Ether "ets" or "strict" via a list. In our example, we will use map all TC as ETS "-s ets,ets,ets,ets,ets,ets,ets,ets" .
-t : Set the weighted bandwidth for ETS TCs (not relevant for TCs marked with "strict" transmission mode). In this example, it will be: "-t 0,10,0,40,0,50,0,0" . note, the sum of the list must be equal to 100 (percent).
-r: stands for rate-limiting (in Gb/s). it is possible to perform rate limiting on each TC, not to exclude a specific rate. In this example for 10GbE cards, you can use "-r 0,1,0,4,0,5,0,0"
you might want to check the -s strict option, for strict priority traffic, these types of traffic will be sent prior to the ETS marked traffic.
Same thing should be on the switch. if you mark traffic with strict priority.
switch (config)# no dcb ets enable
,Hi Ronnie
can you please attach the full installation log?
The key seems to be the phrase that comes after the mlnx_qos instructions: “On the switch side…”
I don’t think I need either vconfig or mlnx_qos, because I’m using libpcap and creating my own VLAN tags within my software, where I have direct and unambiguous control over them. My only problem would appear to be how to get the switch to treat the higher priorites and non-drop-eligible bits according to their intuitive meanings.
I have not configured subnet manager to support pkey 0x236. Then again I can not find a reference to subnet manager in the documentation I am using.
MLNX-R307 [standalone: master] # show interfaces proxy-arp 1
Proxy-arp 1
Admin state: Enabled
Operational state: Down
GUID: F4:52:14:03:00:13:62:38
Internet Address: 10.4.136.16/22
Broadcast Address: 10.4.139.255
Description: N/A
MTU: 4092
Counters: Disabled
Bridged interfaces: vlan 1(D), pkey 0x7fff
MLNX-R307 [standalone: master] #
MLNX-R307 [standalone: master] # show interfaces proxy-arp 2
Proxy-arp 2
Admin state: Enabled
Operational state: Down
GUID: F4:52:14:03:00:13:62:38
Internet Address: 10.3.136.16/22
Broadcast Address: 10.3.139.255
Description: N/A
MTU: 4092
Counters: Disabled
Bridged interfaces: vlan 236, pkey 0x236(D)
MLNX-R307 [standalone: master] #
Here is additional configuration from the switch.
##
## Interface Ethernet configuration
##
interface ethernet 1/1 mtu 4096 force
interface ethernet 1/2 mtu 4096 force
interface port-channel 2
interface port-channel 2 mtu 4096 force
interface ethernet 1/1 channel-group 2 mode active
interface ethernet 1/2 channel-group 2 mode active
interface port-channel 2 switchport mode trunk
##
## VLAN configuration
##
vlan 236
interface port-channel 2 switchport trunk allowed-vlan none
interface port-channel 2 switchport trunk allowed-vlan 236
vlan 236 name "GRID_INDY"
##
## LAG configuration
##
lacp
##
## Network interface configuration
##
interface ib0 comment ib0 interface
no interface mgmt0 dhcp
interface mgmt0 ip address 172.18.10.48 /24
##
## Proxy Arp configuration
##
interface proxy-arp 1
interface proxy-arp 1 ha member MLNX-R307
interface proxy-arp 1 ha member MLNX-R307 ip address 10.4.136.16
interface proxy-arp 1 ip netmask /22
interface proxy-arp 1 ip pkey 0x7FFF
interface proxy-arp 1 ip vlan 1
interface proxy-arp 1 mtu 4092
no interface proxy-arp 1 shutdown
interface proxy-arp 2
interface proxy-arp 2 ha member MLNX-R307
interface proxy-arp 2 ha member MLNX-R307 ip address 10.3.136.16
interface proxy-arp 2 ip netmask /22
interface proxy-arp 2 ip pkey 0x0236
interface proxy-arp 2 ip vlan 236
interface proxy-arp 2 mtu 4092
no interface proxy-arp 2 shutdown
ip proxy-arp
##
## Other IP configuration
##
interface ib0 ip arp timeout 60
ip route 0.0.0.0 0.0.0.0 172.18.10.1
hostname MLNX-R307
Than the best thing for you to have two types of flows, each one with different L2 priority. and configure the switch with SP to prefer one of the priority on the other.
map each priority to different traffic class on the switch.
Ophir.