Mellanox:training - Revision history

Antony: save mellanox fw from one card and upload to another

2023-11-07T17:13:15Z

save mellanox fw from one card and upload to another

David: /* Installation */

2017-09-01T14:47:22Z

Installation

Michael: /* Bad perforance? */

2014-04-07T16:08:53Z

Bad perforance?

Michael: /* Bad perforance? */

2014-04-07T15:06:38Z

Bad perforance?

Michael: Created page with "== Check the card is detected == [root@nodeA ~]# lspci | grep -i Mellanox 06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] == Me..."

2014-04-07T15:04:47Z

Created page with "== Check the card is detected == <syntaxhighlight> [root@nodeA ~]# lspci | grep -i Mellanox 06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] </syntaxhighlight> == Me..."

New page

== Check the card is detected ==

<syntaxhighlight>
[root@nodeA ~]# lspci | grep -i Mellanox
06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
</syntaxhighlight>

== Mellanox OFED ==

Don't change the kernel. Its build for the default version of the kernel.

If the kernel changes MLNX_OFED must be rebuilt for the running kernel

== Installation ==

There are a number of options available. To see them all run:

<syntaxhighlight>
./mlnxofedinstall --l
</syntaxhighlight>

<syntaxhighlight>
yum install tcl tk libnl-devel gcc-gfortran
./mlnxofedinstall
</syntaxhighlight>

It will try to update the firmware at the end of the install:

<syntaxhighlight>
Device #1:
----------

Device: 0000:06:00.0
Part Number:
Description:
PSID: MT_1060110019

Versions: Current Available
FW 2.10.0000 N/A

Status: No matching image found
</syntaxhighlight>

== Restart the driver ==

Either reboot the node or run:

<syntaxhighlight>
/etc/init.d/openibd restart
</syntaxhighlight>

==Check the state ==

<syntaxhighlight>
[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0030:48ff:ffff:e535
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 4: PortConfigurationTraining
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
</syntaxhighlight>

== Start the Subnet manager ==

the subnet manager must be running somewhere - the switch, a node or a service

=== Start it on the swtich===

<syntaxhighlight>
IB SM Management
Base SM
SM enable
apply
</syntaxhighlight>

The state of the connection will become active and a LID will be assigned.

<syntaxhighlight>
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.10.0
Hardware version: 0
Node GUID: 0x003048ffffffe534
System image GUID: 0x003048ffffffe537
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 2
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x003048ffffffe535
Link layer: InfiniBand

</syntaxhighlight>

Each Link will have a separate GUID - these are basically the equivalent of a MAC address. They should be unique to rvery device unless someone has been messing around.

Communation is based of the LID - this

== Subnet manager ==

Only one subnet manager needs to be running. An extra instances will be used if the running on fails. If there are multiple back ups there is an election to decide who takes over.

The subnet manager assigns the LIDs and builds the routing table. This can take a while depending on how complicated the topology is.

If the SM is running on the switch it can be managed under the IB SM MGMT tab.

== Testing ==

Again there are numerous options, but they must be the same on both sides.

<syntaxhighlight>
ib_read_bw <any ip on system>
</syntaxhighlight>

<syntaxhighlight>
[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# ib_read_bw

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x02 QPN 0x0058 PSN 0xfd98dc OUT 0x10 RKey 0x001900 VAddr 0x007f247ebd0000
remote address: LID 0x03 QPN 0x0058 PSN 0xcdbe1f OUT 0x10 RKey 0x001900 VAddr 0x007f6012610000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 1000 6041.81 6037.05 0.096593
---------------------------------------------------------------------------------------
</syntaxhighlight>

ib_read_bw -a -b. If warnings stop cpuspeed, bios cpu max perf, should be around 14

ib_read_lat -a shows latnecies

== Bad perforance? ==

Verify the fabric using melanox tools. It should be version 2 or higher.

<syntaxhighlight>
ibdiagnet
</bahs>

<syntaxhighlight>
#clear the counters
ibdiagnet -pc
# run
ibdiagnet -P all=1
</syntaxhighlight>

<syntaxhighlight>
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 2 0
-I- Nodes Information 0 2
-I- Speed / Width checks 0 0
-I- Partition Keys 0 0
-I- Alias GUIDs 0 0
</syntaxhighlight>

<syntaxhighlight>
vim /var/tmp/ibdiagnet2/ibdiagnet2.log
vim /var/tmp/ibdiagnet2/ibdiagnet2.pm

</syntaxhighlight>

<syntaxhighlight>
ibdiagnet -P all=1 --ber_test --pm_pause_time 30
</syntaxhighlight>

← Older revision		Revision as of 17:13, 7 November 2023
Line 248:		Line 248:
	OEM firmwre		OEM firmwre
	supermicro		supermicro
		+
		+	== Extract Firmware from NIC and update another ==
		+
		+	You will need to do this if you have an order with mixed FW versions and they are custom SMC ones and you don't want to wait for SMC support
		+
		+	I assume both nodes have mellanox fw tools installed , ion this case both nodes are our ubunutu desktop USB boot stick which does.
		+
		+	donor node has new firmware on it we run these commands to get the device path and show version info
		+
		+	<pre>
		+	mst start
		+	root@ubuntu:~# mlxfwmanager
		+	Querying Mellanox devices firmware ...
		+
		+	Device #1:
		+	----------
		+
		+	Device Type: ConnectX4LX
		+	Part Number: Super_Micro_AOC-S25G-m2S
		+	Description: ConnectX-4 Lx EN adapter card; 25GbE dual-port SFP28; PCIe3.0 x8
		+	PSID: SM_2001000001034
		+	PCI Device Name: /dev/mst/mt4117_pciconf0
		+	Base MAC: 7cc255633c32
		+	Versions: Current Available
		+	FW 14.32.1250
		+	PXE 3.6.0502
		+	UEFI 14.25.0017
		+	</pre>
		+
		+	now we can download it and scp it to our 2nd node like this (it takes about 60s to read the FW). note this file does NOT exist yet and you should give it a sensible name
		+	<pre>
		+	flint -d /dev/mst/mt4117_pciconf0 ri mlnx_fw_14.32.1250.bin
		+	scp mlnx_fw_14.32.1250.bin 172.16.40.8:
		+	</pre>
		+
		+	now ssh into the second node and run mlxfwmanager in the same dir as you scped the file and try to udpate omit the -u to just check versions
		+	<pre>
		+	root@ubuntu:~# mlxfwmanager -u
		+	Querying Mellanox devices firmware ...
		+
		+	Device #1:
		+	----------
		+
		+	Device Type: ConnectX4LX
		+	Part Number: Super_Micro_AOC-S25G-m2S
		+	Description: ConnectX-4 Lx EN adapter card; 25GbE dual-port SFP28; PCIe3.0 x8
		+	PSID: SM_2001000001034
		+	PCI Device Name: /dev/mst/mt4117_pciconf0
		+	Base MAC: 7cc255633c32
		+	Versions: Current Available
		+	FW 14.32.1010 14.32.1250
		+	PXE 3.6.0502 3.6.0502
		+	UEFI 14.25.0017 14.25.0017
		+
		+	Status: Update required
		+
		+	---------
		+	Found 1 device(s) requiring firmware update...
		+
		+	Perform FW update? [y/N]: y
		+	Device #1: Updating FW ...
		+	FSMST_INITIALIZE - OK
		+	Writing Boot image component - OK Fail : The Digest in the signature is wrong
		+	</pre>
		+
		+	that looks bad right luckily mellanox have documented what to do in this case and its this:
		+	<pre>
		+	root@ubuntu:~# flint -i mlnx_fw_14.32.1250.bin sign
		+	Updating SIGNATURE section - OK
		+	Updating ITOC section - OK
		+	</pre>
		+
		+	after this it updates perfectly

← Older revision		Revision as of 16:08, 7 April 2014
Line 204:		Line 204:
	ibdiagnet -P all=1 --get_cable_info		ibdiagnet -P all=1 --get_cable_info
	</syntaxhighlight>		</syntaxhighlight>
		+
		+
		+	== Firmware updates ==
		+
		+	<syntaxhighlight>
		+	mst start
		+	mst status
		+	</syntaxhighlight>
		+
		+
		+	<syntaxhighlight>
		+	[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# mst status
		+	MST modules:
		+	------------
		+	MST PCI module loaded
		+	MST PCI configuration module loaded
		+
		+	MST devices:
		+	------------
		+	/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
		+	domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92
		+	Chip revision is: 01
		+	/dev/mst/mt4099_pci_cr0 - PCI direct access.
		+	domain:bus:dev.fn=0000:06:00.0 bar=0xdf900000 size=0x100000
		+	Chip revision is: 01
		+	</syntaxhighlight>
		+
		+
		+
		+	<syntaxhighlight>
		+	flint -d <dev> -i <file>
		+	</syntaxhighlight>
		+
		+
		+	== OEM firmware ==
		+
		+	mellanox.com
		+	support
		+	OEM firmwre
		+	supermicro

@@ Line 25: / Line 25: @@
 yum install tcl tk libnl-devel gcc-gfortran
 ./mlnxofedinstall
 </syntaxhighlight>
@@ Line 43: / Line 48: @@
    Status:        No matching image found
 </syntaxhighlight>
 == Restart the driver ==

@@ Line 202: / Line 202: @@
 <syntaxhighlight>
 ibdiagnet -P all=1 --ber_test --pm_pause_time 30
 </syntaxhighlight>