Mellanox:training
Check the card is detected
[root@nodeA ~]# lspci | grep -i Mellanox
06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Mellanox OFED
Don't change the kernel. Its build for the default version of the kernel.
If the kernel changes MLNX_OFED must be rebuilt for the running kernel
Installation
There are a number of options available. To see them all run:
./mlnxofedinstall --lyum install tcl tk libnl-devel gcc-gfortran
./mlnxofedinstallTo install against a non-running kernel
./mlnxofedinstall -k 3.10.0-514.6.1.el7.x86_64 (--without-fw-update if in chroot in OpenHPC)It will try to update the firmware at the end of the install:
Device #1:
----------
Device: 0000:06:00.0
Part Number:
Description:
PSID: MT_1060110019
Versions: Current Available
FW 2.10.0000 N/A
Status: No matching image foundRestart the driver
Either reboot the node or run:
/etc/init.d/openibd restart
Check the state
[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0030:48ff:ffff:e535
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 4: PortConfigurationTraining
rate: 10 Gb/sec (4X)
link_layer: InfiniBandStart the Subnet manager
the subnet manager must be running somewhere - the switch, a node or a service
Start it on the swtich
IB SM Management
Base SM
SM enable
applyThe state of the connection will become active and a LID will be assigned.
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.10.0
Hardware version: 0
Node GUID: 0x003048ffffffe534
System image GUID: 0x003048ffffffe537
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 2
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x003048ffffffe535
Link layer: InfiniBandEach Link will have a separate GUID - these are basically the equivalent of a MAC address. They should be unique to rvery device unless someone has been messing around.
Communation is based of the LID - this
Subnet manager
Only one subnet manager needs to be running. An extra instances will be used if the running on fails. If there are multiple back ups there is an election to decide who takes over.
The subnet manager assigns the LIDs and builds the routing table. This can take a while depending on how complicated the topology is.
If the SM is running on the switch it can be managed under the IB SM MGMT tab.
Testing
Again there are numerous options, but they must be the same on both sides.
ib_read_bw <any ip on system>[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# ib_read_bw
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Read BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x02 QPN 0x0058 PSN 0xfd98dc OUT 0x10 RKey 0x001900 VAddr 0x007f247ebd0000
remote address: LID 0x03 QPN 0x0058 PSN 0xcdbe1f OUT 0x10 RKey 0x001900 VAddr 0x007f6012610000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
65536 1000 6041.81 6037.05 0.096593
---------------------------------------------------------------------------------------
ib_read_bw -a -b. If warnings stop cpuspeed, bios cpu max perf, should be around 14
ib_read_lat -a shows latnecies
Bad perforance?
Verify the fabric using melanox tools. It should be version 2 or higher.
ibdiagnet
</bahs>
<syntaxhighlight>
#clear the counters
ibdiagnet -pc
# run
ibdiagnet -P all=1Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 2 0
-I- Nodes Information 0 2
-I- Speed / Width checks 0 0
-I- Partition Keys 0 0
-I- Alias GUIDs 0 0
vim /var/tmp/ibdiagnet2/ibdiagnet2.log
vim /var/tmp/ibdiagnet2/ibdiagnet2.pmibdiagnet -P all=1 --ber_test --pm_pause_time 30
ibdiagnet -P all=1 --get_cable_info
Firmware updates
mst start
mst status
[root@nodeB MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64]# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92
Chip revision is: 01
/dev/mst/mt4099_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:06:00.0 bar=0xdf900000 size=0x100000
Chip revision is: 01
flint -d <dev> -i <file>
OEM firmware
mellanox.com support OEM firmwre supermicro
Extract Firmware from NIC and update another
You will need to do this if you have an order with mixed FW versions and they are custom SMC ones and you don't want to wait for SMC support
I assume both nodes have mellanox fw tools installed , ion this case both nodes are our ubunutu desktop USB boot stick which does.
donor node has new firmware on it we run these commands to get the device path and show version info
mst start
root@ubuntu:~# mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX4LX
Part Number: Super_Micro_AOC-S25G-m2S
Description: ConnectX-4 Lx EN adapter card; 25GbE dual-port SFP28; PCIe3.0 x8
PSID: SM_2001000001034
PCI Device Name: /dev/mst/mt4117_pciconf0
Base MAC: 7cc255633c32
Versions: Current Available
FW 14.32.1250
PXE 3.6.0502
UEFI 14.25.0017
now we can download it and scp it to our 2nd node like this (it takes about 60s to read the FW). note this file does NOT exist yet and you should give it a sensible name
flint -d /dev/mst/mt4117_pciconf0 ri mlnx_fw_14.32.1250.bin scp mlnx_fw_14.32.1250.bin 172.16.40.8:
now ssh into the second node and run mlxfwmanager in the same dir as you scped the file and try to udpate omit the -u to just check versions
root@ubuntu:~# mlxfwmanager -u
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX4LX
Part Number: Super_Micro_AOC-S25G-m2S
Description: ConnectX-4 Lx EN adapter card; 25GbE dual-port SFP28; PCIe3.0 x8
PSID: SM_2001000001034
PCI Device Name: /dev/mst/mt4117_pciconf0
Base MAC: 7cc255633c32
Versions: Current Available
FW 14.32.1010 14.32.1250
PXE 3.6.0502 3.6.0502
UEFI 14.25.0017 14.25.0017
Status: Update required
---------
Found 1 device(s) requiring firmware update...
Perform FW update? [y/N]: y
Device #1: Updating FW ...
FSMST_INITIALIZE - OK
Writing Boot image component - OK Fail : The Digest in the signature is wrong
that looks bad right luckily mellanox have documented what to do in this case and its this:
root@ubuntu:~# flint -i mlnx_fw_14.32.1250.bin sign Updating SIGNATURE section - OK Updating ITOC section - OK
after this it updates perfectly