Linux: Checking the Infiniband Fabric - Revision history

David at 11:23, 16 May 2017

2017-05-16T11:23:05Z

Michael: /* FDR MPI Performance Figures using fibre cable */

2014-02-13T09:04:13Z

FDR MPI Performance Figures using fibre cable

Michael: /* FDR MPI Performance Figure */

2014-02-13T08:58:54Z

FDR MPI Performance Figure

Michael: /* QDR MPI Performance Figures */

2014-02-13T08:53:48Z

QDR MPI Performance Figures

Michael at 13:10, 4 January 2013

2013-01-04T13:10:13Z

David at 12:59, 8 November 2012

2012-11-08T12:59:08Z

David at 12:55, 8 November 2012

2012-11-08T12:55:15Z

David: Created page with "* Assuming Platform Cluster Manager 3.2 is installed, otherwise please make sure the latest version of OFED which will include openmpi == Check the IB links == Use a command ..."

2012-11-08T12:50:14Z

Created page with "* Assuming Platform Cluster Manager 3.2 is installed, otherwise please make sure the latest version of OFED which will include openmpi == Check the IB links == Use a command ..."

New page

* Assuming Platform Cluster Manager 3.2 is installed, otherwise please make sure the latest version of OFED which will include openmpi

== Check the IB links ==
Use a command called <tt>ibstatus</tt> to the current state of the IB link
<syntaxhighlight>
david@compute000 imb]$ ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0030:48ff:ffff:e57d
base lid: 0x0
sm lid: 0x0
state: 2: INIT
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
</syntaxhighlight>

In this instance we can see that the <tt>state:</tt> is only in an INIT stage. This typically means that the IB link is having trouble with the subnet manager. This will result in warning where running MPI performance tests (check the output from openmpi mpirun for clues:

<syntaxhighlight>
WARNING: There is at least one OpenFabrics device found but there are
no active ports detected (or Open MPI was unable to use them). This
is most certainly not what you wanted. Check your cables, subnet
manager configuration, etc. The openib BTL will be ignored for this
job.
</syntaxhighlight>

OpenMPI will default back to using Ethernet, you can tell by the high latency and low bandwidth:
<syntaxhighlight>
david@compute000 imb]$ module load openmpi-x86_64
[david@compute000 imb]$ which mpirun
/usr/lib64/openmpi/bin/mpirun
[david@compute000 imb]$ pwd
/home/david/benchmarks/imb
[david@compute000 imb]$ cat hosts
compute000
compute001
[david@compute000 imb]$ /usr/lib64/openmpi/bin/mpirun -np 2 -hostfile ./hosts /usr/lib64/openmpi/bin/mpitests-IMB-MPI1
# lots of warning cut out
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 47.79 0.00 # <-- This is high ethernet latency, typical 1GB eth0 latency can be as low as 25usec
1 1000 44.85 0.02
2 1000 45.24 0.04
4 1000 45.87 0.08
8 1000 44.51 0.17
16 1000 43.21 0.35
32 1000 43.76 0.70
64 1000 43.92 1.39
128 1000 43.48 2.81
256 1000 48.91 4.99
512 1000 52.95 9.22
1024 1000 96.30 10.14
2048 1000 403.23 4.84
4096 1000 262.84 14.86
8192 1000 279.54 27.95
16384 1000 333.65 46.83
32768 1000 686.98 45.49
65536 640 1364.94 45.79
131072 320 1668.31 74.93
262144 160 2683.26 93.17
524288 80 5044.39 99.12
1048576 40 9498.91 105.28
2097152 20 18256.90 109.55
4194304 10 36169.60 110.59 # <-- Typical 1GB bandwidth
</syntaxhighlight>

@@ Line 92: / Line 92: @@
 == QDR MPI Performance Figures ==
 Here are some figures for QDR IB (FDR cards, running at QDR speed because of switch)
 <syntaxhighlight>

@@ Line 127: / Line 127: @@
 == FDR MPI Performance Figures using fibre cable ==
 <syntaxhighlight>
@@ Line 160: / Line 162: @@
 </syntaxhighlight>
 == FDR MPI Performance Figure ==

@@ Line 123: / Line 123: @@
        2097152           20       574.40      3481.87
        4194304           10      1144.80      3494.05
 </syntaxhighlight>

← Older revision		Revision as of 08:53, 13 February 2014
Line 125:		Line 125:
	</syntaxhighlight>		</syntaxhighlight>

		+
		+	== FDR MPI Performance Figure ==
		+
		+	Results of IB running at full FDR
		+
		+	<syntaxhighlight>
		+	#---------------------------------------------------
		+	# Benchmarking PingPong
		+	# #processes = 2
		+	#---------------------------------------------------
		+	#bytes #repetitions t[usec] Mbytes/sec
		+	0 1000 1.14 0.00
		+	1 1000 1.20 0.79
		+	2 1000 1.14 1.67
		+	4 1000 1.14 3.36
		+	8 1000 1.16 6.56
		+	16 1000 1.17 13.08
		+	32 1000 1.19 25.65
		+	64 1000 1.23 49.70
		+	128 1000 1.55 78.63
		+	256 1000 1.68 145.28
		+	512 1000 1.85 263.73
		+	1024 1000 2.18 447.45
		+	2048 1000 2.84 686.61
		+	4096 1000 3.48 1123.92
		+	8192 1000 4.90 1595.05
		+	16384 1000 6.72 2323.44
		+	32768 1000 9.46 3303.35
		+	65536 640 16.01 3902.81
		+	131072 320 27.94 4473.51
		+	262144 160 45.63 5479.44
		+	524288 80 87.02 5745.92
		+	1048576 40 169.46 5901.03
		+	2097152 20 335.03 5969.69
		+	4194304 10 668.45 5983.96
		+	</syntaxhighlight>

	== Going too fast==		== Going too fast==

← Older revision		Revision as of 13:10, 4 January 2013
Line 124:		Line 124:
	4194304 10 1144.80 3494.05		4194304 10 1144.80 3494.05
	</syntaxhighlight>		</syntaxhighlight>
		+
		+
		+	== Going too fast==
		+
		+	To check that it is running between multiple nodes check the latency and bandwidth achieved.
		+
		+	Latencies lower than 1 or bandwidths higher than 4000 would suggest another issue as these speeds are higher than would be expected for infiniband. It is likely that the processes are running on the same node.

← Older revision		Revision as of 12:59, 8 November 2012
Line 92:		Line 92:

	== QDR MPI Performance Figures ==		== QDR MPI Performance Figures ==
		+	Here are some figures for QDR IB (FDR cards, running at QDR speed because of switch)
		+	<syntaxhighlight>
		+	#---------------------------------------------------
		+	# Benchmarking PingPong
		+	# #processes = 2
		+	#---------------------------------------------------
		+	#bytes #repetitions t[usec] Mbytes/sec
		+	0 1000 1.29 0.00
		+	1 1000 1.15 0.83
		+	2 1000 1.16 1.65
		+	4 1000 1.16 3.30
		+	8 1000 1.17 6.50
		+	16 1000 1.19 12.79
		+	32 1000 1.22 24.97
		+	64 1000 1.25 48.93
		+	128 1000 1.85 66.03
		+	256 1000 1.96 124.60
		+	512 1000 2.15 227.37
		+	1024 1000 2.50 390.62
		+	2048 1000 2.90 673.74
		+	4096 1000 3.68 1061.62
		+	8192 1000 5.36 1457.39
		+	16384 1000 7.81 1999.63
		+	32768 1000 12.21 2560.43
		+	65536 640 20.84 2999.41
		+	131072 320 38.02 3288.14
		+	262144 160 75.01 3332.78
		+	524288 80 146.31 3417.37
		+	1048576 40 289.19 3457.94
		+	2097152 20 574.40 3481.87
		+	4194304 10 1144.80 3494.05
		+	</syntaxhighlight>

@@ Line 15: / Line 15: @@
 </syntaxhighlight>
-In this instance we can see that the <tt>state:</tt> is only in an INIT stage. This typically means that the IB link is having trouble with the subnet manager. This will result in warning where running MPI performance tests (check the output from openmpi mpirun for clues:
+In this instance we can see that the <tt>state:</tt> is only in an INIT stage. This typically means that the IB link is having trouble with the subnet manager.
 <syntaxhighlight>
@@ Line 25: / Line 29: @@
 </syntaxhighlight>
 OpenMPI will default back to using Ethernet, you can tell by the high latency and low bandwidth:
 <syntaxhighlight>
@@ Line 67: / Line 72: @@
        4194304           10     36169.60       110.59   # <-- Typical 1GB bandwidth
 </syntaxhighlight>