View source for Rocks: Build OFED 1.5.x on Rocks 5.3 cluster

==== Install OFED 1.5.x on a Rocks 5.3 cluster ====

* For Viglen ofed build use: ./install.pl --hpc --prefix /opt/ofed/1.5.1
* Once build, install on compute node, assume ofed 1.5.x build on /share/apps/ofed/1.5.x:
<syntaxhighlight>
- cd /share/apps/ofed/1.5.x
- ./install.pl -l /opt/ofed/1.5.x --hpc -vvv
</syntaxhighlight>
* Full ofed build can be found in:
<syntaxhighlight>
https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster
</syntaxhighlight>

Note:
<syntaxhighlight>
broken header issue with the RHEL 2.6.18_194 kernel and the OFED 1.5.1 distribution.
You will get an an fatal error when building ofa_kernel rpm. 
</syntaxhighlight>

Build OFED 1.5.1 with Kernel 2.6.18-194.el5 (Centos 5.5)
<syntaxhighlight>
remove all qlogic "qib" references in intsall.pl and ofed.conf

Note: ofed.conf is generated by install.pl
</syntaxhighlight>

===== Build Notes =====
* Building with intel/pgi
** Noticed a problem building the libibverbs RPMs with icc. Build OFED just with gcc initially, then load the icc environment and run the install script again. Install script should only then build the outstanding MPIs using icc. Example below:
 <syntaxhighlight>
 Set PATH to point to ICC/IFORT compiler or PGI C/Fortran compiler
 in .bashrc:
  - source /share/apps/intel/bin/compilerbars.sh intel64
  - export INTEL_LICENSE_FILE=/share/apps/intel/licenses

 In ofed.conf, add:
  - mvapich_intel=y
  - mvapich2_intel=y
  - openmpi_intel=y
  - mpitest_mvapich_intel=y
  - mpitest_mvapich2_intel=y
  - mpitest_openmpi_intel=y

 In $OFED_BUILD_DIR (where ofed.conf located)
  - ./install.pl -c ofed.conf --prefix /opt/ofed/1.5.2

 Note:
 For some reason (reason unknow), mpitests_mvapich2_intel-3.2-923.x86_64.rpm
 failed to install and complaining on missing libraries dependencies:
 - libimf.so()(64bit)
 - linintlc.so.5()(64bit)
 - libsvml.so()(64bit)

 To let the install.pl to continue build and install the remaining package, temporary
 set mpitests_mvpich2_intel=n in ofed.fong file and re-run:
   ./install.pl -c ofed.conf --prefix /opt/ofed/1.5.2

 Install mpitests_mvapich2_intel-3.2-923.x86_64.rpm with using --nodeps flag
 ie:
     rpm --nodeps -iv mpitests_mvapich2_intel-3.2-923.x86_64.rpm
 </syntaxhighlight>

** Make sure FC/CC are not set in the modules file or some of the MPIs wont build (mvapcih buidls using F77='ifort -i-dynamic')

===== Successful Builds =====
* Kernel 2.6.18-128 and OFED 1.5.2 (GCC, then Intel for additional MPIs)
* Kernel 2.6.18.194.3.1 and OFED 1.5.2 (SL, GCC only)
* Kernel 2.6.18-17.4.el5 and OFED 1.5.2 (GCC only, Rocks 5.4)

===== Cheating the install =====
* Can also setup a post install script to install ofed. 
* Once the build is completed from the steps above, copy over the OFED directory to /share/apps
* Create a rocksconfig,d script to install.pl script, in the <post> section of the extend-compute.xml:
<php>
<file name="/etc/rc.d/rocksconfig.d/post-99-ofedinstall.sh" perms="0700">
#!/bin/bash

# Installation script for Infiniband acroos compute nodes
cd /share/apps/ofed/OFED-1.5.2/ 
./install.pl -l /opt/ofed/1.5.2 --hpc -vvv
service openibd restart
chkconfig --level 2345 openibd on

mv $0 /etc/rc.d/rocksconfig.d/COMPLETED-post-99-ofedinstall.sh.disabled
</file>
</php>


* '''Manually exclude packages from being built, edit ofed.conf and change to package=n'''
<syntaxhighlight>
# Use the -c option, this cannot be used in combination with --hpc|--basic|--all
./install.pl -c ./ofed.conf -l /opt/ofed/1.5.2 -vvv
</syntaxhighlight>

===== Sysfsutils Build Problem =====
* Build fails for mvapich2_gcc due to missing sysfsutils-devel package
* Install the sysfsutils from RPM built on PDD: <file>\\srv-vfs2\PDD_DATA\Product Development\High Performance Computing\HPC Software Information\OFED\sysfsutils-devel|OFED Sysfsutils RPM </file>
* sysfsutils-devel-1.2.0-1.x86_64.rpm download http://linuxsoft.cern.ch/cern/slc4X/x86_64/yum/os/repodata/repoview/sysfsutils-devel-0-1.2.0-1.html

==== OFED 1.5.2/3 Build on Rock 5.4 With SL/Centos 5.5 ====
<syntaxhighlight>
./install.pl --all --build32 --prefix /opt/ofed/1.5.[2|3]

When gets:
    " libibverbs-debuginfo was not created "
do:

mv ofed.conf ofed.conf.orig
grep -v debuginfo ofed.conf.orig > ofed.conf
./install.pl -c ofed.conf --prefix /opt/ofed/1.5.3

When gets:
     " Failed to build infinipath-psm RPM
       See /tmp/OFED.30053.logs/infinipath-psm.rpmbuild.log "
do:
cd SRPMS
rpm -iv infinipath-psm-1.14-1.src.rpm
cd /usr/src/redhat/SPECS/
rpmbuild -bb infinipath-psm.spec
cp /usr/src/redhat/RPMS/x86_64/infinipath-psm-1.14-1.x86_64.rpm /home/viglen/OFED-1.5.3/RPMS/sl-release-5.5-1/x86_64
cp /usr/src/redhat/RPMS/x86_64/infinipath-psm-devel-1.14-1.x86_64.rpm /home/viglen/OFED-1.5.3/RPMS/sl-release-5.5-1/x86_64

cd /home/viglen/OFED-1.5.3
rpm -e scsi-target-utils-0.1-20080828
./install.pl -c ofed.conf --prefix /opt/ofed/1.5.[2|3]
</syntaxhighlight>