Pamcrash: Benchmark with Pamcrash

Running Pamcrash with platform mpi

• Pamcrash comes precomplied with platform mpi , Though other mpi could be used as well

[root@head ~]# mkdir pamcrash
[root@head ~]# cd pamcrash
[root@head pamcrash]# tar -xvf VPSolution-2015.01_Linux-em64t.tar.bz2
[root@head pamcrash]# source setpamhome.bash /root/pamcrash/  

• pamworld wrapper will located in /root/pamcrash/vpsolver/2015.01/

To run license server

1. Copy the binaries lmgrd, pam_lmd and lmuti from ESI ftp folder.
2. Copy the content of license file provides by ESI in 160708.flx

[user@blbdw1 flexlm]$ tar zxvf ../FLEXnet_v11.11.1_Linux-lsb3-x86_64.tar.gz 
lmgrd
lmutil
pam_lmd
../lmgrd -c 160708.flx > flexlm.log

# you may need to install the following if you get an error about the ELF interpreter 
yum install redhat-lsb -y 

1. .Check in flexlm.log if server is started and note if the port number

Set Licence server

# for e.g. 
# [user@blbdw1 flexlm]$ grep 'pam_lmd using' flexlm.log 
# 22:07:03 (lmgrd) pam_lmd using TCP-port 56502
export PAMLIC=CRASHSAF
export PAM_LMD_LICENSE_FILE=56502@hostname (portnumber@hostname)

Running PAMCRASH over OPA with platform mpi

• As of Feb 2016, Platform MPI (pmpi) does not yet support psm2. To run pmpi on an OPA system we need to use psm2-compat. Furthermore, when PSM is selected, pmpi checks that the system has a True Scale HCA. So, we have to modify lspci to report a True Scale HCA.

mv /usr/sbin/lspci /usr/sbin/lspci_orig
cat > /usr/sbin/lspci 
#!/bin/sh /usr/sbin/lspci_orig 
$@ echo "81:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)" 
^D 



pmpi.conf should contain:

MPI_ICLIB_PSM__PSM_MAIN = /usr/lib64/psm2-compat/libpsm_infinipath.so.1 
MPI_ICMOD_PSM__PSM_MAIN = "hfi1" 
MPI_ICLIB_PSM__PSM_PATH = /usr/lib64/psm2-compat/libpsm_infinipath.so.1
MPI_ICMOD_PSM__PSM_PATH = "hfi1" 


/root/pamcrash/vpsolver/2015.01/pamworld -np 24 -mpiext '-machinefile hosts -PSM' VW_UP05_adv2k_v12_noFPM_75.pc

NOTE:-mpiext is used to pass the option to platform mpi.

• Replace test.pc with appropriate file from VW

Running PAMCRASH with Infiniband

/root/pamcrash/vpsolver/2015.01/pamworld -np 24 -mpiext '-machinefile hosts -IBV' -arch em64t test.pc

NOTE: This uses ib verb for running. Can be force to run on -TCP as well with(IPoIB setup)