Setup an Interactive Queue

This example covers adding IB as a Boolean resource to LSF:

Type Boolean: Either a nodes has it (1) or doesnt (0)

Add the Resource to LSF

• In file /etc/cfm/templates/lsf/default.lsf.shared, add to following line to the RESOURCE stanza.

Begin Resource
RESOURCENAME  TYPE    INTERVAL INCREASING  DESCRIPTION        # Keywords
...
ib            Boolean     ()    ()          (Infiniband Availability)
End Resource

Assign Resource to Hosts

• In file /etc/cfm/templates/lsf/default.lsf.cluster, add the following after the XXX_lsfmc_XXX tag. NOTE: Nodes with the IB tag have to be listed individually.

Begin   Host
HOSTNAME  model    type        server r1m  mem  swp  RESOURCES    #Keywords
...
XXX_lsfmc_XXX   !   !   1   3.5   ()   ()   (mg)
cn109   !       !       1       3.5     ()      ()      (openmpi mpich2 mvapich ib)
cn110   !       !       1       3.5     ()      ()      (openmpi mpich2 mvapich ib)
cn111   !       !       1       3.5     ()      ()      (openmpi mpich2 mvapich ib)
cn112   !       !       1       3.5     ()      ()      (openmpi mpich2 mvapich ib)
...
End Host

Restart LSF

addhost -u

Submit a job requesting the resource

#BSUB -R 'select[ib]'

Sample IMB Script
#BSUB -n 48
#BSUB -R 'select[ib]'
#BSUB -R 'span[ptile=12]'
#BSUB -R 'cu[maxcus=1]'
#BSUB -o IB.%J.out
#BSUB -e IB.%J.err
#BSUB -J v_IB
#BSUB -L /bin/bash

. /etc/profile
module load PMPI/modulefile

mpirun -prot -np $LSB_DJOB_NUMPROC /home/viglen/scratch/IMB_3.0/bin/imb-pmpi