Setup an Interactive Queue - Revision history

Michael at 10:04, 13 May 2013

2013-05-13T10:04:42Z

Michael: Blanked the page

2012-12-06T16:09:27Z

Blanked the page

Michael: Created page with "This example covers adding IB as a Boolean resource to LSF: Type Boolean: Either a nodes has it (1) or doesnt (0) ===== Add the Resource to LSF ===== * In file /etc/cfm/te..."

2012-12-06T16:09:11Z

Created page with "This example covers adding IB as a Boolean resource to LSF: Type Boolean: Either a nodes has it (1) or doesnt (0) ===== Add the Resource to LSF ===== * In file /etc/cfm/te..."

New page

This example covers adding IB as a Boolean resource to LSF:

Type Boolean: Either a nodes has it (1) or doesnt (0)

===== Add the Resource to LSF =====
* In file /etc/cfm/templates/lsf/default.lsf.shared, add to following line to the '''RESOURCE''' stanza.
<pre>
Begin Resource
RESOURCENAME TYPE INTERVAL INCREASING DESCRIPTION # Keywords
...
ib Boolean () () (Infiniband Availability)
End Resource
</pre>

===== Assign Resource to Hosts =====
* In file /etc/cfm/templates/lsf/default.lsf.cluster, add the following after the '''XXX_lsfmc_XXX''' tag. NOTE: Nodes with the IB tag have to be listed individually.
<pre>
Begin Host
HOSTNAME model type server r1m mem swp RESOURCES #Keywords
...
XXX_lsfmc_XXX ! ! 1 3.5 () () (mg)
cn109 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)
cn110 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)
cn111 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)
cn112 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)
...
End Host
</pre>

===== Restart LSF =====
<pre>
addhost -u
</pre>

===== Submit a job requesting the resource =====
<pre>
#BSUB -R 'select[ib]'

Sample IMB Script
#BSUB -n 48
#BSUB -R 'select[ib]'
#BSUB -R 'span[ptile=12]'
#BSUB -R 'cu[maxcus=1]'
#BSUB -o IB.%J.out
#BSUB -e IB.%J.err
#BSUB -J v_IB
#BSUB -L /bin/bash

. /etc/profile
module load PMPI/modulefile

mpirun -prot -np $LSB_DJOB_NUMPROC /home/viglen/scratch/IMB_3.0/bin/imb-pmpi
</pre>

← Older revision	Revision as of 10:04, 13 May 2013
Line 1:	Line 1:
	+	===== Setup the queue =====
	+	* lsb.queues file
	+	* once updated, run addhost -u to sync
	+	* bqueues to check the available queues
	+	<syntaxhighlight>
	+	# Example conf, administrators/priority/hosts will differ from cluster to cluster
	+	Begin Queue
	+	QUEUE_NAME=interactive
	+	DESCRIPTION=default for interactive jobs
	+	ADMINISTRATORS=user2 userGroupA
	+	PRIORITY=80
	+	INTERACTIVE=ONLY
	+	NEW_JOB_SCHED_DELAY=0
	+	HOSTS=hostGroupB+5 hostGroupA+2 others
	+	End Queue
	+	</syntaxhighlight>

	+	===== Submit an interactive job =====
	+	<syntaxhighlight>
	+	bsub -Ip -XF bash -l
	+	</syntaxhighlight>
	+
	+	===== X-forwarding not working =====
	+	* Ensure xorg-x11-xauth and xterm are installed
	+	<syntaxhighlight>
	+	# use ngedit to select the following pacakges
	+	- xorg-x11-xauth
	+	- xterm
	+	</syntaxhighlight>

← Older revision		Revision as of 16:09, 6 December 2012
Line 1:		Line 1:
−	~~This example covers adding IB as a Boolean resource to LSF:~~

−	~~Type Boolean: Either a nodes has it (1) or doesnt (0)~~
−
−
−	~~===== Add the Resource to LSF =====~~
−	* In file /etc/cfm/templates/lsf/default.lsf.shared, add to following line to the '''RESOURCE''' stanza.
−	~~<pre>~~
−	~~Begin Resource~~
−	~~RESOURCENAME TYPE INTERVAL INCREASING DESCRIPTION # Keywords~~
−	~~...~~
−	~~ib Boolean () () (Infiniband Availability)~~
−	~~End Resource~~
−	~~</pre>~~
−
−	~~===== Assign Resource to Hosts =====~~
−	* In file /etc/cfm/templates/lsf/default.lsf.cluster, add the following after the '''XXX_lsfmc_XXX''' tag. NOTE: Nodes with the IB tag have to be listed individually.
−	~~<pre>~~
−	~~Begin Host~~
−	~~HOSTNAME model type server r1m mem swp RESOURCES #Keywords~~
−	~~...~~
−	~~XXX_lsfmc_XXX ! ! 1 3.5 () () (mg)~~
−	~~cn109 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)~~
−	~~cn110 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)~~
−	~~cn111 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)~~
−	~~cn112 ! ! 1 3.5 () () (openmpi mpich2 mvapich ib)~~
−	~~...~~
−	~~End Host~~
−	~~</pre>~~
−
−	~~===== Restart LSF =====~~
−	~~<pre>~~
−	~~addhost -u~~
−	~~</pre>~~
−
−	~~===== Submit a job requesting the resource =====~~
−	~~<pre>~~
−	~~#BSUB -R 'select[ib]'~~
−
−	~~Sample IMB Script~~
−	~~#BSUB -n 48~~
−	~~#BSUB -R 'select[ib]'~~
−	~~#BSUB -R 'span[ptile=12]'~~
−	~~#BSUB -R 'cu[maxcus=1]'~~
−	~~#BSUB -o IB.%J.out~~
−	~~#BSUB -e IB.%J.err~~
−	~~#BSUB -J v_IB~~
−	~~#BSUB -L /bin/bash~~
−
−	~~. /etc/profile~~
−	~~module load PMPI/modulefile~~
−
−	~~mpirun -prot -np $LSB_DJOB_NUMPROC /home/viglen/scratch/IMB_3.0/bin/imb-pmpi~~
−	~~</pre>~~