Michael: Created page with "* Note, see the lsf.shared bug at bottom. Changes not maintained in PCM 3.0 (and previous) ===== Multicluster License ===== * Ensure your license includes a line with lsf_mul..."

2013-05-01T09:31:14Z

Created page with "* Note, see the lsf.shared bug at bottom. Changes not maintained in PCM 3.0 (and previous) ===== Multicluster License ===== * Ensure your license includes a line with lsf_mul..."

New page

* Note, see the lsf.shared bug at bottom. Changes not maintained in PCM 3.0 (and previous)

===== Multicluster License =====
* Ensure your license includes a line with lsf_multicluster, otherwise request from platform
<syntaxhighlight>
FEATURE lsf_multicluster lsf_ld 7.000 31-JUL-2011 0 AD3E3C81D267A3C1C0B6 "Platform" DEMO
</syntaxhighlight>

===== Configuration Files =====
The configuration below:
* uses two PCM 3.0 clusters (pcm30, pcm-mctest)
* pcm30 will forward jobs on to pcm-mctest
* pcm-mctest will receive jobs from pcm30
* All change were made in /etc/cfm/templates/lsf/

* '''lsf.cluster''' (or default.lsf.cluster), two additions RemoteCluster and PRODUCTS
<syntaxhighlight>
# Update the PRODUCT line to include multicluster
PRODUCTS=LSF_Base LSF_Manager LSF_MultiCluster
</syntaxhighlight>

<syntaxhighlight>
# drop in an the end of the file
# dp multicluster, note: names are the cluster names as defined by LSF (typically hostname_cluster1)
Begin RemoteClusters
CLUSTERNAME
pcm30_cluster1
pcm-mctest_cluster1
End RemoteClusters
</syntaxhighlight>

* '''lsf.shared (or default.lsf.shared)
<syntaxhighlight>
# Note: replaced XXX_clustername_XXX with the cluster name. Guess this is ok provided the cluster name doesnt change
Begin Cluster
ClusterName Servers
pcm30_cluster1 pcm30
pcm-mctest_cluster1 pcm-mctest
End Cluster
###### NOTE PROBLEM, INFO NOT SYNCd AFTER ADDHOST -U ###########
###### SEE RESOLUTION at the bottom of this page ###########
</syntaxhighlight>

* '''lsf.conf''' (or default.lsf.conf)
<syntaxhighlight>
# Multicluster enable, append to end of file
MC_PLUGIN_REMOTE_RESOURCE=y
</syntaxhighlight>

===== Multicluster Model =====
* '''Job forwarding model'''
In this model, the cluster that is starving for resources sends jobs over to the cluster that has resources to spare. To work
together, two clusters must set up compatible send-jobs and receive-jobs queues.
With this model, scheduling of MultiCluster jobs is a process with two scheduling phases: the submission cluster selects
a suitable remote receive-jobs queue, and forwards the job to it; then the execution cluster selects a suitable host and
dispatches the job to it. This method automatically favors local hosts; a MultiCluster send-jobs queue always attempts
to find a suitable local host before considering a receive-jobs queue in another cluster.

* '''Resource leasing model'''
In this model, the cluster that is starving for resources takes resources away from the cluster that has resources to spare.
To work together, the provider cluster must export resources to the consumer, and the consumer cluster must
configure a queue to use those resources.
In this model, each cluster schedules work on a single system image, which includes both borrowed hosts and local
hosts.
* '''Choosing a model'''
** Consider your own goals and priorities when choosing the best resource-sharing model for your site.
** The job forwarding model can make resources available to jobs from multiple clusters, this flexibility allows maximum throughput when each clusters resource usage fluctuates.
** The resource leasing model can allow one cluster exclusive control of a dedicated resource, this can be more efficient when there is a steady amount of work.
** The lease model is the most transparent to users and supports the same scheduling features as a single cluster.
** The job forwarding model has a single point of administration, while the lease model shares administration between provider and consumer clusters.

===== Job Forwarding Model =====
* '''lsb.queues''' (or lsbatch/default/configdir/lsb.queues)
* On the host sending jobs, create a queue with SNDJOBS_TO (pcm30)
<syntaxhighlight>
Begin Queue
QUEUE_NAME = sendq
PRIORITY = 40
HOSTS = none
SNDJOBS_TO = receiveq@pcm-mctest_cluster1
End Queue
</syntaxhighlight>

* On the cluster receiving jobs, create a queue with RCVJOBS_FROM (pcm-mctest)
<syntaxhighlight>
Begin Queue
QUEUE_NAME = receiveq
RCVJOBS_FROM = sendq@pcm30_cluster1
HOSTS = all
End Queue
</syntaxhighlight>

===== Resource Sharing Model =====
* In this example, cluster pcmtest is exporting a single node to vhpchead
* '''lsb.resources''' file on '''pcmtest'''
<syntaxhighlight>
Begin HostExport
PER_HOST = pcmcomp000 # export host list
SLOTS = 12 # for each host, export 5 job slots
DISTRIBUTION = [vhpchead_cluster1, 6] # share distribution for remote clusters:
# cluster <vhpchead_cluster1> has 6 shares,
End HostExport
</syntaxhighlight>

* '''lsb.queues''' file on '''vhpchead'''
<syntaxhighlight>
# resource borrow queue
Begin Queue
QUEUE_NAME = resourceborrowq
PRIORITY = 40
HOSTS = compute005 pcmcomp000@pcm30_cluster1 # 2 hosts on this queue, one remote host pcmcomp000
DESCRIPTION = Resource Borrow Queue
End Queue
</syntaxhighlight>

* Verify jobs are being run correctly
<syntaxhighlight>
[root@pcmcomp000 ~]# bclusters
[Job Forwarding Information ]
LOCAL_QUEUE JOB_FLOW REMOTE CLUSTER STATUS
receiveq recv - vhpchead_c ok

[Resource Lease Information ]
REMOTE_CLUSTER RESOURCE_FLOW STATUS
vhpchead_cluste EXPORT ok
# Check the hosts that are being exported:
[root@pcmcomp000 ~]# bhosts -e
HOST_NAME MAX NJOBS RUN SSUSP USUSP RSV
pcmcomp000 12 3 3 0 0 0
</syntaxhighlight>

* Also check output from vhpchead
<syntaxhighlight>
[david@vhpchead multicluster]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
21472 david RUN resourcele vhpchead compute005 sleep 60 Aug 3 17:20
21474 david RUN resourcele vhpchead compute005 sleep 60 Aug 3 17:20
21476 david RUN resourcele vhpchead compute005 sleep 60 Aug 3 17:20
21473 david RUN resourcele vhpchead pcmcomp000@ sleep 60 Aug 3 17:20
21475 david RUN resourcele vhpchead pcmcomp000@ sleep 60 Aug 3 17:20
21477 david PEND resourcele vhpchead sleep 60 Aug 3 17:20

[david@vhpchead multicluster]$ bclusters
[Job Forwarding Information ]
LOCAL_QUEUE JOB_FLOW REMOTE CLUSTER STATUS
sendq send receiveq pcm30_clus ok

[Resource Lease Information ]
REMOTE_CLUSTER RESOURCE_FLOW STATUS
pcm30_cluster1 IMPORT ok

[david@vhpchead ~]$ bhosts -w
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
compute000 ok - 12 0 0 0 0 0
compute001 ok - 12 0 0 0 0 0
compute002 ok - 24 0 0 0 0 0
compute003 ok - 8 0 0 0 0 0
compute004 ok - 8 0 0 0 0 0
compute005 ok - 8 0 0 0 0 0
compute007 ok - 8 0 0 0 0 0
compute008 ok - 8 0 0 0 0 0
compute009 ok - 8 0 0 0 0 0
pcmcomp000@pcm30_cluster1 ok - 12 0 0 0 0 0
vhpchead ok - 8 0 0 0 0 0
</syntaxhighlight>

===== Update System =====
<syntaxhighlight>
addhost -u
# Or if you edited files in /opt/lsf/conf
lsadmin reconfig
badmin mbdrestart
</syntaxhighlight>
* Note, if the configuration doesn't apply correctly run the '''lsadmin''' and '''badmin''' commands listed above to verify the configuration files (addhost -u does not report configuration errors correctly!)

===== Setup IPtables =====
* The cluster LSF processes will try and communication, by default only ssh traffic is allowed on eth1, update iptables on both servers
<syntaxhighlight>
# Generated by iptables-save v1.3.5 on Fri Jul 1 17:40:27 2011
*nat
:PREROUTING ACCEPT [1339:165189]
:POSTROUTING ACCEPT [205:14830]
:OUTPUT ACCEPT [516:36221]
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
# Completed on Fri Jul 1 17:40:27 2011
# Generated by iptables-save v1.3.5 on Fri Jul 1 17:40:27 2011
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [43133:352914090]
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
-A INPUT -i eth1 -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 873 -j ACCEPT
-A INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 5432 -j ACCEPT
# multicluster
-A INPUT -i eth1 --source 172.28.10.0/24 -p tcp -m state --state NEW -m tcp --dport 7869 -j ACCEPT
-A INPUT -i eth1 --source 172.28.10.0/24 -p tcp -m state --state NEW -m tcp --dport 6878 -j ACCEPT
-A INPUT -i eth1 --source 172.28.10.0/24 -p tcp -m state --state NEW -m tcp --dport 6881 -j ACCEPT
-A INPUT -i eth1 --source 172.28.10.0/24 -p tcp -m state --state NEW -m tcp --dport 6882 -j ACCEPT
# end multicluster
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
COMMIT
# Completed on Fri Jul 1 17:40:27 2011

</syntaxhighlight>

===== /etc/hosts - add clusters =====
* On each node, add the cluster headnode to the hosts file (if not using external DNS which can resolve both hostname)
* As these are cluster '''external''' hosts, add to '''/etc/hosts/append'''
<syntaxhighlight>
# /etc/hosts.append on pcm-mctest
172.28.10.69 pcm30.viglen.co.uk pcm30
</syntaxhighlight>

* Then update the hosts file and sync across cluster
<syntaxhighlight>
kusu-genconfig hosts > /etc/hosts
cfmsync -f
</syntaxhighlight>

===== Check MultiCluster Status =====
* Use '''bclusters''' and '''lsclusters'''
* Status should be '''ok''', if you see '''disc''' there may be some communication problems
<syntaxhighlight>
root@pcm-mctest lsf]# bclusters
[Job Forwarding Information ]
LOCAL_QUEUE JOB_FLOW REMOTE CLUSTER STATUS
receiveq recv - pcm30_clus ok

[Resource Lease Information ]
No resources have been exported or borrowed
</syntaxhighlight>

<syntaxhighlight>
[root@pcm-mctest lsf]# lsclusters
CLUSTER_NAME STATUS MASTER_HOST ADMIN HOSTS SERVERS
pcm-mctest_clu ok pcm-mctest hpcadmin 2 2
pcm30_cluster1 ok pcmtest.viglen.co. hpcadmin 2 2
</syntaxhighlight>

===== Move Files between Clusters =====
* LSF will use lsrcp
* Need to setup both cluster to use SSH for [lsrcp|rsh|rcp], replace/create link for the binaries on the headnode. Create these files in /etc/cfm/[compute-group]
* All need to ensure ssh keys are setup between the clusters (passwordless access)
<syntaxhighlight>
# Either change lsrcp:
/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/lsrcp -> [scp] #mkdir /etc/cfm/compute-centos-5.6-x86_64/$LSF_BINDIR

# Or change rcp (the default lsrcp will fall back on rcp)
/usr/kerberos/bin/rsh -> [ssh]
/usr/kerberos/bin/rcp -> [scp]

# ssh keys
cat ~/.ssh/id_rsa.pub | ssh user@remote.machine.com 'cat >> .ssh/authorized_keys'
</syntaxhighlight>

* NOTE: Output files created on remote cluster at not automatically copied back
<syntaxhighlight>
# Sample scripts that copied an input file across, and then all output files back.
#BSUB -q sendq
#BSUB -o sendq_output.%J.txt
#BSUB -e sendq_error.%J.txt
#BSUB -f "/home/david/test_input.inp > /home/david/copied_across.inp"
#BSUB -f "/home/david/result_copied.out < /home/david/result.out"
#BSUB -f "/home/david/sendq_output_copied.%J.txt < /home/david/sendq_output.%J.txt"
#BSUB -f "/home/david/sendq_error_copied.%J.txt < /home/david/sendq_error.%J.txt"

echo "hi"
hostname
id
cat /home/david/copied_across.inp

hostname >> result.out
id >> result.out

sleep 30
</syntaxhighlight>

===== lsf.shared bug =====
/etc/cfm/templates/default.lsf.shared '''Cluster''' section gets overwrote on sync. The following changes need to be made:
<syntaxhighlight>
vi /opt/kusu/lib/plugins/genconfig/lsfshared_7_0_6.py

# Change from:
84 if re.compile("^End.*Cluster").search(instr):
85 inClusterSection = False
86 else:
87 if re.compile("^ClusterName").search(instr):
88 pass
89 else:
90 if inClusterSection:
91 print clusterName
92 continue

# Change to:
84 if re.compile("^End.*Cluster").search(instr):
85 inClusterSection = False
86 else:
87 if re.compile("^ClusterName").search(instr):
88 pass
89 else:
90 if inClusterSection and re.compile("^XXX_clustername_XXX").search(instr): # <---- This line!
91 print clusterName
92 continue
93
94 print instr,
</syntaxhighlight>

* Verify the update has been applied correctly:
<syntaxhighlight>
kusu-genconfig lsfshared_7_0_6 'insert-cluster-name'
e.g: kusu-genconfig lsfshared_7_0_6 pcm30_cluster1

# once confirmed as updating correctly:
addhost -u
</syntaxhighlight>

LSF Multicluster - Revision history

Michael: Created page with "* Note, see the lsf.shared bug at bottom. Changes not maintained in PCM 3.0 (and previous) ===== Multicluster License ===== * Ensure your license includes a line with lsf_mul..."