Michael: Created page with "===== Job Deferred ===== * Run checkjob to see why the job is deferred checkjob job_id [root@fotcluster2 ~]# checkjob 3177 ... 'job is deferred. Reason: NoResources ..."

2013-05-13T10:00:29Z

Created page with "===== Job Deferred ===== * Run checkjob to see why the job is deferred <syntaxhighlight> checkjob job_id [root@fotcluster2 ~]# checkjob 3177 ... 'job is deferred. Reason: NoResources ..."

New page

===== Job Deferred =====
* Run checkjob to see why the job is deferred
<syntaxhighlight>
checkjob job_id

[root@fotcluster2 ~]# checkjob 3177
...
'job is deferred. Reason: NoResources (cannot create reservation for job 3177 (intital reservation attempt)'
...
</syntaxhighlight>

* Diagnose the job
<syntaxhighlight>
diagnose -j job_id
diagnose -q
</syntaxhighlight>

* Check the state of the compute nodes
<syntaxhighlight>
checknode -v compute-0-0
</syntaxhighlight>

* Check the torque server logs
<syntaxhighlight>
vi /opt/torque/server_logs/YYYYMMDD
</syntaxhighlight>

* Check the maui server logs
<syntaxhighlight>
vi /opt/maui/log/maui.log
</syntaxhighlight>

* If you have added properties/resources to the pbs nodes file, ensure this has not been overwrote by 'rocks sync config'
<syntaxhighlight>
pbsnodes -a [ensure correct host properties]
/opt/torque/server_priv/nodes
</syntaxhighlight>

* Problem solved? Re-queue the deferred job:
<syntaxhighlight>
runjob -c jobid
releasehold jobid
</syntaxhighlight>

===== Cannot set hostlist =====
* Experienced the following: job cannot be started - cannot set hostlist
* From rocks list: indicates that maui hasn't got the privileges it needs to do the scheduling

* Verify psb config and check
<syntaxhighlight>
qmgr -c "print server"
# your looking for the following line:
set server managers = maui@clustername
set server managers += root@clustername
</syntaxhighlight>

* You can reset the queue configuration using the following:
<syntaxhighlight>
/opt/torque/bin/qterm -t quick
/opt/torque/sbin/pbs_server -t create
(answer yes)
/opt/torque/bin/qmgr < /opt/torque/pbs.default

# /opt/torque/pbs.default contains the default setup for the pbs-roll.
# note: had to create server_priv/nodes files manually
</syntaxhighlight>

* Ended up being a problem with UPPER-CASE letters in the FQDN, avoid!
<syntaxhighlight>
qmgr obj= svr=default: Bad ACL entry in host list MSG=First bad host:
LRC-PS8-RFHEAD
qmgr obj= svr=default: Bad ACL entry in host list MSG=First bad host:
LRC-PS8-RFHEAD
</syntaxhighlight>

Rocks: Jobs deferred in torque/maui - Revision history

Michael: Created page with "===== Job Deferred ===== * Run checkjob to see why the job is deferred checkjob job_id [root@fotcluster2 ~]# checkjob 3177 ... 'job is deferred. Reason: NoResources ..."