Absolute Priority Scheduling

This allows us to configure jobs that increase in priority over time and set certain queues to remain at a static priority, allowing for very low priority jobs only to be run when the cluster is quite.

Set JOB_PRIORITY_OVER_TIME

• In lsb.params

MAX_USER_PRIORITY = 100
JOB_PRIORITY_OVER_TIME=5/1 # 5 increment every 1 minutes

Set up the Queues

Begin Queue
QUEUE_NAME   = normal
PRIORITY     = 30
NICE         = 20
INTERACTIVE   = NO
HOSTS         = all
DESCRIPTION  = Queue for default batch jobs
APS_PRIORITY = WEIGHT [[JPRIORITY, 1] [QPRIORITY, 10]]
End Queue

Begin Queue
QUEUE_NAME   = low
PRIORITY     = 1
NICE         = 20
INTERACTIVE   = NO
HOSTS         = all
USERS         = all
DESCRIPTION  = Low Priority Queue for batch jobs
APS_PRIORITY = WEIGHT [[QPRIORITY, 1]]
End Queue

# Note: master slave setup also possible: check jobpriorities.html
# add: QUEUE_GROUP = low, in the normal queue definition

Check APS with bjobs

With this in place I started submitted jobs as user david, 7 to the normal queue, then 6 to the low queue (with a job limit of 5)

[david@vhpchead ~]$ bjobs -aps
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME   APS
61581   david   RUN   normal     vhpchead    gpu-01-00   sleep 600  Jun  2 14:55   -
61582   david   RUN   normal     vhpchead    gpu-01-00   sleep 600  Jun  2 14:55   -
61583   david   RUN   normal     vhpchead    gpu-01-00   sleep 600  Jun  2 14:55   -
61584   david   RUN   normal     vhpchead    gpu-01-00   sleep 600  Jun  2 14:55   -
61585   david   RUN   normal     vhpchead    gpu-01-00   sleep 600  Jun  2 14:55   -
61586   david   PEND  normal     vhpchead                sleep 600  Jun  2 14:55  345.00
61587   david   PEND  normal     vhpchead                sleep 600  Jun  2 14:55  345.00
61588   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00
61589   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00
61590   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00
61591   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00
61592   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00
61593   david   PEND  low        vhpchead                sleep 600  Jun  2 14:55  1.00