Absolute job priority scheduling

Absolute job priority scheduling (APS) provides a mechanism to control the job dispatch order to prevent job starvation.

When configured in a queue, APS sorts pending jobs for dispatch according to a job priority value calculated based on several configurable job-related factors. Each job priority weighting factor can contain subfactors. Factors and subfactors can be independently assigned a weight.

APS provides administrators with detailed yet straightforward control of the job selection process.
  • APS only sorts the jobs; job scheduling is still based on configured LSF scheduling policies. LSF attempts to schedule and dispatch jobs based on their order in the APS queue, but the dispatch order is not guaranteed.

  • The job priority is calculated for pending jobs across multiple queues based on the sum of configurable factor values. Jobs are then ordered based on the calculated APS value.

  • You can adjust the following for APS factors:
    • A weight for scaling each job-related factor and subfactor

    • Limits for each job-related factor and subfactor

    • A grace period for each factor and subfactor

  • To configure absolute priority scheduling (APS) across multiple queues, define APS queue groups. When you submit a job to any queue in a group, the job's dispatch priority is calculated using the formula defined in the group's master queue.

  • Administrators can also set a static system APS value for a job. A job with a system APS priority is guaranteed to have a higher priority than any calculated value. Jobs with higher system APS settings have priority over jobs with lower system APS settings.

  • Administrators can use the ADMIN factor to manually adjust the calculated APS value for individual jobs.

Scheduling priority factors

To calculate the job priority, APS divides job-related information into several categories. Each category becomes a factor in the calculation of the scheduling priority. You can configure the weight, limit, and grace period of each factor to get the desired job dispatch order.

LSF sums the value of each factor based on the weight of each factor.

Factor weight

The weight of a factor expresses the importance of the factor in the absolute scheduling priority. The factor weight is multiplied by the value of the factor to change the factor value. A positive weight increases the importance of the factor, and a negative weight decreases the importance of a factor. Undefined factors have a weight of 0, which causes the factor to be ignored in the APS calculation.

Factor limit

The limit of a factor sets the minimum and maximum absolute value of each weighted factor. Factor limits must be positive values.

Factor grace period

Each factor can be configured with a grace period. The factor only counted as part of the APS value when the job has been pending for a long time and it exceeds the grace period.

Factors and subfactors


Factors

Subfactors

Metric

FS (user based fairshare factor)

The existing fairshare feature tunes the dynamic user priority

The fairshare factor automatically adjusts the APS value based on dynamic user priority.

FAIRSHARE must be defined in the queue. The FS factor is ignored for non-fairshare queues.

The FS factor is influenced by the following fairshare parameters defined in lsb.queues or lsb.params:

  • CPU_TIME_FACTOR

  • RUN_TIME_FACTOR

  • RUN_JOB_FACTOR

  • HIST_HOURS

RSRC (resource factors)

PROC

Requested processors is the max of bsub -n min, max, the min of bsub -n min, or the value of PROCLIMIT in lsb.queues.

MEM

Total real memory requested (in MB).

Memory requests appearing to the right of a || symbol in a usage string are ignored in the APS calculation.

For multi-phase memory reservation, the APS value is based on the first phase of reserved memory.

SWAP

Total swap space requested (in MB).

As with MEM, swap space requests appearing to the right of a || symbol in a usage string are ignored.

WORK (job attributes)

JPRIORITY

The job priority specified by:

  • Default specified by MAX_USER_PRIORITY in lsb.params

  • Users with bsub -sp or bmod -sp

  • Automatic priority escalation with JOB_PRIORITY_OVER_TIME in lsb.params

QPRIORITY

The priority of the submission queue.

ADMIN

Administrators use bmod -aps to set this subfactor value for each job. A positive value increases the APS. A negative value decreases the APS. The ADMIN factor is added to the calculated APS value to change the factor value.The ADMIN factor applies to the entire job. You cannot configure separate weight, limit, or grace period factors. The ADMIN factor takes effect as soon as it is set.


Where Platform LSF gets the job information for each factor


Factor or subfactor

Gets job information from...

MEM

The value for jobs submitted with -R "rusage[mem]"

For compound resource requirements submitted with -R "n1*{rusage[mem1]} + n2*{rusage[mem2]}" the value of MEM depends on whether resources are reserved per slot.

  • If RESOURCE_RESERVE_PER_SLOT=N, then MEM=mem1+mem2

  • If RESOURCE_RESERVE_PER_SLOT=Y, then MEM=n1*mem1+n2*mem2

SWAP

The value for jobs submitted with -R "rusage[swp]"

For compound resource requirements, SWAP is determined in the same manner as MEM.

PROC

The value of n for jobs submitted with bsub -n (min, max), or the value of PROCLIMIT in lsb.queues

JPRIORITY

The dynamic priority of the job, updated every scheduling cycle and escalated by interval defined in JOB_PRIORITY_OVER_TIME defined in lsb.params

QPRIORITY

The priority of the job submission queue

FS

The fairshare priority value of the submission user