LSF calculates a dynamic user priority for individual users or for a group, depending on how the shares are assigned. The priority is dynamic because it changes as soon as any variable in formula changes. By default, a user’s dynamic priority gradually decreases after a job starts, and the dynamic priority immediately increases when the job finishes.
By default, LSF calculates the dynamic priority for each user based on:
If you enable additional functionality, the formula can also involve additional resources used by jobs belonging to the user:
LSF measures resource usage differently, depending on the type of fairshare:
For queue-level fairshare, LSF measures the resource consumption of all the user’s jobs in the queue. This means a user’s dynamic priority can be different in every queue.
For host partition fairshare, LSF measures resource consumption for all the user’s jobs that run on hosts in the host partition. This means a user’s dynamic priority is the same in every queue that uses hosts in the same partition.
For queue-based fairshare, LSF measures the resource consumption of all jobs in each queue.
By default, LSF calculates dynamic priority according to the following formula:
dynamic priority = number_shares / (cpu_time * CPU_TIME_FACTOR + run_time * RUN_TIME_FACTOR + (1 + job_slots) * RUN_JOB_FACTOR + fairshare_adjustment*FAIRSHARE_ADJUSTMENT_FACTOR)
For cpu_time, run_time, and job_slots, LSF uses the total resource consumption of all the jobs in the queue or host partition that belong to the user or group.
The cumulative CPU time used by the user (measured in hours). LSF calculates the cumulative CPU time using the actual (not normalized) CPU time and a decay factor such that 1 hour of recently-used CPU time decays to 0.1 hours after an interval of time specified by HIST_HOURS in lsb.params (5 hours by default).
The adjustment calculated by the fairshare adjustment plugin (libfairshareadjust.*).
You can give additional weight to the various factors in the priority calculation by setting the following parameters for the queue in lsb.queues or for the cluster in lsb.params. When the queue value is not defined, the cluster-wide value from lsb.params is used.
If you modify the parameters used in the dynamic priority formula, it affects every fairshare policy in the cluster.
In some cases the dynamic priority equation may require adjustments beyond the run time, cpu time, and job slot dependencies provided by default. The fairshare adjustment plugin is open source and can be customized once you identify specific requirements for dynamic priority.
All information used by the default priority equation (except the user shares) is passed to the fairshare plugin. In addition, the fairshare plugin is provided with current memory use over the entire cluster and the average memory allocated to a slot in the cluster.
Jobs assigned to a single slot on a host can consume host memory to the point that other slots on the hosts are left unusable. The default dynamic priority calculation considers job slots used, but doesn’t account for unused job slots effectively blocked by another job.
The fairshare adjustment plugin example code provided by Platform LSF is found in the examples directory of your installation, and implements a memory-based dynamic priority adjustment as follows:
fairshare adjustment= (1+slots)*((used_memory /used_slots)/(slot_memory*THRESHOLD))