Understanding distribution policies

The most important part of LSF License Scheduler is license distribution. The license distribution policy determines how licenses are shared among projects. Whenever there is competition, the configured share assignment determines the portion of licenses each project is entitled to.

You can use different methods of distribution of License Scheduler policies:

  • Fairshare helps you ensure that all projects receive the share of license tokens they are entitled to.

  • Ownership lets high priority projects preempt licenses on demand, but share them when not required.

  • Non-shared licenses are not shared with other projects. When not in use by the project, they are always only available to that project.

Share assignments

Whenever there are licenses to spare, license projects can get as many tokens as they need. The share assignment is defined in the policy, but is ignored.

Whenever a project is using all its licenses and needs more, License Scheduler attempts to assign additional licenses to it. This is possible if another project is not using all its assigned licenses.

The total number of licenses managed by License Scheduler depends on the following:

  • The number of active license servers in the service domain

  • The number of licenses checked out by non-License Scheduler users

  • The number of new licenses that are added

  • The number of licenses that expired

The License Scheduler distribution policy entitles each license project to a minimum portion of the available licenses.

The share assignment in the License Scheduler distribution policy determines what portion of the total licenses is assigned to each project.

For example, create three License Scheduler projects, and share the licenses equally. If one project does not need a license, another project can use it.

(projectA 1 projectB 1 projectC 1)

License Scheduler distributes the licenses evenly to each project. If you have 264 licenses, each project gets 88 licenses.

Not all license projects in a cluster have a full workload at all times. Free licenses can be shared across projects, so that idle licenses for one project are available to other projects.

Example of share assignments with demand

The following illustrates how licenses are shared according to the share assignment ratio when there is demand.

The AppZ feature has a total of 120 tokens.

AppZ has the following configuration:

Begin Feature
NAME = AppZ
DISTRIBUTION = LanServer(A 1 B 1)
End Feature

Initially, before demand is introduced, the projects have the following token distribution:


Project

Used, Reserved, and Free

Demand

A

70

0

B

0

0


Because this feature controls 120 tokens and only 70 are used, 50 tokens are free.

If both projects A and B demand 100 tokens:


Project

Used, Reserved, and Free

Demand

A

70

100

B

0

100


The total number of shared tokens is calculated as the number of shared tokens consumed by P1 + shared tokens consumed by P2 + free tokens.

70 + 0 + 50 = 120 shared tokens

Both A and B have a share assignment of 1 configured, which means they each deserve the same number of tokens. In this case, they each deserve 60 (half of 120) tokens.

Because project A is already using 70 shared tokens, it is allocated no additional free tokens.

Because project B is using 0 shared tokens and it deserve 60 shared tokens, all 50 free tokens are allocated to project B.

After token distribution, the token usage and demand situation is as follows:


Project

Used, Reserved, and Free

Demand

A

70

100

B

50

50


License ownership

License ownership gives license projects the right to use their licenses on demand, while still allowing License Scheduler to distribute the licenses to other projects when the owner is not using them.

Whenever there is competition, the configured share assignment determines the portion of licenses each license project is entitled to. Whenever there are licenses to spare, the license distribution policy still defines which projects can receive tokens, but the share assignment is ignored (the specified projects can receive as many tokens as they need).

How license ownership and preemption work

In some cases, licenses can be shared among license projects, but one project has priority. This project is the owner of the licenses. When there is competition between the owner and other projects, the licenses should become available to the owner immediately.

License Scheduler only distributes tokens for free licenses. By default, if all licenses are in use, a license has to be released before License Scheduler can provide a license token to another license project. Therefore, a project that is entitled to a license may have to wait for another project’s running job to finish. There is no way to predict how long this takes.

If license ownership is configured, the owner should never have to wait to use the owned licenses. If a license project requires a license and is entitled to it by right of ownership, but there are no licenses free, License Scheduler preempts a running job in order to take a license away from another project.

License projects that own some licenses can participate in license sharing with projects that do not own any. A service domain can include both owned and unowned licenses.

When a license project’s share assignment is more than the number of licenses it owns, and there is competition, the project is entitled to use its configured ownership entitlement. Preemption can occur only while the project is not yet using the specified number of owned licenses and no free licenses are available. Once the project is using the number of licenses it owns, License Scheduler waits for licenses to become free and then distributes additional tokens until the project is using its fair share.

The jobs that are preempted by LSF are automatically resumed by LSF as licenses become available.

You can enable LSF to release the job slot of a suspended job when License Scheduler preempts the license from the job.

Note:

For License Scheduler to give a license token to another license project, the applications must be able to release their licenses upon job suspension.

Preemption when JOB_CONTROLS are defined

If the LSF administrator has defined JOB_CONTROLS in lsb.queues so that preemption uses the signal SIGTSTP, they must also define LIC_SCHED_PREEMPT_STOP=Y in lsf.conf for License Scheduler preemption to work.

How LSF License Scheduler selects projects for preemption

When a project needs to preempt a license token currently in use by other projects, License Scheduler determines which license token to preempt and which project gets the preempted token. It is possible for a project to use more licenses than it owns—this is an overfed project. It is also possible for a project to use fewer licenses than it owns—this is an underfed project. License Scheduler typically preempts license tokens from the overfed projects and gives these preempted tokens to the underfed projects.

License Scheduler determines the order in which overfed and underfed projects are selected for preemption by looking at the accumulative inuse of each project—the time in which the project has been overfed or underfed.

You can manually override this behavior by adding a PRIORITY column to the Projects section of lsf.licensescheduler and assigning a priority to each project. This enables License Scheduler to look at the project priority along with the hierarchical fairshare. All the projects in the hierarchy are assigned a priority for preemption ordering.

If no hierarchical project groups are defined, the default project configuration is flat. The priority of a project has nothing to do with its position in the hierarchy. Project priority values can be compared between all leaf nodes.

You can also minimize the overall number of preempted jobs by enabling job list optimization. When you set the parameter ENABLE_MINJOB_PREEMPTION=Y in the Feature section of lsf.licensescheduler, License Scheduler preempts the minimum number of jobs needed to obtain the required licenses. For example, for a job that requires 10 licenses, License Scheduler preempts one job that uses 10 or more licenses rather than 10 jobs that each use one license.

Overfed projects by priority

If there is more than one overfed project, License Scheduler preempts tokens from these projects in the following order:

  • If there are fewer license tokens than the project owns, License Scheduler preempts license tokens from the lowest priority project.

  • If there are more licenses than the project owns, License Scheduler preempts license tokens from projects according to the order projects are listed in the Projects section.

Underfed projects by priority

If there is more than one underfed project, License Scheduler assigns the preempted tokens to these projects in the following order:

  • If there are fewer licenses than the project owns, License Scheduler gives the preempted license tokens to the highest priority project

  • If there are more licenses than the project owns, License Scheduler gives the preempted license tokens to projects according to the order projects are listed in the Projects section.

How LSF preemption and License Scheduler preemption coexist

Jobs belonging to a license project that has ownership in License Scheduler can trigger preemption even when no more slots are available in LSF. Configured together with LSF_LIC_SCHED_PREEMPT_SLOT_RELEASE, license job preemption works together with LSF slot-based preemption.

Example

Project proj1 has ownership of 3 of the license AppX.

MXJ = 5, and LSF_LIC_SCHED_PREEMPT_SLOT_RELEASE=Y is configured in lsf.conf.

5 jobs are submitted and started using AppX, in proj2. Then 2 jobs are submitted to proj1, and pend waiting for a AppX license token. Although the slots are full, the request is sent to License Scheduler, which recognizes the ownership and preempts 2 jobs in proj2. The jobs are suspended, both their licenses and slots are released, and the 2 jobs in proj1 can run.

Maximum preemption limits

Both LSF jobs and taskman jobs using licenses managed by License Scheduler can be preempted. To ensure lower priority jobs are not preempted too many times, maximum preemption time limits can be enabled with LS_ENABLE_MAX_PREEMPT.

For LSF jobs the parameter MAX_JOB_PREEMPT sets the maximum number of times a job can be preempted. MAX_JOB_PREEMPT can be defined in lsb.params, lsb.queues, or lsb.applications, with the application setting overriding the queue setting and the queue setting overriding the cluster-wide lsb.params definition.

License Scheduler taskman job preemption limits are controlled by the parameter LS_MAX_TASKMAN_PREEMPT in lsf.licensescheduler.

How LSF License Scheduler calculates shares and owned licenses

To configure license ownership, edit the DISTRIBUTION parameter in the Feature section of lsf.licensescheduler. Each license project’s share assignment is defined by specifying the project name followed by its share:

license_project_name number_shares

To indicate ownership, the share assignment is followed by a slash and the number of licenses owned by that license project. The two numbers cannot be compared because they are different units—the ownership figure is always a fixed number of licenses, while share assignment can represent a ratio or percentage of the total instead of an actual number of licenses.

license_project_name number_shares/number_licenses_owned F

Example

DISTRIBUTION=LanServer1(Lp1 1 Lp2 2/6)

This example allows Lp1 to use one third of the available licenses and Lp2 to use two thirds of the licenses. However, Lp2 is always entitled to six licenses, and can preempt other license projects to obtain the licenses immediately if they are needed. If the projects are competing for a total of 12 licenses, Lp2 is entitled to eight (six on demand, and two more as soon as they are free). If the projects are competing for only six licenses in total, Lp2 is entitled to all of them, and Lp1 can only use licenses when Lp2 does not need them.

Total licenses

The total number of licenses managed by License Scheduler depends on whether all license servers in the domain are active, and whether licenses have been checked out by non-LSF users. The number of licenses available to LSF changes when licenses are added to any license server in the domain and when licenses expire.

Share assignment

The share assignment determines what fraction of the total licenses is assigned to each license project.

The formula for converting a number of shares to a number of licenses is:

(shares assigned to a project) x (total number of licenses) ----------------------------------------------------------        (sum of all shares assigned to all projects)

The number of shares assigned to a license project is only meaningful when you compare it to the number assigned to other projects, or to the total number of shares.

Example

  • (Lp1 1 Lp2 1)

In this example, you configure a 1:1 ratio. No matter how many licenses you have, LSF assigns half to each license project. If you have 264 licenses, LSF assigns 132 licenses to each.

  • (Lp1 1 Lp2 1 Lp3 1)

In this example, you add another license project, and assign it one share. Now LSF evenly distributes one third of the available licenses to each project. If you have 264 licenses, each project receives 88 licenses.

Percentage of share

If you set LS_FEATURE_PERCENTAGE=Y in lsf.licensescheduler, you configure license ownership in percentages instead of absolute numbers. When not combined with hierarchical projects, affects DISTRIBUTED and NON_SHARED_DISTRIBUTION values only. When using hierarchical projects, percentage is applied to OWNERSHIP, LIMITS, and NON_SHARED values.

Example 1

Begin Feature
LS_FEATURE_PERCENTAGE = Y
DISTRIBUTION = LanServer (p1 1 p2 1 p3 1/20)
...
End Feature

The service domain LanServer shares licenses equally among three License Scheduler projects. P3 is always entitled to 20% of the total licenses, and can preempt another project to get the licenses immediately if they are needed.

Example 2

Hierarchical project groups:

Begin ProjectGroup
GROUP      SHARES    OWNERSHIP    LIMITS    NON_SHARED
(R (A p4))  (1  1)     ()         ()         ()
(A (B p3))  (1  1)     (- 10)     (- 20)     ()
(B (p1 p2)) (1  1)     (30 -)     ()         (- 5)
End ProjectGroup

Project p1 owns 30% of the total licenses, and project p3 owns 10% of total licenses. P3's LIMITS is 20% of total licenses, and p2's NON_SHARED is 5%.

Non-shared licenses

If you have a project that is so important that you must always guarantee the availability of a license token to the project, you can distribute non-shared licenses to a project. These licenses cannot be shared with other projects.

Some license agreements do not allow license sharing. These can be distributed to license projects as non-shared licenses.

Default projects

Default projects are projects that are not specified in job submission (for example, with bsub -Lp), but use license tokens that are managed by License Scheduler.

If you do not want the default project to get shares of license tokens, you do not need to define a default project in the distribution policy for a feature. If you do configure the default project in a policy, it receives its assigned share of the license tokens. All jobs requiring a license feature that is managed by License Scheduler, and are not submitted to a configured project for the feature, are treated as jobs submitted to the default project unless LSF_LIC_SCHED_STRICT_PROJECT_NAME=y in lsf.conf and you have not configured a default project in DISTRIBUTION parameter, in which case the job is rejected.

If a job is submitted to a license project that is not defined in the DISTRIBUTION in lsf.licensescheduler, it does not run. You have two options to ensure your job does not pend:
  1. Define a default project in DISTRIBUTION and do not submit job to a license project (the job uses tokens from the default project).

  2. Define the license project and share in DISTRIBUTION.