Hierarchical fairshare among project groups

Project groups pool multiple service domains together and treat them as one source for licenses, and distribute them in a hierarchical fairshare tree. The leaves of the policy tree are the license projects that jobs can belong to. Each project group in the tree has a set of values, including shares and limits.

License ownership can only be configured at the leaf level; that is, on individual license projects. Ownership of a given internal node equals to sum of the ownership of all of its direct children.

The grouping of projects is per feature. Each feature has its own hierarchical group, but features can share the same hierarchy. The hierarchical scheduling is done per feature across service domains. This is the difference between hierarchical and non-hierarchical scheduling, which is done per feature and per service domain. With a hierarchical grouping of projects defined, service domains become license containers, free or in use.

Example project group configuration

The following project group configuration in lsf.licensescheduler implements this hierarchy:

Begin ProjectGroup 
GROUP                 SHARES    OWNERSHIP   LIMITS     NON_SHARED 
(topgrp (g1 g2))      (1 1)     (4 4)       (10 10)    (4 4) 
(g1 (g3 g4))          (1 1)     (0 4)       (10 10)    (0 4) 
(g2 (g5 g6))          (1 1)     (2 2)       (- 5)      (2 2) 
(g3 (p1 p2 p3))       (1 1 2)   ()          (3 4 5)    () 
(g4 (p4 p5 p6))       (1 1 1)   (1 1 1)     ()         (- 3 0) 
(g5 (p7 p8 p9))       (1 1 1)   (2 - 2)     ()         (1 - 1) 
(g6 (p10 p11 p12))    (1 1 1)   (2 2 2)     (4 4 4)    (1 0 1) 
End ProjectGroup 

Begin Feature 
NAME = AppZ 
GROUP_DISTRIBUTION = topgrp 
SERVICE_DOMAINS = LanServer WanServer 
End Feature

In this example a dash (-) denotes the default value, as in LSF. This default value depends on the column, for example by default OWNERSHIP should be 0, whereas LIMITS should be infinity (unlimited). A dash is not allowed in the SHARES column.

Since the service domains are no longer used in the distribution policies, they only need to be listed under the SERVICE_DOMAINS parameter in each Feature section.

Maximum token limit in project groups

By default, License Scheduler distributes all available tokens if possible. Project group configuration enables you to set hard limits on available tokens so that some tokens may not be distributed even if they are available.

In the following example, a total of 6 licenses are available:

Begin ProjectGroup
GROUP       SHARES OWNERSHIP LIMITS NON_SHARED 
(Root(A B)) (1 1)   ()        ()     () 
(A (c d))   (1 1)   ()        (1 1)  () 
(B (e f))   (1 1)   ()        ()     () 
End ProjectGroup

When there is no demand for license tokens, License Scheduler only allocates 5 tokens according to the distribution. License Scheduler gives 3 tokens to group A and 3 tokens to group B, but project c and project d are limited to 1 token each, so 1 token will not be allocated within group A. As more demand comes in for project e and project f, the unallocated tokens are distributed to group B.

Shared and non-shared licenses

Normally, the total number of non-shared licenses should be less than the total number of license tokens available. License tokens may not be available to project groups if the total non-shared licenses for all groups is greater than the number of shared tokens available.

For example, feature p4_4 is configured as follows, with a total of 4 tokens:

Begin Feature 
NAME =p4_4 
# total token value is 4 
GROUP_DISTRIBUTION=final 
SERVICE_DOMAINS=LanServer 
End Feature

The correct configuration is:

GROUP             SHARES    OWNERSHIP   LIMITS      NON_SHARED 
(final (G2 G1))   (1 1)     (2 0)          ()          (2 0) 
(G1 (AP2 AP1))    (1 1)     (1 1)          ()          (1 1)

Project group configuration like the following is valid, but could cause tokens not to be available for the leaf-level projects of group G1.

Begin ProjectGroup
GROUP           SHARES  OWNERSHIP   LIMITS    NON_SHARED 
(final(G1 G2))  (1 1)   (2 2)          ()        (2 2) 
(G1(AP2 AP1 ))  (1 1 )  (1 1)          ()        (1 1) 
End ProjectGroup

The total non-shared tokens is 6, but the total available is 4, which can cause non-shared license tokens not to be available. License scheduler satisfies the non-shared configuration first, so it gives 2 tokens to group G2 and 2 tokens to group G1. No tokens are left, and the non-shared configuration for projects AP2 and AP1 is not satisfied.

For projects defined with NON_SHARED_DISTRIBUTION, you must assign the project OWNERSHIP an equal or greater number of tokens defined in the DISTRIBUTION line.

Viewing information about project groups

Use blstat -G to view the hierarchical dynamic license information:

blstat -G 
FEATURE: p1_f1 
SERVICE_DOMAINS: 
TOTAL_INUSE: 0    TOTAL_RESERVE: 0    TOTAL_FREE: 5    OTHERS: 0   
SHARE_INFO_FOR:  /topgrp  
GROUP/PROJECT           SHARE   OWN  INUSE RESERVE FREE   DEMAND    
g2                      100.0 %   4     0    0        4     0         
SHARE_INFO_FOR:  /topgrp/g2  
GROUP/PROJECT           SHARE   OWN  INUSE RESERVE FREE   DEMAND
p3                      50.0 %   0     0    0        2     0   
p4                      50.0 %   0     0    0        2     0 
FEATURE: p1_f2 
SERVICE_DOMAINS: 
TOTAL_INUSE: 0    TOTAL_RESERVE: 0    TOTAL_FREE: 10   OTHERS: 0   
SHARE_INFO_FOR:  /topgrp  
GROUP/PROJECT          SHARE   OWN  INUSE RESERVE FREE   DEMAND       
g2                     100.0 %   4     0    0        4     0    
SHARE_INFO_FOR:  /topgrp/g2  
GROUP/PROJECT          SHARE   OWN  INUSE RESERVE FREE   DEMAND       
p3                     50.0 %   0     0    0        2     0  
p4                     50.0 %   0     0    0        2     0

blinfo -G

Use blinfo -G to view the hierarchical configuration:

blinfo -G 
GROUP                 SHARES    OWNERSHIP   LIMITS     NON_SHARED 
(topgrp (g1 g2))      (1 1)     (4 4)          (10 10)    (4 4) 
(g1 (g3 g4))          (1 1)     (2 4)          (10 10)    (0 4) 
(g2 (g5 g6))          (1 1)     (2 2)          (- 5)      (2 2) 
(g3 (p1 p2 p3))       (1 1 2)   ()          (3 4 5)    () 
(g4 (p4 p5 p6))       (1 1 1)   (1 3 1)     ()         (- 3 0) 
(g5 (p7 p8 p9))       (1 1 1)   (2 - 2)     ()         (1 - 1) 
(g6 (p10 p11 p12))    (1 1 1)   (2 2 2)     (4 4 4)    (1 0 1)

Use blinfo -G to view hierarchical project group priority information.

blinfo -G

GROUP            SHARES    OWNERSHIP   LIMITS   NON_SHARED  PRIORITY
(root (A B C))    (1 1 1)      ()          ()          ()          (3 2 0)
(A (P1 D))        (1 1)      ()         ()         ()          (3 5)
(B (P4 P5))       (1 1)      ()         ()         ()          ()
(C (P6 P7 P8))    (1 1 1)      ()          ()          ()          (8 3 0)
(D (P2 P3))        (1 1)     ()         ()         ()          (2 1)

Defining flat project priority

If no hierarchical project groups are defined, the default project configuration is flat. The priority of a project has nothing to do with its position in the hierarchy. Project priority values can be compared between all leaf nodes.

To configure flat project priority in lsf.licensescheduler, set the PRIORITY column in the Project section. For example:

Begin Projects
PROJECTS    PRIORITY
P1             2
P2             3
P3             5
P4             1
…              …
P8             7
End Projects

Configure tree priority to define priorities for all nodes in the hierarchy

To configure tree priority in lsf.licensescheduler, specifiy PRIORITY in the ProjectGroup configuration section. For example:

Begin ProjectGroup
GROUP             SHARES   OWNERSHIP  LIMITS  NON_SHARED PRIORITY
(root(A B C))     (1 1 1)  ()         ()         ()      (3 2 -)
(A (P1 D))        (1 1)    ()         ()         ()      (3 5)
(B (P4 P5))       (1 1)    ()         ()         ()      ()
(C (P6 P7 P8))    (1 1 1)  ()         ()         ()      (8 3 0)
(D (P2 P3))       (1 1)    ()         ()         ()      (2 1)
End ProjectGroup

By default, priorities are evaluated from top to bottom. The priority of a given node is first decided by the priorities of its parent nodes. The values are only comparable between siblings.

The following figure illustrates the example configuration:

The priority of each node is shown beside the node name. If priority is not defined, by default is set to 0 (nodes P4 and P5 under node B).

To find the highest priority leaf node in the tree License Scheduler traverses the tree from root to node A to node D to project P2.

To find the lowest priority leaf node in the tree, License Scheduler traverses the tree from root to node C to project P8.

When two nodes have the same priority, for example, projects P4 and P5, priority is determined by accumulated inuse usage at the time the priorities are evaluated.

When a leaf node in branch A wants to preempt a token from branch B or C, branch C is picked because it has a lower priority than branch B.

How preemption happens in a project group hierarchy

License preemption in License Scheduler takes place when no more free tokens are available, and there is at least one underfed project and at least one overfed project. When PRIORITY is configured in the project group, License scheduler can preempt license tokens at leaf nodes across the branches in the group hierarchy configuration.

By default, when no PRIORITY is configured in the project group, License Scheduler only preempts within a particular branch for tokens to preempt. This approach works like a flat project configuration—projects are underfed only when they have need and their inuse tokens are less than their owned tokens. Projects are overfed when their inuse tokens are greater than their owned tokens.

Configure hierarchical project group preemption

Configure hierarchical project group preemption two ways:

  • Top-down—License Scheduler always tries to preempt tokens from the projects that are furthest away in the hierarchy first. This balances the token ownership from top to bottom. Top-down preemption does not support ENABLE_MINJOB_PREEMPTION.

To enable top-down preemption, configure priorities for projects in the project group hierarchy. Top-down preemption is the default when priorities are configured in a project group hierarchy.

  • Bottom-up—License Scheduler always tries to preempt tokens from the closest projects in the hierarchy first. This balances token ownership from bottom to top. Bottom-up preemption does not conform to GROUP preemption in a flat project group configuration. Bottom-up preemption does not support ENABLE_MINJOB_PREEMPTION.

To enable bottom-up preemption, configure priorities for projects in the project group hierarchy, and specify LS_PREEMPT_PEER=Y in the Parameters section of lsf.licensescheduler.

Hierarchical project group preemption examples

The following configuration examples use the ownership configuration illustrated in the following figure:

Top-down configuration

  • Needs from a leaf node with ownership not satisfied:

P1: inuse=0

P2: inuse=2

P3: inuse=0

P4: inuse=2

If P1 needs one token, it will preempt one token from P4 instead of P2.

  • Needs from a leaf node without ownership or ownership are already satisfied:

P1: inuse=2

P2: inuse=0

P3: inuse=0

P4: inuse=2

If P2 needs one token, it can trigger preemption and get one token from P4. Because the ownership of its grandparent node (group A owns 3) is not satisfied.

  • Add limits to the tree. Assume limits defined on the internal nodes are same as ownership: group B limits=2, group A limits=3. Needs from a leaf node with ownership not satisfied:

P1: inuse=0

P2: inuse=2

P3: inuse=0

P4: inuse=2

When P1 needs one token, it can only preempt from P2 because the limit of group B is 2 to satisfy the total limits in that branch.

  • Add limits to the tree. Assume limits defined on the internal nodes are same as ownership: group B limits=2, group A limits=3. Needs from a leaf node without ownership or ownership are already satisfied:

P1: inuse=2

P2: inuse=0

P3: inuse=0

P4: inuse=2

If P2 needs one token, now because of limits, P2 can no longer preempt any tokens.

Bottom-up configuration

  • Needs from a leaf node with ownership not satisfied:

P1: inuse=0

P2: inuse=2

P3: inuse=0

P4: inuse=2

If P1 needs one token, it will preempt from P2 (instead of P4 as in the top-down configuration). This balances token preemption within its own sub-tree first.

  • Needs from a leaf node without ownership or ownership are already satisfied:

P1: inuse=2

P2: inuse=0

P3: inuse=0

P4: inuse=2

If P2 needs one token, it can preempt a token form P4 because the ownership of its grandparent (group A) has not been satisfied yet.

  • Add limits to the tree. Assume limits defined on the internal nodes are same as ownership: group B limits=2, group A limits=3. Needs from a leaf node with ownership not satisfied:

P1: inuse=0;

P2: inuse=2;

P3: inuse=0;

P4: inuse=2;

If P1 needs one token, it will preempt from P2, the closest leaf node, where the limits of this subtree have already been met.

  • Add limits to the tree. Assume limits defined on the internal nodes are same as ownership: group B limits=2, group A limits=3. Needs from a leaf node without ownership or ownership are already satisfied:

P1: inuse=2

P2: inuse=0

P3: inuse=0

P4: inuse=2

If P2 needs one token, it can no longer get tokens from the other branches because the limits of its parent node B have already been reached.