Project groups pool multiple service domains together and treat them as one source for licenses, and distribute them in a hierarchical fairshare tree. The leaves of the policy tree are the license projects that jobs can belong to. Each project group in the tree has a set of values, including shares and limits.
License ownership can only be configured at the leaf level; that is, on individual license projects. Ownership of a given internal node equals to sum of the ownership of all of its direct children.
The grouping of projects is per feature. Each feature has its own hierarchical group, but features can share the same hierarchy. The hierarchical scheduling is done per feature across service domains. This is the difference between hierarchical and non-hierarchical scheduling, which is done per feature and per service domain. With a hierarchical grouping of projects defined, service domains become license containers, free or in use.
The following project group configuration in lsf.licensescheduler implements this hierarchy:
Begin ProjectGroupGROUP SHARES OWNERSHIP LIMITS NON_SHARED(topgrp (g1 g2)) (1 1) (4 4) (10 10) (4 4)(g1 (g3 g4)) (1 1) (0 4) (10 10) (0 4)(g2 (g5 g6)) (1 1) (2 2) (- 5) (2 2)(g3 (p1 p2 p3)) (1 1 2) () (3 4 5) ()(g4 (p4 p5 p6)) (1 1 1) (1 1 1) () (- 3 0)(g5 (p7 p8 p9)) (1 1 1) (2 - 2) () (1 - 1)(g6 (p10 p11 p12)) (1 1 1) (2 2 2) (4 4 4) (1 0 1)End ProjectGroup
Begin FeatureNAME = AppZGROUP_DISTRIBUTION = topgrpSERVICE_DOMAINS = LanServer WanServerEnd Feature
In this example a dash (-) denotes the default value, as in LSF. This default value depends on the column, for example by default OWNERSHIP should be 0, whereas LIMITS should be infinity (unlimited). A dash is not allowed in the SHARES column.
Since the service domains are no longer used in the distribution policies, they only need to be listed under the SERVICE_DOMAINS parameter in each Feature section.
By default, License Scheduler distributes all available tokens if possible. Project group configuration enables you to set hard limits on available tokens so that some tokens may not be distributed even if they are available.
Begin ProjectGroupGROUP SHARES OWNERSHIP LIMITS NON_SHARED(Root(A B)) (1 1) () () ()(A (c d)) (1 1) () (1 1) ()(B (e f)) (1 1) () () ()End ProjectGroup
When there is no demand for license tokens, License Scheduler only allocates 5 tokens according to the distribution. License Scheduler gives 3 tokens to group A and 3 tokens to group B, but project c and project d are limited to 1 token each, so 1 token will not be allocated within group A. As more demand comes in for project e and project f, the unallocated tokens are distributed to group B.
Normally, the total number of non-shared licenses should be less than the total number of license tokens available. License tokens may not be available to project groups if the total non-shared licenses for all groups is greater than the number of shared tokens available.
For example, feature p4_4 is configured as follows, with a total of 4 tokens:
Begin FeatureNAME =p4_4# total token value is 4GROUP_DISTRIBUTION=finalSERVICE_DOMAINS=LanServerEnd Feature
GROUP SHARES OWNERSHIP LIMITS NON_SHARED(final (G2 G1)) (1 1) (2 0) () (2 0)(G1 (AP2 AP1)) (1 1) (1 1) () (1 1)
Project group configuration like the following is valid, but could cause tokens not to be available for the leaf-level projects of group G1.
The total non-shared tokens is 6, but the total available is 4, which can cause non-shared license tokens not to be available. License scheduler satisfies the non-shared configuration first, so it gives 2 tokens to group G2 and 2 tokens to group G1. No tokens are left, and the non-shared configuration for projects AP2 and AP1 is not satisfied.
For projects defined with NON_SHARED_DISTRIBUTION, you must assign the project OWNERSHIP an equal or greater number of tokens defined in the DISTRIBUTION line.
Use blstat -G to view the hierarchical dynamic license information:
blstat -G
FEATURE: p1_f1SERVICE_DOMAINS:TOTAL_INUSE: 0 TOTAL_RESERVE: 0 TOTAL_FREE: 5 OTHERS: 0SHARE_INFO_FOR: /topgrpGROUP/PROJECT SHARE OWN INUSE RESERVE FREE DEMANDg2 100.0 % 4 0 0 4 0SHARE_INFO_FOR: /topgrp/g2GROUP/PROJECT SHARE OWN INUSE RESERVE FREE DEMANDp3 50.0 % 0 0 0 2 0p4 50.0 % 0 0 0 2 0FEATURE: p1_f2SERVICE_DOMAINS:TOTAL_INUSE: 0 TOTAL_RESERVE: 0 TOTAL_FREE: 10 OTHERS: 0SHARE_INFO_FOR: /topgrpGROUP/PROJECT SHARE OWN INUSE RESERVE FREE DEMANDg2 100.0 % 4 0 0 4 0SHARE_INFO_FOR: /topgrp/g2GROUP/PROJECT SHARE OWN INUSE RESERVE FREE DEMANDp3 50.0 % 0 0 0 2 0p4 50.0 % 0 0 0 2 0
Use blinfo -G to view the hierarchical configuration:
blinfo -G
GROUP SHARES OWNERSHIP LIMITS NON_SHARED(topgrp (g1 g2)) (1 1) (4 4) (10 10) (4 4)(g1 (g3 g4)) (1 1) (2 4) (10 10) (0 4)(g2 (g5 g6)) (1 1) (2 2) (- 5) (2 2)(g3 (p1 p2 p3)) (1 1 2) () (3 4 5) ()(g4 (p4 p5 p6)) (1 1 1) (1 3 1) () (- 3 0)(g5 (p7 p8 p9)) (1 1 1) (2 - 2) () (1 - 1)(g6 (p10 p11 p12)) (1 1 1) (2 2 2) (4 4 4) (1 0 1)
Use blinfo -G to view hierarchical project group priority information.
blinfo -G
If no hierarchical project groups are defined, the default project configuration is flat. The priority of a project has nothing to do with its position in the hierarchy. Project priority values can be compared between all leaf nodes.
To configure flat project priority in lsf.licensescheduler, set the PRIORITY column in the Project section. For example:
Begin Projects
PROJECTS PRIORITY
P1 2
P2 3
P3 5
P4 1
… …
P8 7
End Projects
To configure tree priority in lsf.licensescheduler, specifiy PRIORITY in the ProjectGroup configuration section. For example:
Begin ProjectGroupGROUP SHARES OWNERSHIP LIMITS NON_SHARED PRIORITY(root(A B C)) (1 1 1) () () () (3 2 -)(A (P1 D)) (1 1) () () () (3 5)(B (P4 P5)) (1 1) () () () ()(C (P6 P7 P8)) (1 1 1) () () () (8 3 0)(D (P2 P3)) (1 1) () () () (2 1)End ProjectGroup
By default, priorities are evaluated from top to bottom. The priority of a given node is first decided by the priorities of its parent nodes. The values are only comparable between siblings.
The priority of each node is shown beside the node name. If priority is not defined, by default is set to 0 (nodes P4 and P5 under node B).
To find the highest priority leaf node in the tree License Scheduler traverses the tree from root to node A to node D to project P2.
To find the lowest priority leaf node in the tree, License Scheduler traverses the tree from root to node C to project P8.
When two nodes have the same priority, for example, projects P4 and P5, priority is determined by accumulated inuse usage at the time the priorities are evaluated.
When a leaf node in branch A wants to preempt a token from branch B or C, branch C is picked because it has a lower priority than branch B.
License preemption in License Scheduler takes place when no more free tokens are available, and there is at least one underfed project and at least one overfed project. When PRIORITY is configured in the project group, License scheduler can preempt license tokens at leaf nodes across the branches in the group hierarchy configuration.
By default, when no PRIORITY is configured in the project group, License Scheduler only preempts within a particular branch for tokens to preempt. This approach works like a flat project configuration—projects are underfed only when they have need and their inuse tokens are less than their owned tokens. Projects are overfed when their inuse tokens are greater than their owned tokens.
Configure hierarchical project group preemption two ways:
To enable top-down preemption, configure priorities for projects in the project group hierarchy. Top-down preemption is the default when priorities are configured in a project group hierarchy.
Bottom-up—License Scheduler always tries to preempt tokens from the closest projects in the hierarchy first. This balances token ownership from bottom to top. Bottom-up preemption does not conform to GROUP preemption in a flat project group configuration. Bottom-up preemption does not support ENABLE_MINJOB_PREEMPTION.
To enable bottom-up preemption, configure priorities for projects in the project group hierarchy, and specify LS_PREEMPT_PEER=Y in the Parameters section of lsf.licensescheduler.
The following configuration examples use the ownership configuration illustrated in the following figure:
If P1 needs one token, it will preempt one token from P4 instead of P2.
If P2 needs one token, it can trigger preemption and get one token from P4. Because the ownership of its grandparent node (group A owns 3) is not satisfied.
When P1 needs one token, it can only preempt from P2 because the limit of group B is 2 to satisfy the total limits in that branch.
If P2 needs one token, now because of limits, P2 can no longer preempt any tokens.
If P1 needs one token, it will preempt from P2 (instead of P4 as in the top-down configuration). This balances token preemption within its own sub-tree first.
If P2 needs one token, it can preempt a token form P4 because the ownership of its grandparent (group A) has not been satisfied yet.
If P1 needs one token, it will preempt from P2, the closest leaf node, where the limits of this subtree have already been met.
If P2 needs one token, it can no longer get tokens from the other branches because the limits of its parent node B have already been reached.