Configure cluster mode

Use cluster mode to distribute licenses across LSF clusters, leaving the scheduler for each LSF cluster to schedule jobs, allocate licenses to projects within the cluster, and preempt jobs.

Configure parameters

  1. Cluster mode can be set globally, or for individual license features. Set individually when using cluster mode for some features and project mode for some features.
    1. If using cluster mode for all license features, define CLUSTER_MODE=Y in the Parameters section of lsf.licensescheduler.
    2. If using cluster mode for some license features, define CLUSTER_MODE=Y for individual license features in the Feature section of lsf.licensescheduler.

      The Feature section setting of CLUSTER_MODE overrides the global Parameter section setting.

  2. List the License Scheduler hosts.

    By default with an LSF installation, the HOSTS parameter is set to the LSF_MASTER_LIST.

    • List the hosts in order from most preferred to least preferred. The first host is the master license scheduler host.

    • Specify a fully qualified host name such as hostX.mycompany.com unless all your License Scheduler clients run in the same DNS domain.

    HOSTS=host1 host2

  3. Specify the data collection frequency between License Scheduler and FlexNet.

    The default is 60 seconds.

    LM_STAT_INTERVAL=seconds

  4. Specify the path to the FlexNet command lmstat.

    For example, if lmstat is located in /etc/flexlm/bin:

    LMSTAT_PATH=/etc/flexlm/bin
    Tip:

    If the lmstat command is not included in the flexlm/bin directory, you can find it packaged with your LSF distribution in $LSF_SERVERDIR.

Configure clusters

Configure the clusters permitted to use Platform License Scheduler in the Clusters section of the lsf.licensescheduler file.

This is only required if you are using more than one cluster.

In the Clusters section, list all clusters that can use Platform License Scheduler.

For example:

Begin Clusters 
CLUSTERS 
cluster1 
cluster2
End Clusters 

Cluster mode service domains

A service domain is a group of one or more FlexNet license servers. Platform License Scheduler manages the scheduling of the license tokens, but the license server actually supplies the licenses.

In cluster mode, each cluster can access licenses from one WAN and one LAN service domain.

Platform License Scheduler does not control application checkout behavior. If the same license is available from both the LAN and WAN service domains, License Scheduler expects jobs to try to obtain the license from the LAN first.

Configure ServiceDomain sections

You configure each service domain, with the license server names and port numbers that serve licenses to a network, in the ServiceDomain section of the lsf.licensescheduler file.

Whether the service domain is a WAN or LAN service domain is specified later in the Feature section.

  1. Add a ServiceDomain section, and define NAME for each service domain.

    For example:

    Begin ServiceDomain 
    NAME=DesignCenterA 
    End ServiceDomain
  2. Specify the FlexNet license server hosts for that domain, including the host name and FlexNet port number.

    For example:

    Begin ServiceDomain 
    NAME=DesignCenterA 
    LIC_SERVERS=((1700@hostA))
    End ServiceDomain

    For multiple license servers:

    LIC_SERVERS=((1700@hostA)(1700@hostB))

    For redundant servers, the parentheses are used to group the three hosts that share the same license.dat file:

    LIC_SERVERS=((1700@hostD 1700@hostE 1700@hostF))

    Note:

    If FlexNet uses a port from the default range, you can specify the host name without the port number. See the FlexNet documentation for the values of the default port range.

    LIC_SERVERS=((@hostA))

Configure LAN service domain

You configure LAN service domains in the Feature section of lsf.licensescheduler. Only a single cluster and service domain can be specified in each LAN Feature section. Licenses from the LAN service domain are statically allocated to the cluster.

In the Feature section, set
CLUSTER_DISTRIBUTION=service_domain(cluster_name share)

Use the service domain name defined in the ServiceDomain section.

For example:

Begin Feature
NAME=verilog
CLUSTER_DISTRIBUTION=MyLanServer(tokyo_cluster 1)
End Feature

Configure WAN service domain

WAN configuration includes all clusters sharing the WAN service domain. As for a LAN service domain, you set this in the CLUSTER_DISTRIBUTION parameter in the Feature section of the lsf.licensescheduler file.

For a WAN service domain, you can optionally configure dynamic license sharing based on past license use across all clusters served by the WAN service domain, and if required set minimum and maximum allocations for each cluster.

  1. Set the WAN service domain name in the CLUSTER_DISTRIBUTION parameter.
    CLUSTER_DISTRIBUTION = service_domain(cluster share/min/max...)

    Use the service domain name defined in the ServiceDomain section.

  2. Configure each cluster.

    All clusters with access to the WAN service domain licenses must be included.

    1. Set the cluster name.
    2. Set the share for each cluster.

      The share is a non-negative integer representing the share of licenses each cluster should receive in a static license allocation, and the starting share in a dynamic license allocation.

  3. Optionally, set ALLOC_BUFFER in the Feature section of the lsf.licensescheduler file. When set, this enables a dynamic sharing policy.
    ALLOC_BUFFER = buffer

    or

    ALLOC_BUFFER = cluster1 buffer1 cluster2 buffer2...default buffer
    • When extra license tokens are available, each cluster’s allocation increases to as much as PEAK+BUFFER.

      The value BUFFER is set by ALLOC_BUFFER in the Feature section, and the value PEAK is the peak value of dynamic license token use over a time interval set by PEAK_INUSE_PERIOD in the Parameters or Feature section.

    • When allocated tokens are not being use in a cluster, the cluster’s allocation goes down to PEAK+BUFFER.

      Since tokens are not being used in the cluster, the peak use value PEAK decreases, thus PEAK+BUFFER also decreases.

    The allocation buffer sets both the rate at which the cluster allocation can grow, and the number of licenses that can go unused, depending on demand.

    Allocation buffers help determine the maximum rate at which tokens can be transferred to a cluster as demand increases in the cluster. The maximum rate of transfer to a cluster is given by the allocation buffer divided by MBD REFRESH INTERVAL. Be careful not to set the allocation buffer too large so that licenses are not wasted because they are be allocated to a cluster that cannot use them.

  4. Optionally, when dynamic sharing is enabled (ALLOC_BUFFER is defined) you can set the minimum and maximum allocation for each cluster.

    The minimum allocation reserves license tokens for exclusive use by the cluster; the maximum allocation limits the total number of license tokens received by the cluster.

    Cluster shares take precedence over minimum allocations configured. If the minimum allocation exceeds the cluster's share of the total tokens, a cluster's allocation as given by bld may be less than the configured minimum allocation.

To allow a cluster to be able to use licenses only when another cluster does not need them, you can set the cluster distribution for the cluster to 0, and specify an allocation buffer for the number of tokens that the cluster can request.

For example:

Begin Feature
CLUSTER_DISTRIBUTION=Wan(CL1 0 CL2 1)
ALLOC_BUFFER=5
End Feature

When no jobs are running, the token allocation for CL1 is 5. CL1 can get more than 5 tokens if CL2 does not require them.

Examples

Static example (no allocation buffer set):

Begin Feature
NAME=verilog
CLUSTER_DISTRIBUTION=MyWanServer(tokyo_cl 1 newyork_cl 1 toronto_cl 2)
End Feature

In this example, licenses are statically allocated based solely on the number of shares assigned to each cluster. If the number of licenses is not evenly divisible by the number of shares, the additional licenses are distributed round-robin to clusters in the order they appear in CLUSTER_DISTRIBUTION. Thus if there are 98 licenses in total, tokyo_cl receives 25, newyork_cl receives 25, and toronto_cl receives 48. Each cluster limits the total rusage of running jobs based on the allocated license tokens.

Dynamic example (allocation buffer set):

Begin Feature
NAME=verilog
CLUSTER_DISTRIBUTION=MyWanServer(tokyo_cl 1 newyork_cl 1 toronto_cl 2/10/50)
ALLOC_BUFFER=tokyo_cl 5 newyork_cl 1 toronto_cl 2
End Feature

In this example, licenses are initially distributed according to the assigned shares. Since allocation buffers are set, dynamic sharing based on past use is enabled. Based on the allocation buffers, toyko_cl receives license tokens the fastest when there is demand within the cluster. Minimum amd maximum allocations of 10 and 50 respectively are set for toronto_cl, which also has the largest share.

LAN and dynamic WAN example:

Begin Feature
NAME=verilog
CLUSTER_DISTRIBUTION=MyWan(c1 1/1/25 c2 1/1/30 c3 2/5/100) MyLan(c1 1)
ALLOC_BUFFER=c3 5 default 2
End Feature

In this example the verilog license feature is available from both WAN and LAN service domain, however only cluster c1 receives the license feature from both servers. Licenses from the WAN service domain are initially distributed according to the assigned shares. Since allocation buffers are set, dynamic sharing based on past use is enabled. Based on the allocation buffers cluster c3 receives license tokens the fastest when there is demand within the cluster.

Configure license features

Each type of license requires a Feature section in the lsf.licensescheduler file.

  1. Define the feature name used by FlexNet to identify the type of license.

    You only need to specify this parameter if the License Scheduler token name is not identical to the FlexNet feature name.

    Begin Feature
    FLEX_NAME=201-AppZ 
    End Feature
  2. (Optional) Use the NAME parameter to define an alias between License Scheduler and FlexNet feature names.

    LSF does not support names that start with a number, or names containing a dash or hyphen character (-), which may be used in the FlexNet feature name. In these cases, define a NAME for the feature as well.

    In this example, the FlexNet feature name 201-AppZ has an alias of AppZ201.

    Begin Feature
    FLEX_NAME=201-AppZ 
    NAME=AppZ201 
    End Feature

Configure taskman jobs in cluster mode

Optionally, to run taskman (interactive) jobs in cluster mode, include the dummy cluster interactive in your service domain configuration.

In the Feature section:
  1. Include the dummy cluster interactive in the CLUSTER_DISTRIBUTION parameter.
  2. Set a share for the dummy cluster interactive .
  3. Optionally, set an allocation buffer for the dummy cluster interactive to enable dynamic allocation.

Examples

Begin Feature
NAME=licenseA
CLUSTER_DISTRIBUTION=MyLanServer(tokyo_cl 1 interactive 1)
End Feature
 
Begin Feature
NAME=licenseB
CLUSTER_DISTRIBUTION=MyWanServer(tokyo_cl 1 newyork_cl 1 interactive 2)
End Feature

Allocate licenses to non-LSF jobs

Applies to WAN service domains only.

Set WORKLOAD_DISTRIBUTION in the Feature section to allocate licenses for non-LSF use.
WORKLOAD_DISTRIBUTION=service_domain_name(LSF lsf_distribution NON_LSF non_lsf_distribution)

If WORKLOAD_DISTRIBUTION is set for a LAN service domain in cluster mode, the parameter is ignored.

For example, to set aside 20% of licenses for use outside of LSF:

Begin Feature
NAME=licenseB
CLUSTER_DISTRIBUTION=MyWanServer(tokyo_cl 1 newyork_cl 1)
WORKLOAD_DISTRIBUTION=MyWanServer(LSF 8 NON_LSF 2)
End Feature

Restart to implement configuration changes

  1. Run lsadmin limrestart or bladmin reconfig to restart the bld.
  2. If you have added, changed, or deleted any Feature sections, you may need to restart mbatchd. In this case a message is written to the log file prompting the restart.

    If required, run badmin mbdrestart to restart each LSF cluster.

View license allocation

Run blstat -t token_name to view information for a specific license token (as configured in a Feature section).

blstat output differs for cluster mode and project mode.