Failover provisioning for WANs

Similar to LANs, you can configure your site for a failover mechanism across multiple sites in a wide network.

You only need one host to run the LSF License Scheduler, but you can configure your site for a failover mechanism with multiple candidate hosts to take over the scheduling in case of a failure.

License scheduling across sites can be streamlined because LSF License Scheduler supports service provisioning during breaks in wide area network connections. This allows you to run LSF License Scheduler from one host that controls license scheduling across multiple sites.

Configure and start LSF License Scheduler in a WAN

In a WAN configuration:

  1. As the root user, install LSF License Scheduler on each cluster in the WAN configuration and select one cluster to be the cluster.

  2. In the cluster that contains the WAN license server, log on as the primary License Scheduler administrator.

  3. Edit the following items in LSF_CONFDIR/lsf.licensescheduler:

  4. Specify a space-separated list of hosts for the HOSTS parameter:

    HOSTS=hostname_1 hostname_2 ... hostname_n

    Where:

    hostname_1 is the most preferred host for running LSF License Scheduler.

    hostname_n is the least preferred host for running LSF License Scheduler.

  5. In the Clusters section, specify the names of the clusters in the WAN.

    For example:

    Begin Clusters
    CLUSTERS
    design_SJ
    design_BOS
    End Clusters

  6. In the cluster that contains the WAN license server, as the LSF primary administrator, edit LSF_CONFDIR/lsf.conf. Lines that begin with # are comments:

    Specify a space-separated list of hosts for the LSF_LIC_SCHED_HOSTS parameter:

    LSF_LIC_SCHED_HOSTS="hostname_1 hostname_2 ... hostname_n"

    Where:

    hostname_1, hostname_2, ..., hostname_n are hosts on which the LSF LIM daemon starts the LSF License Scheduler daemon (bld).

    The first host listed in the HOSTS list will be the default master License Scheduler host for the WAN.

    The order of the host names in LSF_LIC_SCHED_HOSTS is ignored.

  7. In the other clusters in the WAN:

  8. Configure the LSF_LIC_SCHED_HOSTS parameter in lsf.conf with a local list of candidate hosts.

  9. Configure the HOSTS parameter in the Parameters section lsf.licensescheduler with the following list of hosts:

    • Start the list with the same list of candidate hosts as the HOSTS parameter in the cluster that contains the WAN license server.

    • Continue the list with the local cluster’s list of hosts from the LSF_LIC_SCHED_HOSTS parameter in lsf.conf.

  10. In the cluster that contains the WAN license server and the other clusters in the WAN, run the following commands:

    1. Run bld -C to test for configuration errors.

    2. Run bladmin reconfig to configure LSF License Scheduler.

    3. Run lsadmin reconfig to reconfigure LIM.

    4. Use ps -ef to make sure that bld is running on the candidate hosts.

    5. Run badmin reconfig to reconfigure mbatchd.

    Tip:

    Although the bld daemon is started by LIM, bld runs under the account of the primary License Scheduler administrator. If you did not configure the LIM to automatically start the bld daemon on your License Scheduler hosts, run LSF_BINDIR/blstartup on each host to start the bld daemon.

WAN example

A design center contains the following hosts configuration in a WAN:

LIM starts bld on the following hosts:

  • lsf.conf in Design Center A

    LSF_LIC_SCHED_HOSTS="hostA1.designcenter_a.com hostA2.designcenter_a.com hostA3.designcenter_a.com"

  • lsf.conf in Design Center B

    LSF_LIC_SCHED_HOSTS="hostB1.designcenter_b.com hostB2.designcenter_b.com hostB3.designcenter_b.com"

License Scheduler candidate hosts are listed in the following order of preference:

  • lsf.licensescheduler in Design Center A

    HOSTS=hostB1.designcenter_b.com hostB2.designcenter_b.com hostA1.designcenter_a.com hostA2.designcenter_a.com hostA3.designcenter_a.com 

  • lsf.licensescheduler in Design Center B

    HOSTS=hostB1.designcenter_b.com hostB2.designcenter_b.com hostB3.designcenter_b.com

The following diagram shows hostB1.designcenter_b.com, the License Scheduler host for the WAN containing Design Center A and Design Center B.

How it works

The LSF LIM daemon starts the LSF License Scheduler daemon (bld) on each host listed in LSF_LIC_SCHED_HOSTS in Design Center A and Design Center B.

Each host in the HOSTS list in Design Center A is a potential LSF License Scheduler candidate in Design Center A and is running the bld daemon, but only one host becomes the LSF License Scheduler host—the first host in the HOSTS list that is up and that is running the bld daemon. Similarly, the License Scheduler host in Design Center B is the first host in the HOSTS list that is up and that is running the bld daemon.

License Scheduler manages the licenses in Design Center A and Design Center B as follows:

Both design centers list hostB1.designcenter_b.com at the top of their HOSTS lists. hostB1.designcenter_b.com is the License Scheduler host for Design Center A and for Design Center B. The rest of the hosts in both design centers remain on standby as candidate License Scheduler hosts. License Scheduler manages the license scheduling across the WAN connection.

Service provisioning at the host and network levels

In the above example configuration, there are two potential points of failure:

  • Host failure:

If hostB1.designcenter_b.com fails, and bld stops running, a candidate License Scheduler host must take over the license management. The next host on the HOSTS list in both Design Center A and Design Center B is hostEB2designcenter_b.com. License Scheduler fails over to this host if it is up and running.

  • Network failure:

If the network connection between Design Center A and Design Center B breaks, Design Center A can no longer communicate with the hosts in Design Center B, so hostB1.designcenter_b.com and hostB2.designcenter_b.com are no longer candidate license scheduling hosts for Design Center A. The next candidate host for Design Center A is hostA1.designcenter_a.com. License management then runs locally in Design Center A on hostA1.designcenter_a.com. In Design Center B, hostB1.designcenter_b.com continues to run License Scheduler, but only manages the local network as long as the wide area network connection is down.

The local License Scheduler host, hostA1.designcenter_a.com, checks for a heartbeat from hostB1.designcenter_b.com at regular intervals, then returns license management back to hostB1.designcenter_b.com when the network connection returns.