Configuring Liberty collective replica sets

You can configure collective replica sets. A replica set provides highly available management capabilities for a Liberty administrative domain.

For distributed platformsFor IBM i platforms

Before you begin

The collectiveController-1.0 feature and its capabilities are available only in WebSphere® Application Server Liberty Network Deployment and WebSphere Application Server Liberty for z/OS®. The feature is not available in WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core. If you have a WebSphere Application Server Liberty Network Deployment installation, you can use its collectiveController-1.0 feature to work with collective members from WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core installations. A collective member can be from any WebSphere Application Server Liberty release.

About this task

A replica set is a set of collective controllers that are configured to work together. Each replica contains all the repository updates that the other replicas within the set have processed. Therefore, there is no need for a member or client to connect with a particular collective controller each time that it interacts with the collective; any of the collective controllers that are configured in the replica set can provide the same data.

For detailed instructions about creating and configuring a collective controller, see Configuring a Liberty collective.

You can complete the following procedures to configure a collective replica set:

  1. Add a replica to an existing replica set.
  2. Modify the default configuration of the initial replica set.

See Example: Create and activate a replica set for an example of creating a replica set consisting of three collective controllers on the same host.

A replica set requires an odd number of replicas.

Procedure

  1. Add a replica to an existing replica set.

    During the life of a replica set, it might become necessary to add one or more replicas to an existing set, to increase capacity for example.

    The existing replicas in the replica set do not require any updates to their configuration. You can update them so that their configurations in the server.xml files more accurately reflect the replicas that form the replica set, but this update is unnecessary and does not affect their behaviors.

    Note: It is not necessary to change the replicaSet value in the server.xml file of an existing replica in the set. No change to the configuration of an existing replica is required. If you do want to update the replicaSet values in the configurations of existing replicas in the set so that the configuration values are consistent across all replicas in the set, you must set the isInitialReplicaSet value in the configurations of the existing replicas to false. After you alter the replicaSet value, it describes a changed replica set rather than the initial replica set.
    Note: When you refer to a replica, you must be consistent and use the same host:port value. If a hostname is used, it must always be used. Or, if an IP address is used, it must always be used.

    To add a replica, follow these steps:

    1. Ensure that the existing replica set is running and that most of the replicas are available.
    2. Create a server to be the new collective controller.
      wlp/bin/server create MyNewController
    3. Replicate to transform the new server into a collective controller.
      wlp/bin/collective replicate MyNewController 
         --host=host_of_running_controller  
         --port=https_port_of_running_controller 
         --user=userName_for_running_controller 
         --password=userPassword_for_running_controller 
         --keystorePassword=keystore_password_for_new_controller

      The replicate command writes XML output to a console screen. You copy the output into the server.xml file.

      To write the output of the command to a file, instead of to a console screen, specify the --createConfigFile=output_file_path parameter. Then, include the outputted file in the collective configuration by adding an include statement to the server.xml file:
      <include location=output_file_path />
    4. Configure the new replica's server.xml file by using output of the replicate command.
      Copy the XML output of the replicate command into the server.xml file. You can modify the replica configuration as follows:
      • Required settings:
        The values must be explicitly set. The XML printed by the replicate command contains this configuration and is sufficient for these settings.
        hostAuthInfo
        Host authentication information containing properties that a remote client needs to start the server. This setting is only required when the host of the new replica is not enabled for SSH. Typically, Linux hosts are enabled for SSH and Windows hosts are not enabled for SSH. Thus, this setting might only be needed on Windows hosts. Set RPC credentials for the replica in either of two ways:
        • Set rpcUser to an operating system login user ID for the host on which the replica resides, and set rpcUserPassword to the operating system login password for the user ID. For example, if you log into the replica computer with user test1 and password test1pwd, then change the hostAuthInfo element to the following:
          <hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
        • If the replica host is registered with the collective controller, set hostAuthInfo useHostCredentials to true for the replica to inherit RPC credentials from its host.
          <hostAuthInfo useHostCredentials="true" />

        See Overriding Liberty server host information for information about hostAuthInfo settings.

        replicaSet
        Space-delimited list that contains the host:port for each of the replicaHosts and replicaPorts in the replica set, excluding the values for the collective controller now being added to the set
        For example,
        original.host.com:10001 some.other.host.com:10003
        At least one of the values in this set must already be a replica of the existing replica set.
      • Optional settings:
        These settings default, but can be altered.
        isInitialReplicaSet
        False
        replicaHost
        Hostname for each individual replica
        replicaPort
        Port for each individual replica

        This port is not the http or https port of the collective controller, but it is a unique port that is used for communication between the replicas of the replica set.

        repositoryDir
        Directory location that is used to store repository data
      Here is an example of what you might add to a new replica server.xml file:
      <collectiveController replicaHost="localhost" 
         replicaPort="10012" 
         replicaSet="localhost:10010 localhost:10011" 
         isInitialReplicaSet="false"/> 
      The XML printed by the replicate command requires updating the server's security configuration and specifying the collectiveRootKeys keystore password. The server's security configuration must be identical to the original collective controller's configuration, and the collectiveRootKeys keystore password must be the password that is used for the original collective controller's collectiveRootKeys keystore password. If the replica was created from the controller that is created in Configuring a Liberty collective, the new controller's configuration must contain the following:
      <quickStartSecurity userName="adminUser" userPassword="adminPassword" />
        <!-- collective root signers keystore -->
        <keyStore id="collectiveRootKeys" password="yourPassword"
          location="${server.config.dir}/resources/collective/rootKeys.jks" />
    5. Start the new replica by starting the new collective controller.
    6. Confirm that the original collective controller successfully connected to the new replica. Look for message CWWKX6009I in the messages.log file of the original collective controller and the replica.
      Tip: For scripts that run the replicate and addReplica commands, add a 10-second wait after the replicate command runs to ensure the original collective controller and replica connect before running the addReplica command.
    7. Invoke the addReplica operation on the collective utility to activate the new replica. The argument to the addReplica set must be the replica endpoint (in the form "replicaHost:replicaPort") of the replica to add.
      wlp/bin/collective addReplica localhost:10012 
        --host=host_of_running_controller
        --port=port_of_running_controller
        --user=user_for_running_controller
        --password=user_password
  2. Optional: If necessary, you can modify the default configuration of the initial replica set. This step is recommended but not required.

    The configuration of the initial replica set occurs when the initial collective controller is created. If it is necessary to modify the default replica configuration, the following properties can be changed in the server.xml file:

    Optional settings: These values default, but can be altered.
    replicaHost
    Hostname for each individual replica
    replicaPort
    Port for each individual replica

    This port is not the http or https port of the collective controller, but it is a unique port that is used for communication between the replicas of the replica set.

    repositoryDir
    Directory location that is used to store the repository data

Example: Creating and activating a replica set

This example describes how to create a replica set and then activate it. A replica set must have at least three replicas, preferably on different hosts, for high availability. This example has the replicas on the same host, which requires that you assign unique port numbers for the replicas. When replicas are on different hosts, the replicas can use the same port numbers.

  1. Create a replica set.

    To create a replica set, you increase the number of collective controllers and configure them so that they can communicate with each other. Each new collective controller is called a replica because the added collective controllers have the same security configuration as the original controller and because all information written to any one controller is automatically replicated to all other active controllers. Once configured, all collective controllers in the replica set can perform the same operations as the original controller.

    1. If you do not have a collective controller, create one. See step 1 of Configuring a Liberty collective.
    2. Ensure the existing collective controller is running. For an existing controller named myController, run the status command:
      wlp/bin/server status myController
      If the collective controller is not running, then start it using the start or run command:
      wlp/bin/server start myController
    3. Create a server to be the new collective controller.
      wlp/bin/server create myController2
    4. Replicate the existing collective controller configuration into the new collective controller. The new collective controller is called a replica.

      Run a replicate command that uses the administrative security domain configuration of the existing collective controller and sets a new keystore password for the replica. Look at the server.xml file of the existing collective controller to find the values for the --host, --port, --user, and --password parameters. For --keystorePassword, set a value to use for the keystore, such as myController2. For information about these required parameters and about optional parameters, run collective help replicate at a command line.

      wlp/bin/collective replicate myController2 --host=host_of_existing_controller --port=https_port_of_existing_controller --user=userName_for_existing_controller --password=userPassword_for_existing_controller --keystorePassword=keystore_password_for_new_controller  

      If prompted to accept the certificate chain, enter y (yes).

      The replicate command writes XML output to a console screen. You copy the output into the server.xml file.

      To write the output of the command to a file, instead of to a console screen, specify the --createConfigFile=output_file_path parameter. Then, include the outputted file in the collective configuration by adding an include statement to the server.xml file:
      <include location=output_file_path />
    5. Add the XML output of the replicate command into the server.xml file of the replica and edit the parameter values as needed.
      • Ensure that the httpEndpoint element sets replica httpPort and httpsPort values that are unique port numbers on the host. For example, suppose the original controller named myController and the replica are both on the same localhost and myController has the following the httpEndpoint element:
        <httpEndpoint id="defaultHttpEndpoint"
                      host="*"
                      httpPort="9080"
                      httpsPort="9443" />
        Change the values for myController2 to:
        <httpEndpoint id="defaultHttpEndpoint"
                      host="*"
                      httpPort="9085"
                      httpsPort="9448" />
      • Set RPC credentials for hostAuthInfo. You can set RPC credentials for the replica in either of two ways:
        • Set hostAuthInfo RPC user and password values. Set rpcUser to an operating system login user ID for the host on which the replica resides, and set rpcUserPassword to the operating system login password for the user ID. For example, if you log into the replica computer with user test1 and password test1pwd, then change the hostAuthInfo element to the following:
          <hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
        • If the replica host is registered with the collective controller, set hostAuthInfo useHostCredentials to true for the replica to inherit RPC credentials from its host.
          <hostAuthInfo useHostCredentials="true" />

        See Overriding Liberty server host information for information about hostAuthInfo settings.

      • Ensure that replicaPort sets a port number for the replica that is unique among the replica set and that replicaSet sets host:port values that identify the replica set. For example, if the original controller named myController and the replica are both on the same localhost, change the values for myController2 from null:
        <collectiveController replicaPort="null"
                              replicaSet="localhost:null"
                              isInitialReplicaSet="false" />
        to 10011 for the replica port and 10010 for the replica set port:
        <collectiveController replicaPort="10011"
                              replicaSet="localhost:10010"
                              isInitialReplicaSet="false" />
      • Ensure the security configuration sets the same values as those used by the original controller. For example, both myController and the myController2 replica use:
        <quickStartSecurity userName="adminUser" userPassword="adminPassword" />
      • Ensure the collective root signers keystore element sets the same password as that used by the original controller. For example, copy the collectiveRootKeys keystore password from the myController server.xml file and paste it into the myController2 replica server.xml file. This example shows a generated password:
        <!-- collective root signers keystore -->
        <keyStore id="collectiveRootKeys" password="{xor}Lz4sLCgwLTs="
                  location="${server.config.dir}/resources/collective/rootKeys.jks"/>
    6. Start the new replica by starting the new collective controller.
      wlp/bin/server start myController2
    7. Verify that the original collective controller can communicate with the new replica.
      1. Open an editor on the original controller messages log, $WLP_USER_DIR/servers/myController/logs/messages.log.
      2. Look for the following message, which might have different IP addresses in your environment:
        CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10011. Current active replica set is [127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010]. The connected standby replicas are [127.0.0.1:10011].
    8. Verify that the new replica can communicate with the original collective controller.
      1. Open an editor on the replica messages log, $WLP_USER_DIR/servers/myController2/logs/messages.log.
      2. Look for the following message, which might have different IP addresses in your environment:
        CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10010. Current active replica set is []. The configured replica set is []. The connected standby replicas are [127.0.0.1:10011, 127.0.0.1:10010].
  2. Activate the new replica.

    Run an addReplica command that uses the administrative security domain configuration of the collective controller and specifies the endpoint of the replica that you want to activate in the form replicaHost:replicaPort. Look at the server.xml file of the collective controller to find the values for the --host, --port, --user, and --password parameters. Look at the server.xml file of the replica to find the values for replicaHost:replicaPort. For information about these parameters, run collective help addReplica at a command line.

    wlp/bin/collective addReplica replicaHost:replicaPort --host=host_of_existing_controller --port=port_of_existing_controller --user=user_for_existing_controller --password=user_password

    For this example, which has the existing collective controller and the replica on the same host, localhost, run:

    wlp/bin/collective addReplica localhost:10011 --host=localhost --port=9443 --user=adminUser --password=adminPassword

    If prompted to accept the certificate chain, enter y (yes).

  3. Repeat steps 1 and 2 for additional replicas. For example, add a third replica to the replica set. Name the new replica myController3 and specify replicaPort="10012". A replica set must have at least three replicas for high availability.
    After the third replica is added to the replica set, you can verify that the original collective controller and new replicas synchronized successfully.
    • Look for the following messages in the original controller messages log. The messages might have different IP addresses in your environment:
      CWWKX6015I: A request to change the active collective controller replica set was received and is now processing. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011}. The requested new active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}.
      
      CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}.
      
      CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012].
      
      CWWKX6014I: This collective controller replica finished synchronizing the data with the other replicas.
    • Look for the following messages in the messages logs of the added replicas. The messages might have different IP addresses in your environment:
      CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}.
      
      CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012].
      
      CWWKX8112I: The server's host information was successfully published to the collective repository.
      
      CWWKX8114I: The server's paths were successfully published to the collective repository.
      
      CWWKX8116I: The server STARTED state was successfully published to the collective repository.

Icon that indicates the type of topic Task topic



Timestamp icon Last updated: Monday, 5 December 2016
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=cord&product=was-libcore-mp&topic=tagt_wlp_configure_replicas
File name: tagt_wlp_configure_replicas.html