Configuring Liberty collective replica sets
You can configure collective replica sets. A replica set provides highly available management capabilities for a Liberty administrative domain.


Before you begin
The collectiveController-1.0 feature and its capabilities are available only in WebSphere® Application Server Liberty Network Deployment and WebSphere Application Server Liberty for z/OS®. The feature is not available in WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core. If you have a WebSphere Application Server Liberty Network Deployment installation, you can use its collectiveController-1.0 feature to work with collective members from WebSphere Application Server Liberty, or WebSphere Application Server Liberty Core installations. A collective member can be from any WebSphere Application Server Liberty release.
About this task
A replica set is a set of collective controllers that are configured to work together. Each replica contains all the repository updates that the other replicas within the set have processed. Therefore, there is no need for a member or client to connect with a particular collective controller each time that it interacts with the collective; any of the collective controllers that are configured in the replica set can provide the same data.
For detailed instructions about creating and configuring a collective controller, see Configuring a Liberty collective.
You can complete the following procedures to configure a collective replica set:
- Add a replica to an existing replica set.
- Modify the default configuration of the initial replica set.
See Example: Create and activate a replica set for an example of creating a replica set consisting of three collective controllers on the same host.
A replica set requires an odd number of replicas.
Procedure
- Add a replica to an existing replica set.
During the life of a replica set, it might become necessary to add one or more replicas to an existing set, to increase capacity for example.
The existing replicas in the replica set do not require any updates to their configuration. You can update them so that their configurations in the server.xml files more accurately reflect the replicas that form the replica set, but this update is unnecessary and does not affect their behaviors.
Note: It is not necessary to change the replicaSet value in the server.xml file of an existing replica in the set. No change to the configuration of an existing replica is required. If you do want to update the replicaSet values in the configurations of existing replicas in the set so that the configuration values are consistent across all replicas in the set, you must set the isInitialReplicaSet value in the configurations of the existing replicas to false. After you alter the replicaSet value, it describes a changed replica set rather than the initial replica set.Note: When you refer to a replica, you must be consistent and use the same host:port value. If a hostname is used, it must always be used. Or, if an IP address is used, it must always be used.To add a replica, follow these steps:
- Ensure that the existing replica set is running and that most of the replicas are available.
- Create a server to be the new collective controller.
wlp/bin/server create MyNewController
- Replicate to transform the new server into a collective controller.
wlp/bin/collective replicate MyNewController --host=host_of_running_controller --port=https_port_of_running_controller --user=userName_for_running_controller --password=userPassword_for_running_controller --keystorePassword=keystore_password_for_new_controller
The replicate command writes XML output to a console screen. You copy the output into the server.xml file.
To write the output of the command to a file, instead of to a console screen, specify the --createConfigFile=output_file_path parameter. Then, include the outputted file in the collective configuration by adding an include statement to the server.xml file:<include location=output_file_path />
- Configure the new replica's server.xml file by using output of the
replicate command. Copy the XML output of the replicate command into the server.xml file. You can modify the replica configuration as follows:Here is an example of what you might add to a new replica server.xml file:
- Required settings:The values must be explicitly set. The XML printed by the replicate command contains this configuration and is sufficient for these settings.
- hostAuthInfo
- Host authentication information containing properties that a remote client needs to start the
server. This setting is only required when the host of the new replica is not enabled for SSH.
Typically, Linux hosts are enabled for SSH and Windows hosts are not enabled for SSH. Thus, this
setting might only be needed on Windows hosts. Set RPC credentials for the replica in either of two
ways:
- Set rpcUser to an operating system login user ID for the host on which the
replica resides, and set rpcUserPassword to the operating system login password for
the user ID. For example, if you log into the replica computer with user test1 and
password test1pwd, then change the hostAuthInfo element to the
following:
<hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
- If the replica host is registered with the collective controller, set
hostAuthInfo
useHostCredentials to true for the replica to inherit RPC
credentials from its
host.
<hostAuthInfo useHostCredentials="true" />
See Overriding Liberty server host information for information about hostAuthInfo settings.
- Set rpcUser to an operating system login user ID for the host on which the
replica resides, and set rpcUserPassword to the operating system login password for
the user ID. For example, if you log into the replica computer with user test1 and
password test1pwd, then change the hostAuthInfo element to the
following:
- replicaSet
- Space-delimited list that contains the host:port for each of the
replicaHosts and replicaPorts in the replica set, excluding the
values for the collective controller now being added to the setFor example,
At least one of the values in this set must already be a replica of the existing replica set.original.host.com:10001 some.other.host.com:10003
- Optional settings:
These settings default, but can be altered.
- isInitialReplicaSet
- False
- replicaHost
- Hostname for each individual replica
- replicaPort
- Port for each individual replica
This port is not the http or https port of the collective controller, but it is a unique port that is used for communication between the replicas of the replica set.
- repositoryDir
- Directory location that is used to store repository data
<collectiveController replicaHost="localhost" replicaPort="10012" replicaSet="localhost:10010 localhost:10011" isInitialReplicaSet="false"/>
The XML printed by the replicate command requires updating the server's security configuration and specifying the collectiveRootKeys keystore password. The server's security configuration must be identical to the original collective controller's configuration, and the collectiveRootKeys keystore password must be the password that is used for the original collective controller's collectiveRootKeys keystore password. If the replica was created from the controller that is created in Configuring a Liberty collective, the new controller's configuration must contain the following:<quickStartSecurity userName="adminUser" userPassword="adminPassword" /> <!-- collective root signers keystore --> <keyStore id="collectiveRootKeys" password="yourPassword" location="${server.config.dir}/resources/collective/rootKeys.jks" />
- Required settings:
- Start the new replica by starting the new collective controller.
- Confirm that the original collective controller successfully connected to the new replica. Look
for message CWWKX6009I in the messages.log file of the original collective
controller and the replica. Tip: For scripts that run the replicate and addReplica commands, add a 10-second wait after the replicate command runs to ensure the original collective controller and replica connect before running the addReplica command.
- Invoke the addReplica operation on the collective utility to activate the new
replica. The argument to the addReplica set must be the replica endpoint (in the
form "replicaHost:replicaPort") of the replica to add.
wlp/bin/collective addReplica localhost:10012 --host=host_of_running_controller --port=port_of_running_controller --user=user_for_running_controller --password=user_password
- Optional: If necessary, you can modify the default configuration of the initial replica set. This step is
recommended but not required.
The configuration of the initial replica set occurs when the initial collective controller is created. If it is necessary to modify the default replica configuration, the following properties can be changed in the server.xml file:
Optional settings: These values default, but can be altered.- replicaHost
- Hostname for each individual replica
- replicaPort
- Port for each individual replica
This port is not the http or https port of the collective controller, but it is a unique port that is used for communication between the replicas of the replica set.
- repositoryDir
- Directory location that is used to store the repository data
Example: Creating and activating a replica set
This example describes how to create a replica set and then activate it. A replica set must have at least three replicas, preferably on different hosts, for high availability. This example has the replicas on the same host, which requires that you assign unique port numbers for the replicas. When replicas are on different hosts, the replicas can use the same port numbers.
- Create a replica set.
To create a replica set, you increase the number of collective controllers and configure them so that they can communicate with each other. Each new collective controller is called a replica because the added collective controllers have the same security configuration as the original controller and because all information written to any one controller is automatically replicated to all other active controllers. Once configured, all collective controllers in the replica set can perform the same operations as the original controller.
- If you do not have a collective controller, create one. See step 1 of Configuring a Liberty collective.
- Ensure the existing collective controller is running. For an existing controller named
myController, run the status
command:
If the collective controller is not running, then start it using the start or run command:wlp/bin/server status myController
wlp/bin/server start myController
- Create a server to be the new collective
controller.
wlp/bin/server create myController2
- Replicate the existing collective controller configuration into the new collective controller.
The new collective controller is called a replica.
Run a replicate command that uses the administrative security domain configuration of the existing collective controller and sets a new keystore password for the replica. Look at the server.xml file of the existing collective controller to find the values for the --host, --port, --user, and --password parameters. For --keystorePassword, set a value to use for the keystore, such as myController2. For information about these required parameters and about optional parameters, run collective help replicate at a command line.
wlp/bin/collective replicate myController2 --host=host_of_existing_controller --port=https_port_of_existing_controller --user=userName_for_existing_controller --password=userPassword_for_existing_controller --keystorePassword=keystore_password_for_new_controller
If prompted to accept the certificate chain, enter y (yes).
The replicate command writes XML output to a console screen. You copy the output into the server.xml file.
To write the output of the command to a file, instead of to a console screen, specify the --createConfigFile=output_file_path parameter. Then, include the outputted file in the collective configuration by adding an include statement to the server.xml file:<include location=output_file_path />
- Add the XML output of the replicate command into the
server.xml file of the replica and edit the parameter values as needed.
- Ensure that the httpEndpoint element sets replica httpPort and
httpsPort values that are unique port numbers on the host. For example, suppose the
original controller named myController and the replica are both on the same
localhost and myController has the following the httpEndpoint
element:
Change the values for myController2 to:<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080" httpsPort="9443" />
<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9085" httpsPort="9448" />
- Set RPC credentials for hostAuthInfo. You can set RPC credentials for the
replica in either of two ways:
- Set hostAuthInfo RPC user and password values. Set rpcUser to
an operating system login user ID for the host on which the replica resides, and set
rpcUserPassword to the operating system login password for the user ID. For
example, if you log into the replica computer with user test1 and password
test1pwd, then change the hostAuthInfo element to the
following:
<hostAuthInfo rpcUser="test1" rpcUserPassword="test1pwd" />
- If the replica host is registered with the collective controller, set
hostAuthInfo
useHostCredentials to true for the replica to inherit RPC
credentials from its
host.
<hostAuthInfo useHostCredentials="true" />
See Overriding Liberty server host information for information about hostAuthInfo settings.
- Set hostAuthInfo RPC user and password values. Set rpcUser to
an operating system login user ID for the host on which the replica resides, and set
rpcUserPassword to the operating system login password for the user ID. For
example, if you log into the replica computer with user test1 and password
test1pwd, then change the hostAuthInfo element to the
following:
- Ensure that replicaPort sets a port number for the replica that is unique among
the replica set and that replicaSet sets
host:port values that identify the replica
set. For example, if the original controller named myController and the replica are
both on the same localhost, change the values for myController2 from
null:
to 10011 for the replica port and 10010 for the replica set port:<collectiveController replicaPort="null" replicaSet="localhost:null" isInitialReplicaSet="false" />
<collectiveController replicaPort="10011" replicaSet="localhost:10010" isInitialReplicaSet="false" />
- Ensure the security configuration sets the same values as those used by the original controller.
For example, both myController and the myController2 replica
use:
<quickStartSecurity userName="adminUser" userPassword="adminPassword" />
- Ensure the collective root signers keystore element sets the same password as that used by the
original controller. For example, copy the collectiveRootKeys keystore password
from the myController
server.xml file and paste it into the myController2 replica
server.xml file. This example shows a generated
password:
<!-- collective root signers keystore --> <keyStore id="collectiveRootKeys" password="{xor}Lz4sLCgwLTs=" location="${server.config.dir}/resources/collective/rootKeys.jks"/>
- Ensure that the httpEndpoint element sets replica httpPort and
httpsPort values that are unique port numbers on the host. For example, suppose the
original controller named myController and the replica are both on the same
localhost and myController has the following the httpEndpoint
element:
- Start the new replica by starting the new collective
controller.
wlp/bin/server start myController2
- Verify that the original collective controller can communicate with the new replica.
- Open an editor on the original controller messages log, $WLP_USER_DIR/servers/myController/logs/messages.log.
- Look for the following message, which might have different IP addresses in your
environment:
CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10011. Current active replica set is [127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010]. The connected standby replicas are [127.0.0.1:10011].
- Verify that the new replica can communicate with the original collective controller.
- Open an editor on the replica messages log, $WLP_USER_DIR/servers/myController2/logs/messages.log.
- Look for the following message, which might have different IP addresses in your
environment:
CWWKX6009I: The collective controller successfully connected to replica 127.0.0.1:10010. Current active replica set is []. The configured replica set is []. The connected standby replicas are [127.0.0.1:10011, 127.0.0.1:10010].
- Activate the new replica.
Run an addReplica command that uses the administrative security domain configuration of the collective controller and specifies the endpoint of the replica that you want to activate in the form replicaHost:replicaPort. Look at the server.xml file of the collective controller to find the values for the --host, --port, --user, and --password parameters. Look at the server.xml file of the replica to find the values for replicaHost:replicaPort. For information about these parameters, run collective help addReplica at a command line.
wlp/bin/collective addReplica replicaHost:replicaPort --host=host_of_existing_controller --port=port_of_existing_controller --user=user_for_existing_controller --password=user_password
For this example, which has the existing collective controller and the replica on the same host, localhost, run:
wlp/bin/collective addReplica localhost:10011 --host=localhost --port=9443 --user=adminUser --password=adminPassword
If prompted to accept the certificate chain, enter y (yes).
- Repeat steps 1 and 2 for additional replicas. For example, add a third replica to the replica
set. Name the new replica myController3 and specify
replicaPort="10012". A replica set must have at least three replicas for high
availability.After the third replica is added to the replica set, you can verify that the original collective controller and new replicas synchronized successfully.
- Look for the following messages in the original controller messages log. The messages might have
different IP addresses in your
environment:
CWWKX6015I: A request to change the active collective controller replica set was received and is now processing. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011}. The requested new active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}. CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012]. CWWKX6014I: This collective controller replica finished synchronizing the data with the other replicas.
- Look for the following messages in the messages logs of the added replicas. The messages might
have different IP addresses in your
environment:
CWWKX6016I: The active collective controller replica set changed successfully. The current active replica set is {127.0.0.1:10010,127.0.0.1:10011,127.0.0.1:10012}. The previous active replica set was {127.0.0.1:10010,127.0.0.1:10011}. CWWKX6011I: The collective controller is ready, and can accept requests. The leader is 127.0.0.1:10010. Current active replica set is [127.0.0.1:10012, 127.0.0.1:10011, 127.0.0.1:10010]. The configured replica set is [127.0.0.1:10010, 127.0.0.1:10011, 127.0.0.1:10012]. CWWKX8112I: The server's host information was successfully published to the collective repository. CWWKX8114I: The server's paths were successfully published to the collective repository. CWWKX8116I: The server STARTED state was successfully published to the collective repository.
- Look for the following messages in the original controller messages log. The messages might have
different IP addresses in your
environment:


http://www14.software.ibm.com/webapp/wsbroker/redirect?version=cord&product=was-libcore-mp&topic=tagt_wlp_configure_replicas
File name: tagt_wlp_configure_replicas.html