The DB2 high availability agent acts like a mediator between DB2 and SC2.x. It provides a way for the Sun Cluster 2.2 software to control DB2 in a clustered environment, without having intimate knowledge of DB2. There is one agent for both EE and EEE instances. The agent supports both administrative instances and database instances.
To work with SC2.2, the DB2 HA agent must be registered. Registering a data service tells SC2.2 which control methods are available, and in which directory they reside. A special script called hadb2_reg, which is shipped with the HA agent, can register the hadb2 service for both EE and EEE instances. The hadb2_reg script needs to be run only once for the entire cluster.
Although there is only one set of control methods for the DB2 HA agent, the way they are registered depends on whether or not an EEE instance will be used in a mutual takeover configuration. For an EE instance or EEE instance in a hot standby configuration, HA-NFS is not used; therefore, the "-d nfs" switch, which tells the SC2.2 software that the hadb2 service is dependent on HA-NFS, is not needed.
The actual command that hadb2_reg uses to register the DB2 V7.1 control methods for an EEE instance is:
hareg -r hadb2 -b /opt/IBMdb2/V7.1/ha -m START=hadb2_start,START_NET=hadb2_startnet,STOP_NET=hadb2_stopnet, FM_START=hadb2_fmstart,FM_STOP=hadb2_fmstop -t START_NET=$TIMEOUT,STOP_NET=$TIMEOUT -d nfs
The -b switch tells SC2.x to look in the opt/IBMdb2/V7.1/ha directory for all of the control methods. The -m switch defines the actual control methods for the hadb2 service. The -t switch defines the timeout for the START_NET and STOP_NET control methods. For a detailed description of each control method, refer to the Sun Cluster documentation.
The hadb2_unreg script can be used to unregister the hadb2 service and, like hadb2_reg, needs to be run only once for the cluster.
The hadb2tab file is the main configuration file for the DB2 HA agent. Each control method consults this file to find out which instances are highly available. The hadb2tab file is located under the /var/db2/v71/ directory for DB2 UDB Version 7.1. The file supports multiple instances, and each non-commented line represents a different HA instance. Following is an example of an hadb2tab file:
<scadmin@thrash(203)# cat hadb2tab EEE DATA db2eee jolt ON /export/ha_home /log0/home #Added by DB2 HA software EE ADMIN db2ee log1 ON - - #Added by DB2 HA software
The first field indicates to the DB2 HA agent whether the instance is an EE instance, or an EEE instance. The second field indicates whether the instance is a data instance, or an administrative instance. The third field contains the user name of the HA instance. The fourth field is the logical host or the HA-NFS host for the instance, depending on whether it is an EE or an EEE instance. The fifth field indicates whether fault monitoring for the instance is turned on or off. The last two fields are the local mount point, and the remote HA-NFS directory, respectively. These fields should be set to - (hyphen) if they are not used, and should only be used with an EEE mutual takeover configuration. Comments are allowed in the hadb2tab file if the information on the line before a "#" marker is either of zero length, or a valid definition of an instance.
Control methods for SC2.2 agents can be a set of scripts or programs. The agent for DB2 on Solaris is a set of programs that includes the following methods:
For more information about these control methods, refer to the Sun Cluster documentation.
For EE instances, the logical host that is associated with the instance is defined right in the hadb2tab file. For EEE instances, however, the control method must also look in:
~<instance>/sqllib/ha/hadb2-eee.cfg
where ~<instance> is the home directory of the instance owner. This file contains one line for each database partition, and is used to associate database partitions with logical hosts. An example of a valid hadb2-eee.cfg file is:
crackle % cat hadb2-eee.cfg NODE:log0 0 NODE:log0 1 NODE:log1 2 NODE:log1 3
The instance or database partitions follow the corresponding logical host around the cluster. The logical host can move to any machine in the cluster that is supported by the underlying hardware and SC2.2. If the configuration is properly set up, DB2 will support any topology that is supported by the SC2.2 software.
After reading all of the information for an instance, the control method knows which logical hosts are associated with the instance. After parsing the command line arguments, the control method also knows which logical hosts are hosted, and which are not hosted by the current machine.
The following table shows the actions that are taken, depending on which
control method is being run, and whether the logical hosts associated with the
database partition or instance are hosted on the current machine.
| Control Method | Associated logical host(s) are hosted | Associated logical host(s) are not hosted |
|---|---|---|
| START_NET | Start DB2 instance or database partitions | No action |
| STOP_NET | No action | Stop DB2 instance or database partitions |
| FM_START | Start fault monitor for instance | No action |
| FM_STOP | No action | Stop fault monitoring for instance |
The control methods that perform start actions are only concerned with the logical hosts that are currently being hosted, and the control methods that perform stop actions are only concerned with the logical hosts that are not currently being hosted.
The control methods also need to mount the HA-NFS directory in a special way if HA-NFS is being used. If the local mount point and directory for HA-NFS are not defined as - (hyphen), the control method runs a statvfs(2) on the local mount point. If the file system type for the local mount point is not nfs, the agent attempts to mount the file system using information from the hadb2tab line. If the mount point and the directory for HA-NFS are defined as - (hyphen), the vfstab file of the corresponding logical host is required to mount the file system containing the home directory of the instance. The local mount point and the remote directory for HA-NFS should only be defined as - (hyphen) for EE and EEE hot standby configurations.
These scripts are run from the control methods to add additional functionality; they are passed the same command line arguments as the control methods are passed, and are written by the system administrator or the database administrator.
If a program must be run from within a script that is not run in the background, consider backgrounding the program with nohup(1). The nohup program protects the executed program from the SIGHUP (or hangup) signal. Without nohup, a program that is run in the background from a script may die as a result of a SIGHUP signal when the script is finished.
The control methods run the following scripts:
where ~instance is the home directory of the HA instance.
With the exception of the fm_warning script, each user script is run with the same arguments as the control method that invoked it. When using EEE instances, the database partition number is also passed (as the last argument) to the user script.
The /var/db2/v71/failover script is invoked at the beginning of the START_NET method, and runs in the background. Such a script can be used, for example, to e-mail support staff in the event of a failover. Following is an example of a failover script:
#!/bin/ksh # E-mail or page support staff to notify them that a failover has occurred. echo "Failover occurred on machine `hostname`:Running $0!" |/bin/mail admin@sphere.torolab.ibm.com
To e-mail successfully from a script, sendmail(1m) must be properly configured on the system.
As its name suggests, the pre_db2start script is run just before db2start is invoked. This script can be used for such tasks as changing database manager configuration parameters. It is given a maximum of 20 seconds to complete. For EEE instances, this script is run before db2start is invoked on each database partition. This script is applicable only to data instances, not to administrative instances.
Similarly, the post_db2start script is run just after db2start is invoked. This script can be used for such tasks as restarting databases. It is run in the background to ensure that its execution time does not interfere with other instances. This script is applicable only to data instances, not to administrative instances.
The post_failover script under the instance owner's home directory, is run after processing the instance. This script can be used to notify client applications that DB2 is now functional, to activate databases, or to send administrators a status file. It is run in the background to prevent its execution time from delaying actions against the other HA instances. Following is an example of a post-failover script:
#!/bin/ksh # # Send the status file to the administrato-r. mail admin@sphere.torolab.ibm.com </tmp/HA.info.db2eee
Both the START_NET and the STOP_NET method of the DB2 HA agent create a status file after processing each instance. The name of the status file is:
/tmp/HA.info.<instance>
where instance is the user name of the instance owner. The status file contains the start and stop report for the instance, as well as the time it took to run the control method. Following is an example of a status file:
scadmin@crackle(173)# cat /tmp/HA.info.db2eee
----- Elapsed Time: 00:00:18 -----
----- Elapsed Time: 00:00:00 (HA-NFS) -----
NODE ACTION RESULT TRIES RC
---- ------ ------ ----- --
4 stop success 3 1064
5 stop success 1 1064
6 stop success 2 1064
7 stop success 2 1064
8 stop success 1 1064
---------------------------------------------
The pre_db2stop script is run just before db2stop is invoked. This script can be used to notify client applications that DB2 is about to stop. It is given a maximum of 20 seconds to complete. This script is applicable only to data instances, not to administrative instances.
The fault monitor will also run a user script when DB2 is restarted because of an unexpected shutdown. This script is called:
~<instance>/sqllib/ha/fm_warning
The fm_warning script can be used to notify the system administrator that DB2 was restarted by the fault monitor. The system administrator should try to find out why DB2 shut down unexpectedly, and take appropriate actions to prevent this from happening again. The fm_warning script is run in the background.
If an HA data service is turned off, only the stop methods are run during a failover or cluster reconfiguration; the other methods are run only if the HA data service is properly registered and turned on.
Ensure that each machine in the cluster has enough resources to run all of the data services for which it may be responsible. Resources such as CPU load, memory, swap and kernel parameters must be considered before the cluster goes into production. For example, if a machine in the cluster may need to run two DB2 instances, the kernel parameter requirements for that machine will be the sum of what is needed for each instance.
If fault monitoring is turned on, the fault monitor will be started during a cluster reconfiguration or failover. If DB2 is not started by the START_NET script, the fault monitor itself will start DB2. The fault monitor can detect if DB2 did not start, or if it shut down for unknown reasons. Because of this, it is important not to shut down DB2 manually when the fault monitor is turned on. The fault monitor will see this as an unexpected shutdown, and restart DB2. If this happens too many times, it will fail over the appropriate logical host.
When fault monitoring is enabled for an instance, the correct way to start or stop the instance manually is to first turn off fault monitoring or the hadb2 service. Both of these actions can be initiated through the hadb2_setup command using the -f and -s switches (see The hadb2_setup Command).
| Note: | Do not use more than one instance for the same logical host. If more than one instance is associated with a logical host, a healthy instance may be failed over along with an unhealthy one. |
When deciding which database partitions to associate with a logical host, it is important to consider how they will fail over. Consider a two-machine cluster that is to be used with four database partitions between the two machines, as shown in Figure 120.
Figure 120. Two-machine Cluster with Four Database Partitions
You could associate one logical host with each database partition, and one for HA-NFS. In this case, there could be a problem if all of the logical hosts are being hosted by one system. If that system fails, all of the logical hosts must be moved off the system at the same time. Unfortunately, the Sun Cluster software does not move the logical hosts in any predictable order, and it is possible for a logical host that has a database partition associated with it to move before the logical host with HA-NFS. It is usually a good idea to group database partitions together, according to what would be hosted on a single system. This means that two database partitions that are normally hosted on one machine should be associated with a single logical host.
The db2nodes.cfg file used by an EEE instance is updated to indicate the machine on which the database partitions are residing. For example, if all of the database partitions are on a machine called "crackle", the db2nodes.cfg file resembles the following:
scadmin@crackle(193)# cat db2nodes.cfg 0 crackle 0 204.152.65.33 1 crackle 1 204.152.65.33 2 crackle 2 204.152.65.33 3 crackle 3 204.152.65.33 4 crackle 4 204.152.65.33 5 crackle 5 204.152.65.33 6 crackle 6 204.152.65.33 7 crackle 7 204.152.65.33 8 crackle 8 204.152.65.33
If some of these database partitions are moved to a machine called "thrash", the db2nodes.cfg file is updated as follows:
scadmin@crackle(193)# cat db2nodes.cfg 0 crackle 0 204.152.65.33 1 crackle 1 204.152.65.33 2 crackle 2 204.152.65.33 3 crackle 3 204.152.65.33 4 thrash 0 204.152.65.34 5 thrash 1 204.152.65.34 6 thrash 2 204.152.65.34 7 thrash 3 204.152.65.34 8 thrash 4 204.152.65.34
Notice that both the host name and the switch name are changed to reflect the machine name "thrash", and that the port numbers are also different.
If it exists, the /etc/HA.config file can contain a number of configuration options, including the following:
scadmin@thrash(204)# cat /etc/HA.config SYSLOG_FACILITY=LOG_LOCAL3 SYSLOG_LPRIORITY=LOG_INFO SYSLOG_EPRIORITY=LOG_ERR USE_INTERCONNECT=auto SWITCH_NAME=204.152.65.18 DEBUG_LEVEL=2 FAILS_PER_HOUR=2 FAILS_PER_DAY=4 FAILS_PER_WEEK=10 FM_FAIL_SEV=soft DB2START_TIMEOUT=60 DB2STOP_TIMEOUT=500 SCRIPT_USER=bin
| Note: | If the HA.config file does not exist, default values are used. |
The SYSLOG_FACILITY variable sets the SYSLOG facility for logging both messages and errors. The SYSLOG_LPRIORITY and SYSLOG_EPRIORITY variables set the SYSLOG priority for logging informational messages and error messages, respectively.
Some changes may be needed to enable the SYSLOG daemon to log information from the DB2 HA agent. For example, one of the following two lines added to the /etc/syslog.conf file will tell the SYSLOG daemon to write information to a log file.
*.notice /var/adm/SC.x local3.info /var/adm/SC.LOG_LOCAL3
A Sun Cluster usually has a high speed interconnect. To use the high speed interconnect with DB2, set USE_INTERCONNECT to auto or to override. The auto setting (the default) uses the Sun internal logical network interface. This interface will be transferred to another physical interface if the initial interface fails. If USE_INTERCONNECT is set to override, the switch name is taken from the SWITCH_NAME variable. Another option is to set USE_INTERCONNECT to no, which specifies that high speed interconnect is not to be used.
DEBUG_LEVEL specifies how much information is to be logged during a failover. It is a number between 0 and 10, where 10 is the highest debug level. The information is logged at the specified SYSLOG priority and facility. If any problems are encountered, set the debug level to the maximum level, configure SYSLOG to log the output from the HA agents, and send the SYSLOG output to IBM service.
Three of the variables help the DB2 fault monitor decide when to fail over a logical host: FAILS_PER_HOUR, FAILS_PER_DAY, and FAILS_PER_WEEK. Every HA environment is different; you must decide how many DB2 failures are acceptable. After each "acceptable" failure, DB2 is restarted on the same machine. When one of these three failure thresholds is exceeded, the logical host associated with the instance or database partition is failed over.
The FM_FAIL_SEV variable specifies whether the failover is "soft" or "hard". For more information, refer to the Sun Cluster documentation on hactl(1m).
The DB2START_TIMEOUT and DB2STOP_TIMEOUT variables specify the maximum number of seconds that db2start and db2stop are allowed to run. After the specified interval has passed, the HA agent considers the operation to have failed, and try to restart the instance.
There are some user scripts that are not associated with any particular instance. Normally, these scripts are run as root; this can be overridden by the SCRIPT_USER variable, which can be set to specify the user ID that can run these scripts.
The DB2 HA agent uses the su command to run commands as the instance owner. The actual command would look something like:
su - <instance> -c "db2stop"
where instance is the user name of the instance.
It is important to ensure that the .profile file of the instance owner is su-"friendly". If it is not, the su command may not work properly. Invoke the su command manually, or from a script, to verify that the command can run successfully.