Administration Guide

HACMP ES Event Monitoring and User-defined Events

Shutting down DB2 database partitions on an AIX physical node when paging space reaches a certain percentage of fullness, and restarting a DB2 database partition, or initiating a failover operation if a process dies on a given node, are two examples of user-defined events. Examples that illustrate user-defined events, such as shutting down a database partition and forcing a transaction abort to free paging space, can be found in the samples subdirectory.

A rules file, /user/sbin/cluster/events/rules.hacmprd, contains HACMP events. Each event description in this file has the following nine components:

Each object requires one line in the event definition, even if the line is not used. If these lines are removed, HACMP ES Cluster Manager cannot parse the event definition properly, and this may cause the system to hang. Any line beginning with "#" is treated as a comment line.
Note:The rules file requires exactly nine lines for each event definition, not counting any comment lines. When adding a user-defined event at the bottom of the rules file, it is important to remove the unnecessary empty line at the end of the file, or the node will hang.

Following is an example of an event definition for node_up: 

   ##### Beginning of the Event Definition: node_up
   #
   TE_JOIN_NODE
   0
   /usr/sbin/cluster/events/node_up.rp
   2
   0
   # 6) Resource variable - only used for event management events
 
   # 7) Instance vector - only used for event management events
 
   # 8) Predicate  - only used for event management events
 
   # 9) Rearm predicate - only used for event management events
 
   ###### End of the Event Definition: node_up

This example is just one of the event definitions that can be found in the rules.hacmprd file. In this example, the recovery program /usr/sbin/cluster/events/node_up.rp is invoked when the node_up event occurs. Values are specified for the state, recovery type, and recovery level. There are four empty lines for resource variable, instance variable, predicate, and rearm predicate.

You can define other events to react to non-standard HACMP ES events. For example, to define the event that the /tmp file system is over 90 per cent full, the rules.hacmprd file must be modified.

Many events are predefined in the IBM Parallel System Support Program (PSSP). These events can be exploited (when used within user-defined events) as follows:

  1. Stop the cluster.
  2. Edit the rules.hacmprd file. Back up the file before modifying it. Add the predefined PSSP event manually. If you need synchronizing points across all nodes in the cluster, use the barrier command in the recovery program. (Read more about the barrier command, and synchronization of recovery programs in the HACMP Concepts, Installation, and Administration Guides.)
  3. Restart the cluster. The rules.hacmprd file is stored in memory when Cluster Manager is started. To accurately implement the changes, restart all the clusters. There should not be any inconsistent rules in a cluster.
  4. Cluster Manager uses all events in the rules.hacmprd file.

HACMP ES uses PSSP event detection to treat user-defined events. The PSSP Event Management subsystem provides comprehensive event detection by monitoring various hardware and software resources.

Resource states are represented by resource variables. Resource conditions are represented as expressions called predicates.

Event Management receives resource variables from the Resource Monitor, which observes the state of specific system resources and transforms this state into several resource variables. These variables are periodically passed to Event Management. Event Management applies predicates that are specified by the HACMP ES Cluster Manager in rules.hacmprd for each resource variable. When the predicate is evaluated as being true, an event is generated and sent to the Cluster Manager. Cluster Manager initiates the voting protocol, and the recovery program file (xxx.rp) is run (according to event priority) on a set of nodes specified by "node sets" in the recovery program.

The recovery program file (xxx.rp) is made up of one or more recovery program lines. Each line is declared in the following format: 

   relationship     command_to_run     expected_status     NULL

There must be at least one space between each value in the line. "Relationship" is a value used to decide which program should run on which kind of node. Three types of relationship are supported:

"Command_to_run" is a quotation mark-delimited string, with or without a full-path specification to an executable program. Only HACMP-delivered event scripts can use a relative-path definition. Other scripts or programs must use the full-path specification, even if they are located in the same directory as the HACMP event scripts.

"Expected_states" is the return code of the specified command or program. It is either an integer value, or an "x". If "x" is used, Cluster Manager does not care about the return code. All other codes must be equal to the expected return code, otherwise Cluster Manager detects the event failure. The handling of this event "hangs" the process until recovery (through manual intervention) occurs. Without manual intervention, the node does not synchronize with the other nodes. Synchronization across all nodes is required for the Cluster Manager to control all the nodes.

"NULL" is a field reserved for future use. The word "NULL" must appear at the end of each line except the barrier line. If you specify multiple recovery commands between two barrier commands, or before the first one, the recovery commands are run in parallel on the node itself, and between the nodes.

The barrier command is used to synchronize all the commands across all the cluster nodes. When a node hits the barrier statement in the recovery program, Cluster Manager initiates the barrier protocol on this node. Since the barrier protocol is a two-phase protocol, all nodes are notified that both phases have completed when all of the nodes have met the barrier in the recovery program, and "voted" to approve the protocol.

The process can be summarized as follows:

  1. Either Group Services/ES (for predefined events) or Event Management (for user-defined events) notifies HACMP ES Cluster Manager of the event.
  2. Cluster Manager reads the rules.hacmprd file, and determines the recovery program that is mapped to the event.
  3. Cluster Manager runs the recovery program, which consists of a sequence of recovery commands.
  4. The recovery program executes the recovery commands, which may be shell scripts or binary commands. (In HACMP for AIX, the recovery commands are the same as the HACMP event scripts.)
  5. Cluster Manager receives the return status from the recovery commands. An unexpected status "hangs" the cluster until manual intervention (using smit cm_rec_aids or the /usr/sbin/cluster/utilities/clruncmd command) is carried out.

HACMP ES Script Files

The following sample scripts for failover recovery and user-defined events are included with DB2 UDB EEE. The script files are located in the $INSTNAME/sqllib/samples/hacmp/es directory. The scripts will work "as is", or you can customize the recovery action.

The recovery scripts must be installed on each node that will run recovery operations. The script files can be centrally installed from the SP control workstation or other designated SP node:

  1. Copy the scripts from the $INSTNAME/sqllib/samples/hacmp/es directory to one of either the SP control workstation or another SP node that can run the pcp and pexec commands. These commands are required for the install operation.
  2. Customize the reg.parms.SAMPLE and failover.parms.SAMPLE files for your environment by setting key parameters (such as BUFFPAGE) for failover configurations. Typically, for mutual takeover configurations, your failure settings will be adjusted lower to one-half the size of your regular settings or less. Also, you will use a copy of these files renamed with your own name (instead of "SAMPLE").
  3. Customize (as necessary) the five parameters NFS_RETRIES, START_RETRIES, MOUNT_NFS, STOP_RETRIES, and FAILOVER in the rc.db2pe file. The retry and failover settings should be adequate for most implementations. The MOUNT_NFS setting should be configured, depending on whether you will be using the package for NFS server availability. You should specify this setting if you want rc.db2pe to mount and verify the NFS home directory of the DB2 instance owner for you. Setting the FAILOVER parameter to "YES" will invoke db2_proc_restart and launch an attempt to restart a DB2 database partition. If the restart operation is unsuccessful, HACMP will shut down with a failover.
  4. Customize db2_paging_action, db2_proc_recovery, and nfs_auto_recovery in the event file. Edit pwq to change this to the DB2 instance owner. Customize db2_paging_action to specify which action is to be taken if paging space becomes more that ninety percent full. (If this does occur, the DB2 database partition is stopped.) Modify the script if additional recovery actions are required.
  5. Use db2_inst_ha to install the scripts and events on the nodes you specify. (HACMP ES must be pre-installed on these nodes before you begin.) The syntax of db2_inst_ha is: 

       db2_inst_ha $INSTNAME/sqllib/samples/hacmp/es <nodelist> <DATABASENAME>
 
   where
 
      $INSTNAME/sqllib/samples/hacmp/es is the directory in which the
         scripts and the event are located
      <nodelist> is the pcp or pexec style of the nodes; for example, 
         1-16 or 1,2,3,4
      <DATABASENAME> is the name of the database for regular and 
         failover parameter files.

    The reg.parms.SAMPLE and failover.parms.SAMPLE files will be copied to each node and renamed reg.parms.DATABASENAME. db2_inst_ha copies files to each node in /usr/bin, and updates the HACMP event files: 

       /usr/sbin/cluster/events/rules.hacmprd
   /usr/sbin/cluster/events/network_up_complete
   /usr/sbin/cluster/events/network_down_complete
  6. Configure your system and scripts with HACMP.
  7. Use the create_db2_events command to install the monitoring events for problem management resources (pman) and the SP GUI Perspectives. Additional configuration and customization in Perspectives is needed. For more information about Perspectives, read the PSSP Administration Guide.
  8. Use the ha_db2stop command to shutdown the database partitions without HACMP ES failover recovery taking place. To use this command, copy the file to the database user's home directory and make sure permissions and ownership are set for that user. To stop the database without failover recovery, then as that user, type: 
       ha_db2stop
    Note:You must wait for the command to return. Exiting by using a ctrl-C interrupt, or by killing the process, may re-enable failover recovery prematurely, and some database partitions may not be stopped.

DB2 Recovery Script Operations with HACMP ES

HACMP ES invokes the DB2 recovery scripts in the following way:

Note:All events can be monitored through SP problem management and the SP Perspectives GUI.

Other Script Utilities

Other script utilities are available for your use, including:

[ Top of Page | Previous Page | Next Page ]