Scenario: Recovering from a Db2 outage at a requester that results in a Db2 cold start
When an outage at a Db2 requester results in a cold start, the organization that has this situation can recover.
Symptoms
The Db2 subsystem at SEA is started with a conditional restart record in the BSDS to indicate a cold start:
- When the IMS subsystem
reconnects, it attempts to resolve the indoubt thread that is identified
in IMS as
NID=A5. IMS has
a resource recovery element (RRE) for this thread. The SEA Db2 subsystem
informs IMS that
it has no knowledge of this thread. IMS does
not delete the RRE, and the RRE can be displayed by using the IMS DISPLAY
OASN command. The SEA Db2 subsystem
also:
- Generates message DSN3005 for each IMS RRE for which Db2 has no knowledge
- Generates an IFCID 234 trace event
- When the Db2 subsystems
at SJ and LA reconnect with SEA, each detects that the SEA Db2 subsystem
has cold started. Both the SJ Db2 and the
LA Db2 subsystem:
- Display message DSNL411
- Generate alert A001
- Generate an IFCID 204 trace event
- A DISPLAY THREAD report of indoubt threads at both the SJ and LA Db2 subsystems shows the indoubt threads and indicates that the coordinator has cold started.
Causes
An abnormal termination of the SEA Db2 subsystem caused the outage.Environment
The following figure illustrates the environment for this scenario.

The abnormal termination of the SEA Db2 subsystem has left the two DBATs at SJ 1, 3, and the LUWID=16 DBAT at LA 4 indoubt. The LUWID=15 DBAT at LA 2, connected to SJ, is waiting for the SJ Db2 subsystem to communicate the final decision.
The IMS subsystem at SEA is operational and has the responsibility of resolving indoubt units with the SEA Db2 subsystem.
The Db2 subsystems at both SJ and LA accept the cold start connection from SEA. Processing continues, waiting for the heuristic decision to resolve the indoubt threads.
Resolving the problem
Database administrator response:
At this point:- Neither the SJ nor the LA administrator know if the SEA coordinator was a participant of another coordinator. In this scenario, the SEA Db2 subsystem originated LUWID=16. However, the SEA Db2 subsystem was a participant for LUWID=15, which was being coordinated by IMS.
- The administrator at LA also does not know is the fact that SEA distributed the LUWID=16 thread to SJ, where it is also indoubt. Likewise, the administrator at SJ does not know that LA has an indoubt thread for the LUWID=16 thread. Both SJ and LA need to make the same heuristic decision. The administrators at SJ and LA also need to determine the originator of the two-phase commit.
- The recovery log of the originator indicates whether the decision was commit or abort. The originator might have more accessible functions to determine the decision. Even though the SEA Db2 subsystem cold started, you might be able to determine the decision from its recovery log. Alternatively, if the failure occurred before the decision was recorded, you might be able to determine the name of the coordinator, if the SEA Db2 subsystem was a participant. You can obtain a summary report of the SEA Db2 recovery log by running the DSN1LOGP utility.
- The LUWID contains the name of the logical unit (LU) where the distributed logical unit of work originated. This logical unit is most likely in the system that originated the two-phase commit.
- If an application is distributed, any distributed piece of the application can initiate the two-phase commit. In this type of application, the originator of two-phase commit can be at a different system than the site that is identified by the LUWID.
- The administrator must determine if the LU name that is contained in the LUWID is the same as the LU name of the SEA Db2 subsystem. If this is not the case (it is the case in this example), the SEA Db2 subsystem is a participant in the logical unit of work, and it is being coordinated by a remote system. The DBA must communicate with that system and request that facilities of that system be used to determine if the logical unit of work is to be committed or aborted.
- If the LUWID contains the LU name of the SEA Db2 subsystem,
the logical unit of work originated at SEA and can be an IMS, CICS®, TSO, or batch allied thread
of the SEA Db2 subsystem.
The DISPLAY THREAD report for indoubt threads at
a Db2 participant
includes message DSNV458 if the coordinator is remote.
This line provides external information that is provided by the coordinator
to assist in identifying the thread. A Db2 coordinator
provides the following identifier:
where connection-name is:connection-name.correlation-id- SERVER: the thread represents a remote application to the Db2 coordinator and uses DRDA access.
- BATCH: the thread represents a local batch application to the Db2 coordinator.
- Anything else represents an IMS or CICS connection name. The thread represents a local application, and the commit coordinator is the IMS or CICS system by using this connection name.
- In this example, the administrator at SJ sees that both indoubt threads have an LUWID with the LU name that match the SEA Db2 subsystem LU name, and furthermore, that one thread (LUWID=15) is an IMS thread and the other thread (LUWID=16) is a batch thread. The LA administrator sees that the LA indoubt thread (LUWID=16) originates at the SEA Db2 subsystem and is a batch thread.
- The originator of a Db2 batch
thread is Db2.
To determine the commit or abort decision for the LUWID=16 indoubt
threads, the SEA Db2 recovery
log must be analyzed, if possible. Run the DSN1LOGP utility against
the SEA Db2 recovery
log, and look for the LUWID=16 entry. Three possibilities exist:
- No entry is found. That portion of the Db2 recovery log is not available.
- An entry is found but incomplete.
- An entry is found, and the status is committed or aborted.
- In the third case, the heuristic decision at SJ and LA for indoubt thread LUWID=16 is indicated by the status that is indicated in the SEA Db2 recovery log. In the other two cases, the recovery procedure that is used when cold starting Db2 is important. If recovery was to a previous point in time, the correct action is to abort. If recovery included repairing the SEA Db2 database, the SEA administrator might know what decision to make.
- The recovery logs at SJ and LA can help determine what activity took place. If the administrator determines that updates were performed at SJ, LA, or both (but not SEA), and if both SJ and LA make the same heuristic action, data inconsistency probably exists. If updates were also performed at SEA, the administrator can look at the SEA data to determine what action to take. In any case, both SJ and LA should make the same decision.
- For the indoubt thread with LUWID=15 (the IMS coordinator), several alternative paths to recovery are available. The SEA Db2 subsystem has been restarted. When it reconnects with IMS, message DSN3005 is issued for each thread that IMS is trying to resolve with Db2. The message indicates that Db2 has no knowledge of the thread that is identified by the IMS-assigned NID. The outcome for the thread, either commit or abort, is included in the message. Trace event IFCID=234 is also written to statistics class 4, which contains the same information.
- If only one such message exists, or if one such entry is in statistics
class 4,
the decision for indoubt thread LUWID=15 is known and can be communicated
to the administrator at SJ. If multiple such messages exist, or if
multiple such trace events exist, the administrator must match the IMS NID with
the network LUWID. Again, the administrator should use DSN1LOGP to
analyze the SEA Db2 recovery
log if possible. Now four possibilities exist:
- No entry is found. That portion of the Db2 recovery log was not available.
- An entry is found but is incomplete because of lost recovery log data.
- An entry is found, and the status is indoubt.
- An entry is found, and the status is committed or aborted.
- In the fourth case, the heuristic decision at SJ for the indoubt thread LUWID=15 is determined by the status that is indicated in the SEA Db2 recovery log. If an entry is found and its status is indoubt, DSN1LOGP also reports the IMS NID value. The NID is the unique identifier for the logical unit of work in IMS and CICS. Knowing the NID enables correlation to the DSN3005 message, or to the 234 trace event, either of which provides the correct decision.
- If an incomplete entry is found, the NID might have been reported by DSN1LOGP. If it was reported, use it as previously discussed.
- Determine if any of the following conditions exists:
- No NID is found.
- The SEA Db2 subsystem has not been started.
- Reconnecting to IMS has not occurred.
- For IMS,
the correlation-id is:
pst#.psbname - In CICS, the correlation-id consists
of four parts:
Byte 1 - Connection type - G=Group, P=Pool Byte 2 - Thread type - T=transaction, G=Group, C=Command Bytes 3-4 - Thread number Bytes 5—8 - Transaction-id