10.3 Synchronization Export Problems

This section describes problems that can occur during the export phase of synchronization.

To list the exports from your current replica to a sibling replica, use the following command:

cleartool lshistory replica:sibling-replica-name@vob-selector

For example, to list exports from your current replica in VOB family /vobs/dev to the replica sanfran_hub:

cleartool lshistory replica:sanfran_hub@/vobs/dev
12-Jul.16:13 root export sync from replica "boston_hub" to replica "sanfran_hub"
"Exported synchronization information for replica "sanfran_hub".
Row at export was: boston_hub=149 sanfran_hub=115"
29-Jun.16:19 smg change epoch of replica "sanfran_hub"
"Changed epoch row for replica
Old row was: boston_hub=152 sanfran_hub=115
New row is: boston_hub=149 sanfran_hub=115
epoch row set by special connected epoch tool."
29-Jun.10:12 smg export sync from replica "boston_hub" to replica "sanfran_hub"
"Exported synchronization information for replica "sanfran_hub".
Row at export was: boston_hub=149 sanfran_hub=115"
...

Cannot Find Oplog

syncreplica -export can fail with the following warning message:

Can not find oplog from replica replica-name with id oplog-ID
Gap in oplog entries may indicate missing oplog entries

(For more information on oplog entries, see VOB Operations and the Oplog and Scrubbing Parameters for VOB Replicas.)

This error occurs when the sending replica's epoch number matrix does not match its set of oplog entries. For example:

• Before sending an update from sydney to buenosaires, syncreplica checks the epoch number matrix for sydney. It determines that the last sydney operation sent to buenosaires was 3620.

• syncreplica finds that oplog scrubbing in the sydney database has removed some of the operations that follow 3620. The earliest sydney operation remaining in the oplog is 5755.

This discrepancy may be an expected condition. For example, when a VOB family changes its update topology, hosts that have not communicated with each other in the past start exchanging update packets. Synchronizing two replicas (syncreplica -export followed by syncreplica -import) updates epoch number matrix rows for the sending and receiving replicas, but it does not revise the row for any other replica. If two replicas rarely (or never) send updates to each other directly, the relevant rows in their epoch number matrices are out of date (possibly consisting of all zeros). This is not a problem, as long as the replicas receive operations indirectly, for example, through a hub replica.

In this case, you must inform sydney about the true state of buenosaires, information that it has not received through the standard synchronization-update mechanism. This information enables sydney to determine which oplog entries must be sent to buenosaires.

If the sites have an IP connection, use the procedure in Sites Have IP Connection. If the sites do not have an IP connection, use the procedure in Sites Do Not Have IP Connection.

Sites Have IP Connection

At sydney, use the chepoch -actual command to contact buenosaires, retrieve its actual state, and reset the epoch row for buenosaires.

multitool chepoch -actual replica:buenosaires@/vobs/tests

Sites Do Not Have IP Connection

Proceed as follows:

1. At buenosaires, use the lshistory command to determine when the last update packet was processed successfully.

cleartool lshistory replica:buenosaires@/vobs/tests

01-Sep.01:00 garyf import sync from replica "sydney" to replica "buenosaires"
"Imported synchronization information from replica "sydney".
Row at import was: buenosaires=8 sydney=3 boston_hub=0"
01-Aug.07:05 garyf import sync from replica "boston_hub" to replica "buenosaires"
"Imported synchronization information from replica "boston_hub".
Row at import was: buenosaires=2 boston_hub=0"
01-Jul.15:55 garyf create replica "buenosaires"

2. At sydney, use this time in a recoverpacket command to reset the epoch row for buenosaires. Assume that the sydney site is thirteen hours ahead of the buenosaires site.

multitool recoverpacket -since 01-Sep.14:00 buenosaires

If this command succeeds, proceed to Step #3.

If this command fails:

1. At buenosaires (destination site), run lsepoch to determine the actual state of buenosaires:

multitool lsepoch buenosaires@/vobs/tests

2. Send the lsepoch command output back to the sending site, where the administrator of sydney uses this data in a chepoch command to inform sydney about the actual state of buenosaires.

cd /vobs/dev
multitool chepoch buenosaires
Enter specifications for epochs to change in row "buenosaires" (one per line)
<output of lsepoch command>
.

3. At sydney, enter the original syncreplica -export command.

• If the command fails, buenosaires is in jeopardy. Have other replicas in the VOB family perform Step #1 through Step #3, taking the role of sydney to exchange update packets with buenosaires. The hope is that some other replica has not yet scrubbed its copies of the missing oplog entries. If no other replica has the missing oplog entries, you must create a new replica. See Replacing an Existing Replica.

• If the command succeeds and the packet is imported successfully at buenosaires, buenosaires is up to date.

NOTE: Have all sites review their oplog scrubbing procedures. You may have to change the oplog -keep settings in one or more vob_scrubber_params files. See Scrubbing Parameters for VOB Replicas.

Oplog Gap Detected During Creation of Update Packet

syncreplica -export can fail with the following warning message:

Gap in oplog detected for replica replica-name.
Wanted oplog id: oplog-ID. Got oplog id: oplog-ID.

This error message can indicate a serious error, involving an unrecoverable data loss. If the procedures described in Cannot Find Oplog do not work, contact Rational Technical Support.

Export Failure During Version Construction

An export operation can fail with a message like the following:

multitool: Error: Type manager "z_text_file_delta" failed construct_version operation.
multitool: Error: Could not get statistics of the version data file for this operation.
multitool: Error: Synchronization update terminated prematurely due to error -- aborting.

This situation can occur when an export operation tries to access an element that is being modified by a user. In this case, retry the export.

Packets Accumulate in Outgoing Storage Bay

Problems with packet delivery are recoverable errors. In many cases, the MultiSite automatic-retry capability recovers from errors.

A replica-creation or update packet submitted to the store-and-forward facility for transport to one or more other hosts is accompanied by a shipping order file. (A logical packet can include multiple physical packets, each with its own shipping order.) The shipping order typically has an expiration time, determined by one of the following:

• A date-time specified with the -pexpire option in the syncreplica or mkreplica command that generated the packet (or the mkorder command that submits an arbitrary file to the store-and-forward facility)

• On UNIX, the EXPIRATION value in the store-and-forward configuration file (shipping.conf) on the sending host

• On Windows, the Packet Expiration value specified in the MultiSite Control Panel on the sending host

Any number of delivery attempts may take place before the shipping order expires.

Replica Cannot Update Itself

You can receive the following message during export if you specify the sending replica as a destination:

A replica cannot update itself

If the sending replica is the only replica you specified, the syncreplica -export command fails. If you specified other replicas, this message is printed as a warning, and the syncreplica -export command continues its processing.