Administrator's Guide for Rational ClearCase MultiSite, Release 2002.05.00, All Platforms

10.5 Synchronization Import Problems

This section describes problems that can occur during the import phase of synchronization.

To list the imports at your current replica, use the following command:

cleartool lshistory replica:current-replica-name@vob-selector

For example, to list imports at the replica boston_hub in VOB family /vobs/dev:

cleartool lshistory replica:boston_hub@/vobs/dev 25-Jun.11:46 smg import sync from replica "sanfran_hub" to replica "boston_hub" "Imported synchronization information from replica "sanfran_hub". Row at import was: boston_hub=149 sanfran_hub=112" 10-Jun.12:36 smg import sync from replica "sanfran_hub" to replica "boston_hub" "Imported synchronization information from replica "sanfran_hub". Row at import was: boston_hub=136 sanfran_hub=111" 10-Jun.12:01 smg import sync from replica "sanfran_hub" to replica "boston_hub" "Imported synchronization information from replica "sanfran_hub". Row at import was: boston_hub=135 sanfran_hub=63"

Packets Accumulate in Incoming Storage Bay

A recoverable error occurs when an update packet is lost and is not applied at your site. These are the symptoms:

One or more replicas at your site are not being updated on their regular schedules.

An lspacket command shows unprocessed packets accumulating in the storage bay. These packets depend on the missing packet and cannot be processed.

Verify that a packet is missing and determine which operations are needed:

Enter a multitool syncreplica -import -receive command, which processes all incoming packets in the storage bay in the correct order. If syncreplica refuses to process any of them, a packet is missing.

Enter a syncreplica -import command that specifies the oldest packet in the storage bay:

multitool syncreplica -import packet-pathname Sync. packet packet-pathname was not applied to VOB ... - packet depends on changes not yet received Packet requires changes up to 872; VOB has only 756 from replica: sanfran_hub Packet requires changes up to 605; VOB has only 500 from replica: bangalore

In this example, one or more update packets are missing, containing operations 757-872 originally occurring at replica sanfran_hub and operations 501-605 from bangalore. In general, a packet can contain operations from several replicas; the syncreplica -import command fails if operations are missing from any replica.

Locate the missing packets. They may be on a magnetic tape that you forgot to process or in packet files that were not processed because your store-and-forward configuration (the shipping.conf file on UNIX; the MultiSite Control Panel on Windows) specifies the wrong storage bay. If you locate the missing packets:

Process the missing packets by naming them in a syncreplica -import command. (Multiple packet files are imported in the correct order, regardless of the order of the command-line arguments.)

Process all the update packets that have accumulated in the storage bay by entering a single syncreplica -import -receive command.

If you cannot locate the missing packets, go to Recovering from Lost Packets.

Packet is Not Applicable to Any Local VOB Replicas

Import can fail with the following message:

multitool: Error: Sync. packet pathname is not applicable to any local VOB replicas

This error can occur when a replica has been moved and the hostname property has not been updated with the chreplica command. To verify that the hostname property is wrong, enter the following command:

cleartool describe -fmt "%[replica_host]p\n" replica:importing-replica-name@VOB-tag

For example:

cleartool describe -fmt "%[replica_host]p\n" replica:newyork@/vobs/tests manhattan

If the hostname is incorrect, use the chreplica command to change it. At the master replica of the importing replica, enter this command:

multitool chreplica -c "comment" -host new-host replica:importing-replica-name@VOB-tag

For example:

multitool chreplica -c "change hostname" -host brooklyn replica:newyork@/vobs/tests Updated replica information for "newyork".

Send an update packet to the other replicas in the VOB family.

Read from Input Stream Fails

If a syncreplica -import command fails with a message like this one, the packet is corrupted:

multitool: Error: Read from input stream failed: No such file or directory

Delete the packet and ask the administrator at the sending site to re-create the packet and send it again (see Recovering from Lost Packets). Then import it.

Element Changes During Operation

If a syncreplica -import command fails with one of the following messages, restart the import:

Element changed during operation Element changed during checkin

The messages report that multitool was trying to import an operation for an element while another process (for example, a developer using cleartool) was operating on the same element.

If possible, restart the syncreplica -import from within a view. If it fails again, you see more information about what element it is failing on, and you can look through output from the lshistory command to try to find the conflict.

rmreplica Operation Cannot be Imported

Import of an rmreplica operation fails if the importing replica thinks that the removed replica still masters objects. The import fails with an error like the following:

multitool: Error: There are still objects mastered by this replica. multitool: Error: Unable to replay oplog entry 565632: error detected by ClearCase subsystem. 565632: 12 op= rmreplica 13 replica_oid= 48abc01d.123456a7.b890.06:00:08:c4:73:84 (boston_hub.mstr) 14 oplog_id= 23456 15 op_time= 08/07/00 12:35:46 create_time= 08/07/00 12:35:46 16 event comment= "Destroyed replica "boston_hub".

This situation can occur if two VOB replica hosts do not have the same patch level or if a ClearCase upgrade had problems.

You can use the lsmaster command to determine which objects are believed to be mastered by the removed replica. In this example, the administrator at importing replica sanfran_hub uses the lsmaster command to list the objects replica sanfran_hub believes to be mastered by replica boston_hub:

multitool lsmaster -view admin_view boston_hub@/vobs/dev master replica: boston_hub@/vobs/dev "label type" V2.0 master replica: boston_hub@/vobs/dev "label type" V1.1

In this example, the administrator at replica sanfran_hub uses the lsmaster command to contact all replicas in the VOB family and list the objects they believe to be mastered by replica boston_hub:

multitool lsmaster -view admin_view -inreplicas -all boston_hub@/vobs/dev In replica "bangalore" master replica: boston_hub@/vobs/dev "label type" V2.0 In replica "sanfran_hub" master replica: boston_hub@/vobs/dev "label type" V2.0 master replica: boston_hub@/vobs/dev "label type" V1.1

To resolve this problem, contact Rational Technical Support.

Replica Incarnation is Old

The following error can occur during packet import:

multitool: Error: Replica incarnation for "REPLICA_NAME" is old: old-timestamp should be new-timestamp

The replica incarnation is the last time the replica was restored (with the restorereplica command). The incarnation is set to 0 when the replica is created and remains 0 until a restoration occurs.

Each replica keeps a record of the incarnation of each other replica in the VOB family. During packet export, the incarnations of the target replicas are recorded in the packet. The syncreplica -import command at the importing replica checks the incarnation in the packet. If the incarnation in the packet is earlier than the importing replica's own record of its incarnation, the packet is not imported.

If the incarnations are different, the exporting replica does not have a record of the importing replica undergoing restoration. This situation may occur for the following reasons:

The update packet was created before the restoration information arrived at the exporting replica.

The restoration information was not sent to the exporting replica. For example, consider the following synchronization setup:

Replicas A and B synchronize every day, Replicas B and C synchronize once a week, and Replicas A and C synchronize once a month.

Replica A is restored from backup and the administrator runs restorereplica. Because Replica A's last synchronization was with Replica B, the administrator optimizes the process to require an update packet only from Replica B. After the packet is received from Replica B, the restoration is complete and Replica A resumes normal synchronization.

Because neither Replica A nor Replica B synchronized with Replica C during the restoration process, Replica C does not have any information about the restoration, and its record of Replica A's incarnation is not updated.

The next time Replica C sends an export packet to Replica A, the incarnation in the packet is earlier than Replica A's actual incarnation, and the import fails.

To determine which reason applies to your situation:

At the exporting replica, display the incarnation time for the importing replica.

cleartool dump replica:name-of-importing-replica@VOB-tag

In the output, look for a line beginning with incarnation=. This line displays the incarnation time. For example:

cleartool dump replica:boston_hub@/vobs/dev ... incarnation=01-Apr-99.22:40:54UTC ...

Compare this value to the value in the import error message.

If the values are the same after you adjust for time zone differences, the packet was created before the exporting replica received the restoration information. Delete the packet and follow the instructions in Recovering from Lost Packets.

If the values are different, contact Rational Technical Support.

Miscellaneous Problems

Processing of an incoming replica-creation or update packet may fail because of these conditions:

Disk partition is full.

Receiving replica is locked.

ClearCase or MultiSite licensing failure.

Multiple imports occur simultaneously.

Make sure that multiple syncreplica -import commands do not run in the same replica simultaneously. Check the timing of schedule tasks, and adjust them if necessary. (An invocation of the sync_receive script fails if another sync_receive process is running.)

In such cases, fix the problem and reenter the syncreplica -import command.

Recovering from Lost Packets

There are several circumstances in which a replica-creation or update packet is generated but is never applied at one or more of its destinations:

The packet is stored on media that are destroyed or are not readable at the destination host.

A packet file is lost when a hard disk fails.

The packet is intact, but cannot be applied because another packet has been lost. (See Packets Accumulate in Incoming Storage Bay.)

Lost Replica-Creation Packet

To recover a lost replica-creation packet:

At the replica where the mkreplica -export command was entered, remove the new replica with rmreplica.

Reenter the mkreplica command.

Lost Update Packet

The syncreplica -export command assumes successful delivery of the update packet it generates. For example, when replica boston_hub sends an update to replica sanfran_hub, the syncreplica command assumes that the operations originating at boston_hub are imported to the sanfran_hub replica. For simplicity, this example does not reflect the fact that the update packet can also contain operations that originated at other replicas in the VOB family.

However, if the packet is lost, this assumption is invalid, and boston_hub must reset its estimate of the state of replica sanfran_hub. After this correction is made, the next update packet sent from boston_hub to sanfran_hub contains the operations sanfran_hub needs.

To reset the epoch row, use one of the methods described here.

Method 1

At the sending site, use sync_export_list -update or chepoch -actual to set the epoch row to match the actual state of the receiving replica. These commands contact the receiving replica and retrieve its epoch row (the receiving replica's record of its own state). The sync_export_list -update command sends an update packet after it updates the epoch row in the sending replica. The sending and receiving sites must have an IP connection.

For example, use one of the following commands:

/usr/atria/config/scheduler/tasks/sync_export_list -update -replicas sanfran_hub@/vobs/dev

multitool chepoch -actual sanfran_hub@/vobs/dev Entry for bangalore changed from: 985 to 950 Entry for boston_hub changed from: 1400 to 1300 Entry for sanfran_hub changed from: 2562 to 2000

Method 2

At the receiving site, use the lsepoch command to display the replica's epoch number matrix:

multitool lsepoch sanfran_hub@/vobs/dev

`For VOB replica "/vobs/dev":`
`Oplog IDs for row "sanfran_hub" (@ goldengate):`
`oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=950`	`(bangalore)`
`oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=1300`	`(boston_hub)`
`oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=2000`	`(sanfran_hub)`

Use this output in a chepoch command at the sending site:

multitool chepoch sanfran_hub bangalore=950 boston_hub=1300 sanfran_hub=2000 Change oplog ID in row "sanfran_hub", column "bangalore" to 950 [no] yes Change oplog ID in row "sanfran_hub", column "boston_hub" to 1300 [no] yes Change oplog ID in row "sanfran_hub", column "sanfran_hub" to 2000 [no] yes Epoch row successfully set.

Method 3

At the sending site, use lshistory to determine the epoch numbers when the packet was generated:

cleartool lshistory -long replica:sanfran_hub 30-Jul.14:42:50 Susan Goechs (susan.user@minuteman) export sync from replica "boston_hub" to replica "sanfran_hub" "Exported synchronization information for replica "sanfran_hub". Row at export was: bangalore=950 boston_hub=1300 sanfran_hub=2000" 23-Jul.17:36:46 Susan Goechs (susan.user@minuteman) export sync from replica "boston_hub" to replica "sanfran_hub" "Exported synchronization information for replica "sanfran_hub". Row at export was: bangalore=900 boston_hub=800 sanfran_hub=1500" ...

At the sending site, use this output in a chepoch command:

Method 4

At the site that failed to apply the lost packet, use the lshistory command to determine the time of the last successful import of an update packet from the site that sent the lost packet.

GOLDENGATE> cleartool lshistory replica:sanfran_hub 01-Aug.07:08 jcole import sync from replica "boston_hub" to replica "sanfran_hub" "Imported synchronization information from replica "boston_hub". Row at import was: sanfran_hub=2000 boston_hub=1300 bangalore=950" ...

At the sending site, use this time in a recoverpacket command. recoverpacket looks through epoch rows to find an event that occurred prior to the specified time. When it finds a matching row, it resets the epoch row for the receiving site.

susan@minuteman% multitool recoverpacket -since 01-Aug.01:00 sanfran_hub

NOTE: With this method, you must adjust the time from the lshistory output for time zone differences and the amount of time elapsed between export and import.

If there are no saved epoch rows for the receiving replica that are as old as the time specified, you must use one of the chepoch procedures.

Inconsistent Changes to Replica

A recoverable error occurs if syncreplica -import detects that an incoming change is inconsistent with another change that has already been applied to the replica.

NOTE: In some cases, an inconsistency is resolved by syncreplica -import. For example, a replica receives an update that deletes an element, then receives an update from another replica that creates a new version on a branch of that element. The create-version operation in the second update is discarded because the element no longer exists.

Ownership Preservation

If two replicas are ownership-preserving, the OS-level permissions of their individual elements are synchronized. However, synchronizing the VOB group lists of the replicas is a manual task that you perform using cleartool protectvob -add_group.

syncreplica -import generates the following ownership-related error messages:

Can't create object with group that is not in the VOB's group list Can't change to a group that is not in the VOB's group list

These messages indicate that the sending replica added a group to its VOB group list, created a new element in that group or reassigned an existing element to that group, and sent the ownership change to a replica whose VOB group list has not been updated.

These messages may also indicate that the sending replica and/or receiving replica were created incorrectly as ownership-preserving.

If the replicas are intended to be ownership-preserving, follow these steps to recover from this kind of error:

(If necessary) Set a view, change to a directory within the replica, and reenter the syncreplica -import command. This produces diagnostics that include pathnames within VOB directories. For example:

elem_fstat= ino: 0; type: 2; mode: 0444; uid: 1037; gid: 20 . . name_p= "aux_util.c" nsdir_ver_oid= ed2549e2.97f411cd.b3c8.08:00:69:06:4d:f6 (/vobs/dev/src@@/main/ev2/CHECKEDOUT.572)

These lines indicate that the element's pathname in the sending replica is /vobs/dev/src/aux_util.c. Note also that its group-ID (GID) is 20.

Use the cleartool protectvob command to add the new group to your replica's VOB group list:

cleartool protectvob -add_group 20 /vobstg/dev.vbs

Reenter the syncreplica -import command.

NOTE: If the administrators at the sites of ownership-preserving replicas have not informed one another of changes in the shared user/group namespace, you may need to adjust the password and group databases before entering the protectvob command.

If one or both of the replicas should not be ownership-preserving, follow these steps:

Use the multitool chreplica command to change the receiving replica to non-ownership-preserving.

multitool chreplica -npreserve boston_hub@/vobs/dev Updated replica information for "boston_hub".

Import the packet.

multitool syncreplica -import -receive Applied sync. packet /usr/atria/shipping/ms_ship/incoming/sync_sanfran_hub_18-Jan-00.16.54.14_3 86_1 to VOB /net/minuteman/vobstg/dev.vbs

Change the status of the replicas.

If the sending replica should be non-ownership-preserving, change it to non-ownership-preserving.

If you want to retain ownership preservation in the receiving replica, change it back to ownership-preserving.

Export update packets from the sending and receiving replicas to all other replicas in the VOB family.

To avoid this problem in the future, use the procedure described in the section Replica Permission Strategy.

Object Mastership

An object mastered by one replica can depend on an object mastered by another replica. For example, an element and one of its subbranches are dependent objects, but these objects can be mastered by different replicas. As a result, certain kinds of inconsistent changes can be made at different replicas. The inconsistency is detected by syncreplica -import, causing it to fail with a recoverable error.

For example, if a type object is deleted in another replica, the replica at your site may refuse to import this change because a trigger type in the replica at your site depends on the deleted type object. During import, the following error message is displayed:

Can't delete attribute type type-name because of references to it in trigger type restriction lists

If the trigger at your site is useful only with the deleted type object, use cleartool rmtype trtype:type-name to delete the trigger type. Otherwise, replace the trigger type (cleartool mktrtype -replace) with a revised definition that does not depend on the deleted type object.

Reenter the syncreplica -import command.

Automatic Renaming of Type Objects and Replica Objects

The syncreplica -import command resolves naming conflicts among type objects or replica objects created at two or more replicas. For example, a branch type object named v1.0_bugfix is created at two different replicas. At some point, an invocation of syncreplica -import detects the conflict. (This may occur at one of the replicas that created the branch types, or at some other replica.)

syncreplica -import resolves the conflict by renaming the incoming object. In this example, branch type v1.0_bugfix is renamed to boston_hub:v1.0_bugfix, indicating that replica boston_hub is the master of the incoming type. syncreplica -import displays the following message:

multitool: Warning: To avoid name conflict, generated name "boston_hub:v1.0_bugfix" ...

Intervention is not required at this point unless branch types or replicas are renamed. (Renaming of branch types affects the validity of config specs, and renaming of replicas may affect synchronization scripts.) However, if you do not rename the objects, different replicas have different names for the same object. In this example, the boston_hub replica calls a branch type v1.0_bugfix, but at least one other replica calls the same type object boston_hub:v1.0_bugfix.

The various sites involved in such a conflict must coordinate the renaming of all the objects involved, to guarantee that all objects have the same name in all replicas. Here is a general procedure:

The administrators at the sites decide how to rename the objects.

At the master replica of each type object or replica object, the administrator renames the type object or replica object.

The Boston administrator renames the branch type that was created at the boston_hub replica:

cleartool rename brtype:v1.0_bugfix v1.0_bugfix-boston_hub

The San Francisco administrator renames the branch type that was created at the sanfran_hub replica:

cleartool rename brtype:v1.0_bugfix v1.0_bugfix-sanfran_hub

The Bangalore administrator renames the branch type that was created at the bangalore replica:

cleartool rename brtype:v1.0_bugfix v1.0_bugfix-bangalore

All sites exchange update packets to propagate the name changes.

NOTE: The name that caused the original conflict can be reused. One replica (and only one) can change the name to its original value:

cleartool rename brtype:boston_hub:v1.0_bugfix v1.0_bugfix

When this change is propagated to other replicas, it undoes any previous conflict-avoidance name changes, for example, by renaming boston_hub:v1.0_bugfix to v1.0_bugfix. (The propagation of this change must wait until after the other rename commands have been run in the other replicas and propagated throughout the VOB family, to make the name v1.0_bugfix available again.)