10.7 Restoring and Replacing Replicas

Occasionally, a VOB storage directory is lost. This can occur because of a hardware failure (for example, disk crash), a software failure (for example, OS-level file-system corruption), or a human error (for example, an rm -fr or del command). If an unreplicated VOB storage directory is lost, you can restore a recent copy from backup and resume development work. The changes made between the time of the backup and the time of the failure are not recoverable.

Similarly, if you lose the storage directory of a replicated VOB (that is, the storage for the replica used by developers at your site), you can restore a recent copy from backup. But matters are more complicated:

• Some of the work done between the time of the backup and the time of the failure may be recoverable. If some of the operations were sent to other replicas in update packets, these operations must be retrieved and imported.

• The restored copy of the replica is out of date. You must make this replica consistent with the other replicas in the VOB family before development can proceed at your site. Failure to reestablish consistency can lead to irreparable damage.

Because this procedure involves substantial effort, it is intended for situations where serious damage has occurred. (For example, the disk containing a replica is unusable.)

The method you use to restore the replica depends on how you back it up:

• If you lock your primary replica to back it up, you must restore it from the backup medium and perform the restorereplica procedure. See Restoring a Replica from Backup.

• If you never lock your primary replica and rely solely on a replica at your site as backup, you must replace the replica completely. See Replacing an Existing Replica.

Restoring a Replica from Backup

To restore a replica from backup:

1. Follow the procedure in the Administrator's Guide for Rational ClearCase to load the backup copy of the VOB storage directory.

2. As the VOB owner, root user (on UNIX) or a member of the ClearCase administrators group (on Windows), run the special MultiSite command to restore the replica:

multitool restorereplica -invob vob-selector

This places a special lock on the VOB object, which is in addition to the ClearCase lock created during the backup process. Between this point and the completion of Step #7, the syncreplica -import command adjusts the ClearCase lock temporarily to permit application of the update, then restores the full lock. During this time, only syncreplica -import can modify the replica.

3. Verify that all update packets have been processed at their destination replicas.

4. (Applicable only if the replica you're restoring was used to create one or more new replicas between the time of the backup and the time of the failure, and the other replicas in the family do not have information about the new replicas) The new replicas are unknown to your restored replica and all other replicas in the family, and lsreplica does not list them. If this is the case:

1. At each new replica, set the estimated states of the siblings to their actual states. (Use chepoch -actual or lsepoch/chepoch. See Recovering from Lost Packets.)

2. At each new replica, export update packets to all other replicas in the family except the restored replica.

3. Import the packets exported in Step #b.

5. At the restored replica, generate update packets for all other replicas, and send the packets to the sibling replicas.

You can send the packets using your standard synchronization method. To recover the replica more quickly, create the packets with syncreplica -export -fship.

Because your replica is in the special restoration state, each outgoing update packet includes a special request for a return acknowledgment. It also includes your replica's old epoch numbers, which are now its current epoch numbers, by virtue of the restoration backup in Step #1. Each destination replica uses these numbers to roll back its row for your replica.

6. Wait for each other replica in the VOB family to send an update packet to the restored replica. As in Step #5, you can accelerate the creation and delivery of the update packets.

Collectively, these update packets include all the operations that occurred between the time of the backup and the last update that your replica sent out before its storage was lost-even operations that originated at your replica. (The packets also include more recent operations that originated at other replicas.) In addition, each incoming packet includes the requested return acknowledgment from the sending host.

7. Process the incoming update packets with syncreplica -import. When your replica has received return acknowledgments from all other replicas in the VOB family, syncreplica -import reports that restoration of the replica is complete:

VOB has completed restoration: ...

8. (Applicable only if you had to perform Step #4) At one of the replicas that did not have information about the new replicas before the restoration procedure, export update packets to all of the new replicas and import the packets at the new replicas. (Do not perform this export from the restored replica.)

9. Unlock the VOB object in the restored replica.

cleartool unlock vob:pname-in-vob
Unlocked versioned object base "VOB-tag".

Development work in the replica can now resume.

Replacing an Existing Replica

If you must replace an existing replica, you can re-create it from one of the other replicas in the VOB family. For example, if you use Rational ClearCase MultiSite as your only backup mechanism and you must restore from a backup replica, you have to replace the working replica.

In this procedure, "backup replica" refers to the replica from which you restore the lost or deleted replica. If you have multiple replicas in the VOB family and you use more than one as a backup, use the replica that has most recently imported an update packet from the lost replica.

CAUTION: Do not use this procedure to fix import failures unless you have tried all other solutions, and Rational Technical Support advises you to follow these steps.

To replace a replica, use the following procedure (assume boston_hub on host minuteman is to be replaced, and sanfran_hub and bangalore are the other replicas in the VOB family):

1. For all views that use boston_hub, use the lsprivate command to list view-private and checked-out files. (To list views for which the VOB holds objects, use the cleartool describe vob: command.)

2. Check in all files (if possible) and save copies of view-private files out of the view. If you plan to save the views, use the procedure in Saving Views from the Replaced Replica at this point.

3. If boston_hub can export update packets:

1. On host minuteman, send update packets to sanfran_hub and bangalore from boston_hub:

multitool syncreplica -export -fship sanfran_hub bangalore




2. On the hosts where sanfran_hub and bangalore physically reside, import the packet from boston_hub:

multitool syncreplica -import -receive




4. Back up boston_hub's VOB storage to a storage medium.

5. At sanfran_hub, create a new replica, boston_hub2.

multitool mkreplica -export -workdir /tmp/create -nc -fship minuteman:boston_hub2

6. If you did not use the -fship option in Step #5, transport the replica-creation packet to the host minuteman.

7. Create the new replica. On host minuteman:

1. Unregister and remove the VOB-tag for boston_hub:

cleartool umount /vobs/dev cleartool unregister -vob /net/minuteman/vobstg/dev.vbs cleartool rmtag -vob /vobs/dev




2. Import the packet you created in Step #5 (include any special options you need):

multitool mkreplica -import -workdir /tmp/ms_wkdir -tag /vobs/dev2 \
-vob /net/minuteman/vobstg/dev2.vbs -nc -preserve -vrep boston_hub2 \
/var/adm/atria/shipping/ms_ship/incoming/sh_o_repl_sanfran_hub_18-May-99.15:50:00_ 1

3. Mount dev2:

cleartool mount /vobs/dev2




8. Make sure that boston_hub2 can synchronize successfully:

1. Set a view, change to a directory in /vobs/dev2, and generate a new label or attribute type. (Use a new view, not an old one that may have been used in boston_hub.)

2. Create and send update packets to sanfran_hub and bangalore:

multitool syncreplica -export -fship sanfran_hub bangalore




3. At sanfran_hub and bangalore, import the update packet:

multitool syncreplica -import -receive




4. At sanfran_hub and bangalore, list the new type created in Step #a:

cleartool lstype type-selector




9. Transfer mastership of all objects in boston_hub to boston_hub2.

1. Determine which replica masters boston_hub.

2. If boston_hub masters itself, run the following command at boston_hub2; if another replica masters boston_hub, run the following command at that replica:

multitool chmaster -all -obsolete_replica boston_hub boston_hub2




3. If boston_hub did not master itself, send an update packet from the master replica to boston_hub2 and import it.

10. Make sure that sanfran_hub, bangalore, and boston_hub2 can export and import update packets successfully.

11. At the site that masters boston_hub, remove the replica object for boston_hub:

multitool rmreplica boston_hub

12. Synchronize all replicas in the family.

13. Remove the physical storage for boston_hub with standard operating system commands.

14. Remove the views that were used in boston_hub. (If you want to keep these views, use the procedure in Saving Views from the Replaced Replica.)

Saving Views from the Replaced Replica

To save the views used in the replaced replica:

1. Move all view-private files into the view's lost+found directory (replica-uuid is boston_hub's UUID):

cleartool recoverview -vob replica-uuid -tag view-tag

2. List view-private files in each of the views:

cleartool lsprivate -tag view-tag -invob vob-selector

3. Use the uncheckout command to cancel all checkouts in the replica to be replaced; use the -keep option to save copies of the files.

4. Copy the .keep files to temporary directories outside the view. You can refer to these files when the new replica is available and you've checked out the elements again.

5. Use the rmdo command to remove all derived objects associated with the VOB to be replaced.

6. Remove all .cmake.state files.

7. Decide whether any valuable information is in any of the other view-private files associated with the VOB to be replaced.

After the replacement replica is back online, complete these additional steps:

1. Rebuild all derived objects.

2. Reconcile view-private files.

Because view-private files are associated with a particular replica, restoration from backup makes them inaccessible. To continue work on checkouts, you must determine all checkouts, capture the related files, and place them in the correct location.

You can do this by implementing a view backup procedure for files that cannot be re-created easily. For example, write a script that uses the lsprivate command to find all view-private objects (except for derived objects) and back them up to a backup tree. If the structure of this tree mirrors the VOB structure, it is easier to put the files back in their correct locations.

3. Run the recoverview command to free space associated with view-private files for the replica you removed.

An alternative method is based on recoverview. After letting recoverview move private files to the view's lost+found directory, the moved files are captured and placed into a location appropriate for the new replica. The main problem with this method is that the file names recoverview generates are leaf names; any directory structure is lost.

4. Redo changes to pool assignments.

Pool assignments are local to a replica, so re-creating the original replica may undo changes made to them. Major changes to pool structure must be duplicated manually at the backup replica.