|
|
Under normal circumstances, the VOB database maintains a complete and uninterrupted record of all activity in the VOB's source pools. (Cleartext pools are caches and are expendable.) The VOB database reflects all additions (checkins, branch creation, element creation, and so on) and deletions (removal of versions, branches, and elements) in the pools. Each data container is referenced from the database.
NOTE: Derived objects (DOs) are a ClearCase feature that ClearCase LT does not use. ClearCase LT VOBs contain storage pools for these objects, but the pools are never used. checkvob examines these pools as part of its normal procedure, but they never contain any data on a ClearCase LT server.
In general, inconsistencies between the database and storage pools occur because of damage to the VOB database, to the storage pools, or to both. If the damage requires VOB restoration from backup, database or pool inconsistency occurs at restore time if the VOB has been backed up using the semi-live backup approach (database and pools were backed up at different times). That is why a vob_restore operation includes checkvob. In general, checkvob tries to make the pools match the database. If the pools are missing data, checkvob cannot re-create the data, and must adjust the database to compensate for the missing information.
VOB database damage. The database must be retrieved from snapshot or backup and is older than existing storage pools. It does not record recent pool changes (checkin, rmver, and so on).
VOB storage pool damage. The database is newer than the storage pools, which must be retrieved from backup. The database records development activity that is not reflected in the older pools.
Database and pool damage, with semi-live VOB backup procedures in use. The database snapshot and storage pool backups occur at different times (see vob_snapshot). The database retrieved from backup may be older or newer than storage pools retrieved from backup.
In addition, system crashes or network failures can prevent cleartool operations from completing cleanup tasks. This often results in unreferenced data containers. Any aborted operation that fails to clean up properly can have the same effect, as can OS bugs and disk problems.
Given a time lapse between current instances of VOB database and storage pools, any operation that modifies storage pools can create database or source pool inconsistencies. checkvob can identify-and in most cases, fix-these kinds of data container problems:
Misprotected data containers-containers with incorrect access control information
Missing data containers-VOB database references to nonexistent data containers
Unreferenced data containers-data containers for which there is no corresponding VOB database entry
Of these problems, missing containers is the most serious, as it may represent loss of data.
Problem | Found by checkvob |
|---|---|
All pool kinds (source, DO, cleartext) | |
Bad pool roots (pool names, identity info) | Yes (-pool) |
Source and DO pools only | |
Misprotected containers | Yes (-protections) |
Missing containers | Yes (-data) |
Unreferenced containers (debris) | Yes (-debris) |
Corrupted containers | No |
By default, checkvob prints detailed reports on one or more storage pools (-pool) or on specific file elements passed in the command line. With the -fix option, checkvob fixes as many problems as it can, adjusting data containers to match what is expected by the VOB database.
WARNING: Fixing problems detected with -data can update the VOB irreversibly. If version data is recorded in the database but missing from the storage pools, checkvob updates the VOB database by removing the version (rmver -data).
checkvob never updates the VOB database to reference newer pool data. If versions are stored in the pools but not recorded in the database, checkvob moves the unreferenced data containers to the pool's lost+found directory. That is, if pool storage is newer than the database, checkvob does not salvage the latest versions from the unreferenced pool containers and update the database. It uses the unreferenced containers to return the pool to the state expected by the database, and it moves the unreferenced containers (with the latest version data) to pool-dir\lost+found. These versions should be presumed lost. Contact Rational Technical Support for more information. (By contrast, barring unexpected events, any missing data that checkvob reports in this scenario can generally be attributed to rmver operations that were not recorded in the older database; and the fix processing that accepts this data loss is inconsequential.)
Although it does not add new version information to the VOB database, checkvob updates the VOB database in the following ways:
It removes references to versions that are confirmed missing by checkvob and accepted as missing by the user (interactive yes or -force -fix option).
After successfully reconstructing a missing container, in whole or part, checkvob adjusts the database to reference the newly reconstructed container, not the old, missing container.
If you use a customized type manager, it must include the get_cont_info method; if it does not, checkvob cannot repair data containers managed by this type manager. See the type_manager reference page for more information.
checkvob requires operational vob_server and db_server processes on the ClearCase LT server where it executes. As shown in Figure 19, the vob_server, which runs as the VOB owner on UNIX and as the NT Authority\system identity on Windows, is the only process that can create or delete data containers. Similarly, the db_server mediates all VOB database access.
Figure 19 Pool Access Through vob_server and VOB Database Access Through db_server
To understand the checkvob results, you must understand how ClearCase type managers operate on data containers. For example, changes to a data container (checkin, in particular) always result in a new, renamed container, regardless of the applicable type manager. See the type_manager reference page for more details.
To accept as lost any pool data added in the interval between the pool and database reference times, run checkvob -force -fix and supply an appropriate time interval (see Force-Fix Mode).
To accept the default data loss interval of one day, run checkvob -force -fix. Then, run checkvob without -force, and fix any remaining elements with missing containers manually (as prompted for).
The output from checkvob is captured in a log file directory created each time checkvob runs. The default location of the log directory is current-dir\checkvob.date-time, but you can move it with the -log option. A checkvob log directory's contents:
.\checkvob.11-Oct-99.18.30.45\
poolkind_source\transcript
poolkind_derived\transcript
poolkind_cleartext\transcript
summary
NOTE: The output log is a single summary file if you run checkvob on a single object.
Each transcript file contains a detailed report on check and fix processing for each existing pool kind. The summary file contains the header and tail sections from each transcript file.
Figure 20 shows a summary log file. This output was generated with a command like the following, on a VOB with one missing data container:
cleartool checkvob -pool -fix \\saturn\vobstore\src.vbs
Figure 20 checkvob Output Log: Summary File
Figure 21 shows a condensed transcript file for source pool check and fix processing. This output was generated with a command like the following, on a VOB with one missing data container:
cleartool checkvob -fix -force -pool -source \\saturn\vobstore\src.vbs
Figure 21 checkvob Output Log: Condensed Transcript File
The following sections describe checkvob actions.
When run in fix mode (-fix) against individual file-system objects (no -pool option), checkvob writes output like the following to standard output and also to log-dir\transcript. For example:
=================================================================
Processing element "\vob_src\open.c@@".
The element is now locked.
Checking status of 1 referenced containers in pool "s\sdft"...
...
fix processing output
Initial container status: 0 missing, 0 misprotected.
Final container status: 0 missing, 0 misprotected.
The element is now unlocked.
=================================================================
Check the element for -data (missing container) and -protection problems.
If fix mode (-fix), fix protection and missing container problems:
NOTE: Fix processing is blocked if the VOB, source pool, or element is locked.
Lock the element.
Fix missing protection problems.
Fix missing data container problems by scanning the pool for missing or misplaced containers and reconstructing containers as necessary.
Update the VOB database to reference the reconstructed containers.
For missing version data, update the VOB database to dereference lost versions with the equivalent of rmver -data.
Apply minor events to element's event history.
Move alternate (unreferenced) containers for this element to pool's lost+found directory.
Unlock the element.
When run with the -pool option, checkvob examines some or all of the VOB's source, DO, and cleartext pools.
For each kind of pool (source, DO, cleartext):
Check pool roots. Check pool names, locations, and pool identity information (each pool must have a pool_id file, which stores the pool's OID). These problems cannot be fixed with checkvob and must be directed to Rational Technical Support.
For source pools, check the installed type managers for checkvob support (get_cont_info method).
Scan VOB database for missing (referenced, but not found) or misprotected containers.
Generate a list of problem VOB objects.
If -fix is specified, process each problem VOB object as described in Individual File Element or DO Processing.
Scan for unreferenced data containers.
If -fix is specified, move each unreferenced container to the pool's lost+found directory.
If you use the -force -fix options, checkvob prevents you from unintentionally accepting data loss:
Do not allow removal of all non-\branch\0 versions. In force-fix mode, checkvob does not update the VOB database to reflect an element's missing data containers unless at least one version having a version number greater than 0 and its data remain. That is, you must run checkvob in -fix mode, without -force, and agree when prompted to accept a reconstructed element whose only remaining versions have version IDs like \main\br1\0 and \main\br1\br2\0.
Prompts to allow data loss. In force-fix mode, checkvob prints the following prompt:
Do you want to override the default and allow fixing of
elements involving missing version data? [no] n
If you answer yes, checkvob prompts you to specify a time interval for which data loss is allowable (or expected). For example, if you restored source pools from a backup with date-time 6-Oct-99.00:02:00, and your VOB database is current, you can reasonably expect to lose version data created since that time. In this case, you can direct checkvob to allow the loss of data created and recorded in the VOB database after that time:
cleartool checkvob -force -fix -data -pool -source c:\vobstore\proj1.vbs
...
Allow missing data created since: [date-time, <CR>] 6-Oct-99.00:02:00
If checkvob cannot find an expected container with data that was created before 6-Oct-99.00:02:00, it records this fact in the output log but does not update the VOB to dismiss the missing container; it does not accept the data loss. Run checkvob again without the -force option to process such elements individually, or adjust the allowed data loss time.
To silently accept (fix) all missing data containers without regard for creation time, use a very old date-time. The default time interval for allowed data loss is "since yesterday at 00:00:00." If you supply a date older than one week, checkvob forces you to confirm it.
checkvob log files do not capture the time-interval dialogue from -force -fix operations.
See the description of the date-time argument in the lshistory reference page for a list of acceptable values.
checkvob -setup -fix -pool is run by reformatvob when you upgrade a ClearCase LT server to a new ClearCase LT release. If this part of the reformatvob operation fails, you must run checkvob -setup -fix manually. This command must complete successfully to enable checkvob processing in the VOB's storage pools (which is a prerequisite to using the vob_snapshot utility successfully.)
checkvob -setup -fix -pool does the following:
Checks pool roots-pool names, locations, and pool identity information. (Each pool must have a pool_id file, which stores the pool's OID). For any pool, if a missing pool_id file is the only detected error, checkvob adds the file.
(Source pools only) Verifies that the installed type managers support use of checkvob. If a type manager does not support the get_cont_info method, checkvob cannot fix missing data containers managed by that type manager.
(Source pools only) Converts source pool data container pathnames to the correct format. See the type_manager reference page for a description of the source container name format.
VOB, pool, or element locks prevent setup processing. In this case, do the following:
(Pool or VOB locks only) Log on as the VOB owner or a privileged user.
Replace the VOB lock:
cleartool unlock vob:vob-pname
cleartool lock -nusers userid-that-will-run-checkvob vob:vob-pname
Replace pool locks:
cleartool lock -replace -nusers userid-that-will-run-checkvob locked-pools
Replace element locks:
cleartool lock -replace -nusers userid-that-will-run-checkvob locked-elements
Rerun checkvob -setup -fix manually.
The following sections describe how storage pool problems arise and how checkvob fixes them.
Notes on problem descriptions:
Unexpected events. Many problems described here are often side effects of various infrequent or uncommon events. Such events include network failure, system crash, failed or aborted cleanup operations, operating system bugs, disk failure, network reconfiguration events, and so on. In addition, there is a class of events that includes accidental or malicious delete, move, rename, or change-protection operations on the actual containers. These are all varieties of unexpected events.
Major and minor problems. The summary output from checkvob records the number of major and minor problems detected. Bad pool roots and missing data containers are considered major problems. All others are considered minor.
Reference times. A pool or VOB database's reference time is the point at which VOB activity was last recorded there. This can be the current date-time, the date-time when a still-operational VOB was locked, or the date-time when a snapshot or backup operation captured the pool or database. Expected output and fix processing from checkvob varies markedly depending on the relative reference times on the VOB database and storage pools being compared. There are three general cases:
The database is newer.
The pools are newer.
The database and storage pools are synchronized: they're both current, or they were retrieved as a unit from a complete backup of the VOB storage directory.
Fix processing. The following sections describe, under Fix Processing headings, how checkvob fixes the problems it finds. However, some descriptions include additional repair steps for the user.
Misnamed or misidentified pools.
rename pool in the interval between database and storage pool reference times.
Unless pool names and IDs match VOB database expectations exactly, abort processing for this pool kind (source or cleartext) and skip to the next pool kind.
Exception: If, for any pool, the only inconsistency is a missing pool_id file, create the pool_id file.
FAT file system: incorrect RO attribute.
NTFS file system: incorrect RO attribute or ACL.
User copied or restored pools or containers without preserving protection information.
Reset container's RO attribute and ACL as necessary. If the VOB owner's rights to pool contents are insufficient, checkvob reports, but cannot fix, container ACLs. If checkvob cannot repair protection problems, you must take the following steps:
Log on as a member of the ClearCase administrators group.
In Windows Explorer, click File > Properties. On the Security tab, take ownership of all files and directories in the VOB's storage directory tree.
NOTE: If you have identified only a small number of affected files, take ownership of these files only, to avoid a time-consuming checkvob operation in Step #6.
For all files and directories in the VOB storage directory, use the Security tab to grant full rights to the clearcase and vob-owner accounts.
Log out. Log on as VOB owner.
Take ownership of all files and directories in the VOB storage directory tree.
Run checkvob to fix protections on storage pools:
cleartool checkvob -force -fix -protection -pool c:\vobstore\myvob.vbs
checkvob works to reconcile the contents of VOB storage with the information in the VOB database. This reconciliation requires checkvob to categorize data containers based on their relationship to this information. There are two categories:
Missing data container. A container is considered missing if it does not appear under the exact name and location recorded in the VOB database. This definition implies missing version data. During fix-mode processing, checkvob attempts to fix missing containers by scanning all storage pools, looking for alternate containers from which to reconstruct containers for the missing data.
Debris. A container is considered debris if its pathname is not recorded in the VOB database. During -fix -debris processing, checkvob finds all debris in the source pools. It checks various aspects of an unreferenced container before moving it to the applicable pool's lost+found directory.
Whenever the VOB database and storage pools have different reference times (one is older than the other), checkvob is likely to find both missing and unreferenced containers. For example, consider a container whose location has changed as a result of a rename operation on a pool: it stores the data that the database is trying to reference, but at the wrong location in the storage pool. checkvob reports a missing container and an unreferenced container. Note that in fix mode, checkvob resolves these simple location problems.
The VOB database references a source pool data container that does not exist at the expected location. A container is considered missing if it does not appear under the exact name and location recorded in the VOB database.
(Database newer) Database records checkin not found in (older) pool.
(Pool newer) Pool stores updated, renamed container with new checkin data, but (older) database references pre-checkin container name, which no longer exists.
(Pool newer) Pool stores updated container to reflect versions removed with rmver or rmbranch, but the database references old container.
(Pool or database reference times differ) A chpool operation on a file element in the interval between pool and database reference times leaves the database pointing at wrong pool.
Unexpected events.
During fix mode processing, checkvob uses the following algorithm to fix missing containers by scanning all storage pools for alternate containers from which to reconstruct missing data containers.
Scan pools for alternate containers.
In a previous pass over the pools, checkvob found all referenced containers. It now scans for unreferenced containers for that element, looking for alternate containers that can be used to reconstruct a replacement for the missing container by rebinding the alternate container to take the place of the missing one. There are two types of rebind operations:
Optimized rebind: find alternates maintained by the right type manager.
Find best match: identical, superset, or subset.
identical-container with correct contents (user ran chpool during interval between pool and database reference times).
superset-container with superset of versions expected by database. This is common when the pool is newer than the database.
subset-container with subset of versions expected by database. This is common when the database is newer than the pool.
Clone and prune best match. Create a new container and copy the best alternate's contents. Delete extra version data from the new container.
Nonoptimized rebind: alternate containers with the right type manager not available (no "best match" found).
Construct container one version at a time from whatever sources are available: containers maintained by other type managers and cleartext pools.
If container reconstruction is incomplete, collect and report a list of missing versions.
If -force is not specified, prompt user to accept fix.
If -force is specified, safety features prevent some kinds of inadvertent data loss. See Force-Fix Mode for details.
If -force is in effect, or if user accepted fix prompt, update VOB database:
Adjust database to reference reconstructed containers.
With the equivalent of rmver -data, adjust the database to dereference lost version data.
Move all of the element's alternate containers to pool's lost+found directory.
NOTE: Because (unreferenced) alternate containers are moved to lost+found now, rather than during checkvob's subsequent debris processing pass, you have an opportunity to reclaim disk space from lost+found if the disk fills up during reconstruction. Reconstruction can consume substantial disk space.
Source pool includes a data container that is not referenced by the VOB database. Such containers are tentatively classified as debris, but must pass several tests before being moved to the applicable pool's lost+found directory.
(Database newer) Database references newer, post-checkin container name (which is missing), but pool stores older, unreferenced container.
(Database newer) Database records rmver or rmbranch events, but older pool stores container with branches or versions still in place.
(Pool newer) Pool has container with versions from one or more checkin operations not recorded in the older database.
(Pool or database reference times differ) A chpool operation on a file element in the interval between pool and database reference times leaves the database pointing at the wrong pool. The containers, being at an unexpected location, are unreferenced.
Restoring a pool from incremental backups.
Unexpected events.
checkvob usually moves an unreferenced container to the applicable pool's lost+found subdirectory (vob-storage-dir\s\sdft\lost+found, by default). It leaves in place unreferenced containers that fit into these categories:
May be needed. checkvob found, but did not fix, a missing container problem, and the pathname of the current unreferenced container suggests that it may be able to contribute to reconstruction of the missing container on a subsequent checkvob run. If you specify [-force] -fix -debris, without -data, checkvob performs -data check (not fix) processing to identify debris that may be needed.
Underage. The container is less than one hour old. checkvob skips underage containers to avoid removing a newly created container before the VOB database has been updated to reference it. Typically, the time between new container creation and database reference update is less than one second, but checkvob takes a conservative approach because of the critical nature of source containers.
Scheduled for deletion. The VOB is configured for deferred source container deletion (see vob_snapshot_setup and vob_server), and the container is already marked for deletion.
NOTE: When the pool is newer than the database and includes more recent versions not recorded in the (older) database, checkvob does not salvage versions from the unreferenced containers and update the database. It uses the unreferenced containers to return the pool to the state expected by the database, and it moves the unreferenced containers (with the latest version data) to pool-dir\lost+found. These versions should be presumed lost. Contact Rational Technical Support for more information.
File truncated and similar conditions
None. checkvob does not find or fix corrupted data containers. A container with the right pathname is considered healthy.
|
|
Feedback on the documentation in this site? We welcome any comments!
Copyright © 2001 by Rational Software Corporation. All rights reserved. |
