CMVC FREQUENTLY ASKED QUESTIONS: HANDLING FILE, RELEASE AND LEVEL EXTRACT PROBLEMS Document Number TR 29.2xxx Lee Perlov, Angel Rivera and Jianren Yang TeamConnection/CMVC Development IBM Software Solutions Research Triangle Park, North Carolina Copyright (C) 1996,1997 IBM ii Solving CMVC Release -extract problems ABSTRACT This technical report provides answers to frequently asked questions made by CMVC users with respect to file, release and level extract requests and methods of diagnosing problems. ITIRC KEYWORDS o CMVC o File -extract o Release -extract o Level -extract o NFS mount ABSTRACT iii iv Solving CMVC Release -extract problems ABOUT THE AUTHORS LEE R. PERLOV Mr. Perlov is a Staff Software Engineer in the TeamConnection/CMVC development group. He started working for IBM in 1985 in Gaithersburg, Md, working in the Federal Systems Division on various projects for the United States intelligence community. He then moved to RTP to work on library development and support. Mr. Perlov received a B.S.Acc degree in Accounting from the Uni- versity of Florida in 1983. He also completed two years of grad- uate work in the Department of Computer Science at the University of Florida. ANGEL RIVERA Mr. Rivera is an Advisory Software Engineer and team lead for the CMVC development group; he is also a member of the TeamConnection development team. He joined IBM in 1989 and since then has worked in the development and support of library systems. Mr. Rivera has an M.S. in Electrical Engineering from The Univer- sity of Texas at Austin, and B.S. in Electronic Systems Engi- neering from the Instituto Tecnologico y de Estudios Superiores de Monterrey, Mexico. JIANREN YANG Mr. Yang is a Staff Software Engineer in the TeamConnection / CMVC department. He worked at several companies (such as Motorola) before joined IBM in 1995, and since then has worked in the development and support of both CMVC and TeamConnection pro- ducts. Mr Yang has an M.S. in Computer Science from University of Illinois at Urbana-Champaign (known as UIUC) and B.S. in Computer Science from Nanjing University, P. R. China. ABOUT THE AUTHORS v vi Solving CMVC Release -extract problems CONTENTS ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . III ITIRC KEYWORDS . . . . . . . . . . . . . . . . . . . . . iii ABOUT THE AUTHORS . . . . . . . . . . . . . . . . . . . . . . V Lee R. Perlov . . . . . . . . . . . . . . . . . . . . . . . v Angel Rivera . . . . . . . . . . . . . . . . . . . . . . . v Jianren Yang . . . . . . . . . . . . . . . . . . . . . . . v FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . IX 1.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . 1 1.1 About this and previous technical reports . . . . . . 1 1.2 About CMVC and TeamConnection . . . . . . . . . . . . 2 1.3 Acknowledgements . . . . . . . . . . . . . . . . . . . 2 2.0 OVERVIEW OF POSSIBLE PROBLEMS . . . . . . . . . . . . . 3 2.1 Limitation on resources . . . . . . . . . . . . . . . 3 2.2 Usage of CMVC in such way that the extracts impact on other users . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3 Other, less likely problems . . . . . . . . . . . . . 3 3.0 ANALYSIS AND DIAGNOSTICS OF THE PROBLEM . . . . . . . . 5 3.1 What is the error message that is shown? . . . . . . . 5 3.2 Is the problem repeatable or intermittent? . . . . . . 5 3.3 What version of the cmvcd daemon is running? . . . . . 5 3.4 What is the target machine (the one running the NFS server daemon)? . . . . . . . . . . . . . . . . . . . . . . 6 3.5 Is the user extracting the current or committed versions of the release? . . . . . . . . . . . . . . . . . 6 3.6 Is the syslog daemon running? . . . . . . . . . . . . 6 3.7 Are there errors reported by the NFS server daemon on the client? . . . . . . . . . . . . . . . . . . . . . . . . 7 3.8 Does it always fail on the same file? . . . . . . . . 7 3.9 Is there a router hop involved? . . . . . . . . . . . 7 3.10 Are the users starting/killing Release -extract commands? . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.11 Is it possible that mount command is not in the right location? . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.12 Is the date format correct in the family database? 8 3.13 Is CRLF conversion necessary? . . . . . . . . . . . . 9 4.0 ADJUSTING FOR INTERMITTENT FAILURES . . . . . . . . . 11 4.1 By default, CMVC uses NFS to mount a file system . . 11 4.2 Most common failures when using NFS with OS/2 and Windows clients . . . . . . . . . . . . . . . . . . . . . 11 4.3 Changing your system configuration . . . . . . . . . 12 4.3.1 Information about the NFSD and BIOD daemons . . . 12 4.3.2 CMVC_MOUNT_UNIX_PARMS and CMVC_MOUNT_OS2_PARMS . 12 Contents vii 4.4 What is the new function to slow down a release/level extract? . . . . . . . . . . . . . . . . . . . . . . . . 14 4.4.1 The CMVC_EXTRACT_RELIABILITY environment variable 14 4.4.2 The CMVC_EXTRACT_SLEEP environment variable . . . 14 4.4.3 The CMVC_EXTRACT_SLEEP_FS environment variable . 15 4.5 Extracting faster, without NFS . . . . . . . . . . . 15 4.5.1 Using argument "-node NOMOUNT" . . . . . . . . . 16 4.5.2 CMVC_EXTRACT_MOUNT_NAME . . . . . . . . . . . . . 16 4.6 The final option . . . . . . . . . . . . . . . . . . 17 5.0 WHAT TO DO WHEN EVERYTHING ELSE FAILS? . . . . . . . . 19 5.1 How to use the map file for committed levels to extract files? . . . . . . . . . . . . . . . . . . . . . 20 5.2 How to recreate lost map files? . . . . . . . . . . 22 6.0 DEBUGGING HINTS . . . . . . . . . . . . . . . . . . . 25 6.1 Capturing the mount command . . . . . . . . . . . . 25 6.2 Copying a family to recreate a problem . . . . . . . 25 7.0 FILE EXTRACT . . . . . . . . . . . . . . . . . . . . . 29 7.1 Specifying the full path name of a CMVC file using CMVC_TOP . . . . . . . . . . . . . . . . . . . . . . . . 29 7.1.1 Working from your current working directory . . . 30 7.1.2 Using the -relative flag . . . . . . . . . . . . 31 7.1.3 Using the CMVC_TOP variable . . . . . . . . . . . 32 7.2 File path name with forward or backward slashes . . 32 7.3 UNIX: File command adds (dot) files at extract or check-out . . . . . . . . . . . . . . . . . . . . . . . . 33 7.4 Intel: File command adds $ to the file suffix at extract or check-out . . . . . . . . . . . . . . . . . . 34 7.5 Restriction on file extracts into DOS or OS/2 FAT drives . . . . . . . . . . . . . . . . . . . . . . . . . 34 8.0 HANDLING PERFORMANCE PROBLEMS . . . . . . . . . . . . 35 8.1 Using vmstat to find performance problems . . . . . 35 8.1.1 vmstat usage . . . . . . . . . . . . . . . . . . 35 8.1.2 Interpreting the vmstat output . . . . . . . . . 36 8.1.3 Following leads provided by vmstat . . . . . . . 37 8.2 Using NetLS debugging tools that use CMVC licenses . 38 APPENDIX A. RELEASE/LEVEL -EXTRACT ERROR MESSAGES . . . . 39 A.1 Messages related to file system space and file permissions . . . . . . . . . . . . . . . . . . . . . . . 39 A.2 Messages related to the format of date and time . . 40 A.3 Messages related to NFS mount and file extraction . 41 A.4 Messages related to the version control system . . . 43 APPENDIX B. DETAILS ON USING NOMOUNT . . . . . . . . . . . 45 B.1 What is the new function to avoid using NFS on release/level extract? . . . . . . . . . . . . . . . . . 45 B.1.1 Details on the Mount option . . . . . . . . . . . 46 B.1.2 Details on the NOMOUNT option . . . . . . . . . . 46 B.1.3 Details on CMVC_NFS_DISABLE=YES option . . . . . 47 B.1.4 Implications of AFS/NIS/NIS+ . . . . . . . . . . 47 viii Solving CMVC Release -extract problems APPENDIX A. DESCRIPTION OF TOOLS AND HOW TO GET THEM . . . 49 A.1 Description of tools . . . . . . . . . . . . . . . . 49 A.1.1 Sample mount script . . . . . . . . . . . . . . . 49 A.1.2 XFILRELC (eXtract all the FILes for a RELease, Committed) . . . . . . . . . . . . . . . . . . . . . . . 49 A.1.3 XFILLEVU (eXtract all the FILes for a LEVel, Uncommitted) . . . . . . . . . . . . . . . . . . . . . . 50 A.1.4 XFILLEVC (eXtract all the FILes for a LEVel, Committed) . . . . . . . . . . . . . . . . . . . . . . . 50 A.1.5 XFILTRK (eXtract atest FILe change for all the files in a TRacK) . . . . . . . . . . . . . . . . . . . 51 A.1.6 XFILCOMP (eXtract latest FILe change for all files in a COMPonent) . . . . . . . . . . . . . . . . . . . . 51 A.2 Obtaining the Tools . . . . . . . . . . . . . . . . 52 A.2.1 IBM Intranet . . . . . . . . . . . . . . . . . . 52 A.2.2 Internet . . . . . . . . . . . . . . . . . . . . 52 APPENDIX B. BIBLIOGRAPHY . . . . . . . . . . . . . . . . . 55 B.1 How to get electronic copies of manuals and technical reports . . . . . . . . . . . . . . . . . . . . . . . . . 55 B.1.1 IBM Intranet . . . . . . . . . . . . . . . . . . 55 B.1.2 Internet . . . . . . . . . . . . . . . . . . . . 56 APPENDIX C. COPYRIGHTS, TRADEMARKS AND SERVICE MARKS . . . 59 FIGURES 1. Determine the version of CMVC . . . . . . . . . . . . . 5 2. Sample script to handle map files for a manual level extract . . . . . . . . . . . . . . . . . . . . . . . 21 3. Exporting a database using Oracle . . . . . . . . . . 26 4. Importing a database using Oracle . . . . . . . . . . 27 Contents ix x Solving CMVC Release -extract problems 1.0 INTRODUCTION This technical report provides answers to frequently asked questions made by CMVC users with respect to file, release and level extract requests and methods of diagnosing problems. This document is a guide for solving problems with the CMVC Release -extract and Level -extract commands. However, these procedures are sufficiently general to be applied to a variety of CMVC problems, such as performance problems. A procedure is included for copying a family so that debugging may be performed. Finally, a set of REXX samples is included to use a brute-force extract by using directly the "File -extract" command. This technical report is organized as follows: o Overview of possible problems o Analysis and diagnostics of the problem o Adjusting for intermittent failures o What to do when everything else fails? o Debugging hints o File extract o Handling performance problems o Release/Level -extract error messages o Details on using NOMOUNT 1.1 ABOUT THIS AND PREVIOUS TECHNICAL REPORTS This document, as well as other recently released technical reports related to CMVC, contains some information that was pre- viously available in other "unofficial" documents. We have endeavored to better organize the information and improve its readability so that you will find it more useful. This technical report supersedes the technical report TR 29.2130 DIAGNOSING AND SOLVING RELEASE -EXTRACT PROBLEMS WITH CMVC and it replaces the CMVC readme file /USR/LPP/CMVC/DOC/README.FIX.RELEASE.EXTRACT.PROBLEMS. Please use the new technical report when possible, as we have made many improvements and will be removing the older documents from our shipped media and ftp sites. For more information on normal situations on file, release and level extracts, see the companion technical report CMVC FRE- QUENTLY ASKED QUESTIONS: FILES, RELEASES, LEVELS AND TRACKS. Introduction 1 1.2 ABOUT CMVC AND TEAMCONNECTION TeamConnection is the successor product to CMVC. It has greatly expanded capabilities and is built upon an object model. While this and related technical reports discuss CMVC issues, we will periodically refer to new solutions provided in TeamConnection. Where a CMVC technical report covers a topic that might be of importance to TeamConnection users and if the TeamConnection doc- umentation does not already cover the topic, a TeamConnection technical report is or will be available. 1.3 ACKNOWLEDGEMENTS Many of the questions and answers that are compiled in this tech- nical report were obtained from the CMVC forum in the IBMPC con- ferencing disk and from the CMVC6000 forum in the IBMUNIX conferencing disk. We want to thank the main participants in these electronic forums for their support! 2 Solving CMVC Release -extract problems 2.0 OVERVIEW OF POSSIBLE PROBLEMS 2.1 LIMITATION ON RESOURCES o Disk constraints; for example, using up all space in /tmp or in the target file system. o Memory constraints; for example, extracting the current release which contains many files. o Unreliable network; for example, packet timeout or the network router is dropping packets. 2.2 USAGE OF CMVC IN SUCH WAY THAT THE EXTRACTS IMPACT ON OTHER USERS o Not enough daemons running; therefore, other users have to wait to perform CMVC actions. o Extracting -current instead of committed for large releases; ________ this option uses up too much memory and slows down the server. 2.3 OTHER, LESS LIKELY PROBLEMS o Using a Netware file server (older versions of CMVC cannot change the name of read-only files). o Writing to AFS/DFS space after the token has expired, or when the Access Control Lists are not properly set. o Mounting NFS file system as read-only. o Someone has changed the ownership/permissions on the CMVC daemon, cmvcd, which must be: rwsr-xr-x root.system cmvcd NOTE: On SunOS systems, the ownership is "root.root" Overview of possible problems 3 4 Solving CMVC Release -extract problems 3.0 ANALYSIS AND DIAGNOSTICS OF THE PROBLEM 3.1 WHAT IS THE ERROR MESSAGE THAT IS SHOWN? Is the message in the CMVC System Administration and Installation ___________________________________________ manual? We recommend using the version 2.3. If so, then it is possible that the explanation and the action explained in the manual could be helpful to solve the problem. 3.2 IS THE PROBLEM REPEATABLE OR INTERMITTENT? o A repeatable problem indicates a possible file problem (for example, a corrupted SCCS archive), or constraints of memory and/or disk. See 8.0, "Handling performance problems" on page 35 for details. o Intermittent problems indicates network problems. 3.3 WHAT VERSION OF THE CMVCD DAEMON IS RUNNING? To find out, issue the following command from the family account: +---------------------------------------------------------------+ | | | what &grave.whence cmvcd&grave. | grep cmvc | | | +---------------------------------------------------------------+ Figure 1. Determine the version of CMVC Use this command to learn the release and version of any CMVC executable program. This is most often used for the server daemon: cmvcd. We have improved the handling of release and level extract in our latest refreshes of the code. Analysis and diagnostics of the problem 5 3.4 WHAT IS THE TARGET MACHINE (THE ONE RUNNING THE NFS SERVER DAEMON)? o Is it a UNIX system or something slower (such as PC)? If using a PC, then you may need to use the new options explained in 4.4, "What is the new function to slow down a release/level extract?" on page 14. o How is the file system exported? Ensure that the CMVC server host is authorized to mount the exported file system. o Is the file system read-write? Ensure that the CMVC server host can read and write into the exported file system. o Is the CMVC server machine authorized to perform "root" oper- ations? The CMVC server host needs to have this type of access in order to write files using another user id. 3.5 IS THE USER EXTRACTING THE CURRENT OR COMMITTED VERSIONS OF THE RELEASE? o Extracting the current version will generate a query that can consume significant amounts of memory. o Extracting the committed version reads from the map file stored in the CMVC family account, and this operation is much faster. 3.6 IS THE SYSLOG DAEMON RUNNING? It is highly recommended that the syslog facility must be enabled and configured properly because the syslog is the place where the error messages from CMVC server daemon, the database and the operating system will be stored. To enable and configure the syslog facility see the technical report "CMVC frequently asked questions: handling OEM platforms and databases", section "Where are the error messages from the CMVC daemon and databases?". If the syslog facility is enabled, what are the error messages in the syslog file? If using AIX, is there any corresponding errors logged in "errpt"? 6 Solving CMVC Release -extract problems 3.7 ARE THERE ERRORS REPORTED BY THE NFS SERVER DAEMON ON THE CLIENT? In OS/2 the NFS daemon runs in its own window and any errors are displayed there. Other operating systems can run syslog for catching network prob- lems (for example, NFSD losing packets). 3.8 DOES IT ALWAYS FAIL ON THE SAME FILE? If the extract always fails on the same file, is it possible to extract the file using "File -extract"? If the "File -extract" fails, it may be due to: o Lack of space in /tmp or the file system of the family account. If the file is binary, the user needs roughly 3 times the size of the file in /tmp. If the file is text, the user needs roughly 2 times the size of the file in /tmp. o A corrupted SCCS archive. Check the archive with: admin -h vc/../../../../s... (as specified in error message). 3.9 IS THERE A ROUTER HOP INVOLVED? Routers can fill up and drop packets. Use a network load monitor or the error detection tools for the router to make sure that there is no data loss. A slow NFS daemon (for example, the current IBM OS/2, TCP/IP daemon) is slower than the ones on a typical UNIX system. Try extracting locally (that is, use "-node nomount") to see if the problem goes away. For more information on this feature, please see Appendix B, "Details on using NOMOUNT" on page 45. Analysis and diagnostics of the problem 7 3.10 ARE THE USERS STARTING/KILLING RELEASE -EXTRACT COMMANDS? In some cases, there are users who start a "Release -extract" and then due to the slow performance, the users assume that the process is hung and then kill the extract process in the CMVC client. When the user kills the extract on the client side, this action does NOT kill the corresponding process on the server, which will continue running as if nothing had happened and will terminate when it finishes the request. Therefore, it is possible that these users may repeat this process of starting and killing release extracts and eventually use up all the daemons on the server. As a result, no one else will be able to do any work with CMVC. Ask the system administrator to run the CMVC command "monitor" to see how many daemons are busy running extracts; it is possible to use "kill -1" or "kill -15" to recycle those daemons that doing an extract that the user has canceled the corresponding client process. 3.11 IS IT POSSIBLE THAT MOUNT COMMAND IS NOT IN THE RIGHT LOCATION? In some cases, like AIX 4, it is necessary to do extra configura- tion for the "mount" command in order to be found by CMVC. NOTE: In AIX4, the compatibility package must be installed in order to add a symbolic link that makes mount accessible as /etc/mount. 3.12 IS THE DATE FORMAT CORRECT IN THE FAMILY DATABASE? The newer release of CMVC, 2.3.1 or later, uses a new date string format (4-digit year, instead of 2-digit year format). The family database will have to be converted to use 4-digit year format, if you migrated from an old version CMVC to CMVC 2.3.1 or later. Otherwise, the extract will give different results, because the string comparisons for the date (such as, lastUpdate date, "97/05/01" and "1997/05/01") are different. NOTE: A tool, dbSetDate has been provided in CMVC 2.3.1 to convert the old CMVC family databases to the new date format that uses 4-digit year format for all dates. 8 Solving CMVC Release -extract problems 3.13 IS CRLF CONVERSION NECESSARY? Prior to 2.3.0.22, only the OS/2 and Windows CMVC client had support for the -crlf flag with Release/Level -extract, to request to perform the CRLF conversion. Starting with CMVC 2.3.0.22, the -crlf flag is now available from the UNIX CMVC line commands Release/Level. If the files extracted have some strange characters at the end of the line such as ^M and/or at the end of the file such as ^Z, this indicates that CRLF (Carriage Return, Line Feed) conversion is needed. In general, it is necessary to request CRLF conversion (by speci- fying -crlf at the Release/Level -extract command line) when you want to extract release/level from an OS2 platform to a UNIX platform. NOTES: 1. The command with -crlf option will be slower because the con- version has to be performed on each file in the release/level. 2. The corresponding change in the CMVC GUI for UNIX has NOT been made yet; that is, you cannot specify the -crlf flag from the CMVC GUI for UNIX. Analysis and diagnostics of the problem 9 10 Solving CMVC Release -extract problems 4.0 ADJUSTING FOR INTERMITTENT FAILURES We have learned from painful experience that even well managed networks can be intermittently unreliable due to NFS. As a result, we have provided several means of adapting the behavior of CMVC. This chapter describes the use of NFS, some common problems associated with this usage and some alternatives on how to overcome these problems. 4.1 BY DEFAULT, CMVC USES NFS TO MOUNT A FILE SYSTEM CMVC was originally designed to NFS mount a file system from the client's workstation (running the NFS daemon), to the family server. However, since CMVC was designed the dominant user's workstation has changed from a UNIX workstation to a PC running a Windows variant or OS/2. Although NFS daemons are available for Windows and OS/2 work- stations, these daemons have caused significant problems: 1. PC's run slower than their Unix counterparts. Even when the operating system is not slower, the multiprocessing capabili- ties (that is, running lots of processes through timesharing), the device driver performance and smaller device driver buffers impact the overall performance. 2. Extract actions to slower machines cause buffers to fill up and overflow in routers and bridges. 3. After packets are lost, the UDP protocol used by NFS has trouble recovering gracefully. 4.2 MOST COMMON FAILURES WHEN USING NFS WITH OS/2 AND WINDOWS CLIENTS NFS has a critical design weakness: it uses the UDP protocol instead of TCP. What that means is that there is no guarantee that a data packet will reach the intended destination intact. Since NFS simply does not work as well on PCs, this means that extracts to PCs fail more often. Here are the most common fail- ures: 1. A request to create a file on the PC is received by the PC and the file is opened, but the packet containing the file handle is lost by the router or by the PC. Adjusting for intermittent failures 11 After timing out, NFS resends the request to open the new file, however, on the second try the file already exists and the PC returns an error message that implies that the file system is full. 2. When the CR/LF conversion occurs, there is so much Input/Output associated with parsing and rewriting a file (one line at a time), that packets are lost and the file cannot be written. The problem is identified as a failure of CR/LF conversion. 4.3 CHANGING YOUR SYSTEM CONFIGURATION It is sometimes possible to overcome NFS problems with the changes described below. 4.3.1 Information about the NFSD and BIOD daemons __________________________________________________ NFSD is the daemon that handles the requests from the client related to the file system. The BIOD daemon runs on all NFS client systems. When a user on a client wants to read or write to a file on a server, the BIOD daemon sends the request to the server. Depends on the traffic load on your system, the number of daemons (NFSD and BIOD) can be changed. When the extract is very slow, you may want to increase the number of daemons (See your System Administrator for help). 4.3.2 CMVC_MOUNT_UNIX_PARMS and CMVC_MOUNT_OS2_PARMS _____________________________________________________ It is possible to override the default NFS mount parameters for Release/Level -extract. Many of these variable can influence the reliability and performance of NFS during an extract. The CMVC_MOUNT_UNIX_PARMS and CMVC_MOUNT_OS2_PARMS variables (introduced in 2.3.0.24) override the default NFS mount parame- ters for Release/Level -extract. By default, they are not set. Below are some statements that can be used to specify these new variables: 12 Solving CMVC Release -extract problems # Change mount options when UNIX clients mount file systems export CMVC_MOUNT_UNIX_PARMS="-o hard,intr,rw" # Change mount options when OS/2 or Windows clients mount file systems export CMVC_MOUNT_OS2_PARMS="-o hard,intr,rw,retry=4,timeo=14,\ retrans=6,rsize=4096,wsize=4096" Here are the default values used by the CMVC servers; please note that the actual value occupies one line, however, it might be possible that in the printout the value is shown in several lines. HP/UX . CMVC_MOUNT_UNIX_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6" CMVC_MOUNT_OS2_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6,rsize=4096,wsize=4096" SUNOS: CMVC_MOUNT_UNIX_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6" CMVC_MOUNT_OS2_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6,rsize =4096,wsize=4096" SOLARIS: . CMVC_MOUNT_UNIX_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6" CMVC_MOUNT_OS2_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6,rsize =4096,wsize=4096" AIX 4: . CMVC_MOUNT_UNIX_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6" CMVC_MOUNT_OS2_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6,rsize =4096,wsize=4096" AIX 3: . CMVC_MOUNT_UNIX_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6" CMVC_MOUNT_OS2_PARMS "-o hard,intr,rw,retry=4,timeo=14,retrans=6,rsize =4096,wsize=4096" Adjusting for intermittent failures 13 Finally, since the above strings are compiled into cmvcd, you can see them with a command like: strings cmvcd | grep "hard,intr" 4.4 WHAT IS THE NEW FUNCTION TO SLOW DOWN A RELEASE/LEVEL EXTRACT? QUESTION: What is the new function to slow down a release/level extract? ANSWER: The extract function for Release and Level has been expanded to include an option to slow down Release/Level extracts for less reliable networks and networks that include slow clients (such as OS/2 and Windows workstations with slow hard disks and/or slow processors). This is useful in cases where the NFS daemon on a workstation, bridges or routers drop packets. 4.4.1 The CMVC_EXTRACT_RELIABILITY environment variable ________________________________________________________ The CMVC_EXTRACT_RELIABILITY environment variable can be set to YES (yes) to indicate that you want the reliable way of doing Release/Level extracts. By setting CMVC_EXTRACT_RELIABILITY to NO (no), the behavior is changed to default behavior and the following CMVC environment variables will be ignored. 4.4.2 The CMVC_EXTRACT_SLEEP environment variable __________________________________________________ The CMVC_EXTRACT_SLEEP environment variable is ONLY available at the CMVC server when using CMVC_EXTRACT_RELIABILITY. It is set to a numeric value from 0 to 2000 (in units of milliseconds). It is important to notice that the delay specified by CMVC_EXTRACT_SLEEP is applied at multiple points during a Release/Level -extract. For example, setting CMVC_EXTRACT_SLEEP=100 will result in many 1/10th second delays for each file extracted. While this can add minutes to the total time required for a Release/Level -extract, it also reduces the likelihood of network problems. 14 Solving CMVC Release -extract problems 4.4.3 The CMVC_EXTRACT_SLEEP_FS environment variable _____________________________________________________ The CMVC_EXTRACT_SLEEP_FS environment variable is ONLY available at the CMVC server when using CMVC_EXTRACT_RELIABILITY. Often, network problems on Release/Level -extract only occurs on a select group of client workstations (such as workstations running the CMVC client on Windows). For this reason the CMVC_EXTRACT_SLEEP_FS environment variable can be set to the fol- lowing values: o DOS This is the default. It is used for file systems designated by drive:\path. o UNIX It is used for UNIX type file systems. o ALL It is used for all NFS mounted file systems. If CMVC_EXTRACT_SLEEP is set, but CMVC_EXTRACT_SLEEP_FS is not set, the default value for CMVC_EXTRACT_SLEEP_FS=DOS. The CMVC_EXTRACT_SLEEP does not apply in the following situ- ations: o CMVC_EXTRACT_SLEEP is ignored when "node=nomount". Because the NFS is not used, there is no need for the delay. o If CMVC_EXTRACT_SLEEP=0 the delay is disabled. 4.5 EXTRACTING FASTER, WITHOUT NFS With minor changes in your process, it is possible to avoid NFS entirely and greatly increase the speed and reliability of your extracts. Adjusting for intermittent failures 15 4.5.1 Using argument "-node NOMOUNT" _____________________________________ Release/Level -extract can specify the argument "-node NOMOUNT". This writes the code to the family server without using NFS, which dramatically reduces extract times. However, you will need to transfer the files from the server to your client! Here are some methods to accomplish it: 1. Using an alternative network file system. You can perform the equivalent to an NFS mount by using Novell Netware, OS/2 Warp Server, AFS, DCE/DFS, etc. prior to issuing a Release/Level with "-extract -node NOMOUNT". This should improve the reliability of the extract and deliver the files to your client in one step. 2. You can extract to a file system on the server, then use a utility such as "tar" or Info "zip" to bundle the files, send them from the server to the client in a single step, then unbundle the files on the client. Since CR/LF conversion then becomes an issue, CMVC provides the "fixcrlf" tool from the IBM software ftp site as well as internally to IBM through the CMVC Service/Development Home Page. NOTES: 1. The uid/gid will be set to the client id on the server, or the family ID if the client id is not defined. This is the same as the default behavior for the mount option. 2. If the users attempt to set the uid or gid, they will receive a warning and the specified values will be ignored. 3. User must specify the attribute "-node nomount" or "-node NOMOUNT" in order to get this optional behavior. 4. The "-root directory" must be accessible to the userid of the client, that is, the uid/gid on the server must allow access. 4.5.2 CMVC_EXTRACT_MOUNT_NAME ______________________________ You also have the option of moving your mount point out of /tmp. In cases where the file system is being used by many processes, distributing the file I/O across file systems can improve per- formance and reliability. Starting with CMVC 2.3.0.22, it is possible to specify the tempo- 16 Solving CMVC Release -extract problems rary directory to be used by CMVC Release or Level extracts, by using the new CMVC environment variable: export CMVC_EXTRACT_MOUNT_NAME=/tmp The value of the this variable is a path name for the desired temporary file directory. By default, if this variable is not specified or if the specified path name is not accessible, then /tmp is used. 4.6 THE FINAL OPTION In cases where NFS, or another network file system sharing mech- anism will not work, and it is not acceptable for users to log directly onto the family server, it is possible to do a query to identify the files in a release and run a script to extract the files one at a time. Since each extract indicates success or failure, it is possible to retry on each file and guarantee that the files are trans- ferred. See 5.0, "What to do when everything else fails?" on page 19 for more details. Adjusting for intermittent failures 17 18 Solving CMVC Release -extract problems 5.0 WHAT TO DO WHEN EVERYTHING ELSE FAILS? As a workaround to Release/Level -extract problems, it is pos- sible to extract files from a release/level without using NFS and instead, doing a single file extract for each file. The total extraction time will be a bit longer than a pure Release/Level extract, due to the accumulation of the small overhead in dealing with a repeated invocation of the CMVC File command. The following samples are provided using OS/2 REXX that perform the desired function: o XFILRELC (eXtract all the FILes for a RELease, Committed). See A.1.2, "XFILRELC (eXtract all the FILes for a RELease, Committed)" on page 49. This is equivalent to do the full extract of the latest com- mitted level for a release. o XFILLEVU (eXtract all the FILes for a LEVel, Uncommitted) See A.1.3, "XFILLEVU (eXtract all the FILes for a LEVel, Uncom- mitted)" on page 50. This is equivalent to do a delta extract of the latest uncom- mitted level for a release. The above 2 files are all that you need, but for completeness sake we include the following supplemental samples: o XFILLEVC (eXtract all the FILes for a LEVel, Committed) See A.1.4, "XFILLEVC (eXtract all the FILes for a LEVel, Com- mitted)" on page 50. This is equivalent to do a delta extract of a committed level for a release. o XFILTRK (eXtract the latest FILe change for all the files in a TRacK). See A.1.5, "XFILTRK (eXtract atest FILe change for all the files in a TRacK)" on page 51. o XFILCOMP (eXtract the latest FILe change for all the files in a COMPonent). See A.1.6, "XFILCOMP (eXtract latest FILe change for all files in a COMPonent)" on page 51. What to do when everything else fails? 19 5.1 HOW TO USE THE MAP FILE FOR COMMITTED LEVELS TO EXTRACT FILES? QUESTION: How to use the map file for committed levels to extract files? ANSWER: There is no single easy query that can give you all the files that can be extracted when doing a "Level -extract -full". For this reason, the CMVC server stores this information in map files. There is one for each committed level for each release. Their location is: ${FAMILY_HOME}/maps/${CMVC_RELEASE}/${LEVEL_NAME} The format of the maps file is: fileName versionId fileId changeType Level -extract -full is a complicated process, but here are the basics: 1. The list of files in a level are in $HOME/maps/releaseName/levelName. 2. Based on the content of the map file, do the database queries to identify which files to extract. 3. Perform File -extract -version commands to extract the file. NOTE: If the file has been destroyed, you cannot do anything with it. You can experiment with the following script on the levels in your family NOTE: Warning This is a simplistic example and does not cover all scenarios (such as destroyed files). o The ChangeView will return duplicate fileName/SID combina- tions if multiple tracks were specified upon checkin of a file so you may need to sort the output from Report -view ChangeView before using "awk". It is possible that the fol- lowing will work but may not be too eloquent: Report -view ChangeView -where "fileId=$FILEID and \ versionId=$VERSIONID and release Name='$RELEASE' " -raw | sed "s/\|/:/g" | sort -u -k4,5 -t: | awk -F":" ' {system("/usr/lpp/cmvc/bin/File -extract " $5 \ " -relative /yourAFSDir -release " $1 " -version " $4)} ' 20 Solving CMVC Release -extract problems #!/usr/bin/ksh # The following could be made as input parameters export RELEASE=r1 export LEVEL=l1 export mapsFile=$HOME/maps/$RELEASE/$LEVEL while read line do FILENAME=`echo $line | cut -d' ' -f1 ` # field 1 is fileName VERSIONID=`echo $line | cut -d' ' -f2 ` # field 2 is version id FILEID=`echo $line | cut -d' ' -f3 ` # field 3 is file id Report -view ChangeView -where "fileId=$FILEID and \ versionId=$VERSIONID and releaseName='$RELEASE' " -raw | awk -F"|" '{system("/usr/lpp/cmvc/bin/File -extract " $5 \ " -relative /yourAFSDir -release " $1 " -version " $4)} ' if µ $? -ne 0 º then echo "This may have failed because it is a destroyed file" echo "Error found in Report -view ChangeView or File -extract" terminate fi done < $mapsFile exit /* end of file */ Figure 2. Sample script to handle map files for a manual level extract o For destroyed files, the File -extract will not work since the File entry in the database is destroyed. Therefore the only thing you could do is issue a native SCCS command to get the file from the CMVC server. This is NOT recommended because it is not a good idea to allow all users to issue native SCCS commands from the CMVC server. TEAMCONNECTION NOTE: TeamConnection has integrated the map file data into the database, thus allowing users direct access to the version information for each level (Driver in TeamConnection). What to do when everything else fails? 21 5.2 HOW TO RECREATE LOST MAP FILES? QUESTION: Some map files were deleted by mistake from the CMVC family account. Can they be recreated manually? ANSWER: The map contains entries with the following format: pathName versionId fileId type There are easy ways to reproduce a lost map file. One possi- bility that was proposed by the CMVC support group in Austin is: 1. You could take the map file from the previously committed level. 2. Remove any lines with type='delete'. 3. Change the remaining types to "noDelta". 4. Get a list of the changes from the level with the lost map file. For each file, find the highest version (if multiple updates). 5. Now you need the version's id and the file's id. You prob- ably have to query the data base directly. 6. If the file is not new in the missing map file, then it has the same id, so the query could be: select sourceid from files where id=xxx 7. If the file is new, then the query could be: select id,sourceId from fileview where pathName='XXXXXXXX' This assumes that the file has not been renamed in the mean- time. 8. This gets the sourceId which is needed to find the version's id: select id from versions where sourceId=XXX and sid='X.Y' 9. If the file is new in this level, the type will be "create" or "link". If a file is deleted, the type is "delete". Otherwise, the type will be "delta". If the file was renamed, the type is "rename". This type takes precedence over other types (by the way, the field pathName in the map file is the new name). 22 Solving CMVC Release -extract problems Any files unchanged by the level retain the "noDelta" type. What to do when everything else fails? 23 24 Solving CMVC Release -extract problems 6.0 DEBUGGING HINTS 6.1 CAPTURING THE MOUNT COMMAND In some instances, it might be necessary to deal directly with the "mount" command to find out more details on the problem. It is possible to capture the mount command for debugging or other purposes: 1. Locate the directory where the mount command resides, and change to it. 2. Rename the mount command (to be restored later). 3. Create a shell script in place of "mount". You can take a look at the sample mentioned in the appendix A.1.1, "Sample mount script" on page 49. 4. Call the renamed "mount" from the new mount script. 5. The script must have the same permissions as the previous "mount" command 6. This is not a long term solution because the error handling is not always correctly handled in a script. 6.2 COPYING A FAMILY TO RECREATE A PROBLEM The CMVC Server Administration and Installation, Version 2 ______________________________________________________ Release 3 manual provides examples of how to export a database _________ (see Appendix C). This can be used when sending a copy of a family to CMVC development for debugging. Below is a sample pro- cedure, using Oracle, for getting and setting up customer data: 1. Get stuff from customer. a. Export the database to a file: Debugging hints 25 +---------------------------------------------------------------+ | | | $ORACLE_HOME/bin/exp password owner=family \ | | file=fileName.exp grants=y indexes=y | | | | $ORACLE_HOME Home directory for Oracle | | PASSWORD the password for Oracle (such as system/manager)| | FAMILY the name of the CMVC family | | FILENAME the name of the file with the exported database | | | +---------------------------------------------------------------+ Figure 3. Exporting a database using Oracle b. Compress the file (the result will be fileName.exp.Z): $ compress fileName.exp c. Copy the compressed database export into the family account. d. Copy the files from the family into a TAR file: $ tar -cdvf DeviceName . Use relative path from the home directory of the family account 2. Set up the family account Create a new user account on the machine with the correct level of the database used by the customer. NOTE: Get uid and gid from the customer. Log into the family account 3. Extract the customer files from the tar file: $ tar xvf DeviceName 4. Create the family and load the customer data. a. Make the family: $ mkdb FamilyName b. Delete the contents of the tables that might cause trouble: Log into the database from the database administrator's account (for example, ORACLE for Oracle): $ sqldba 26 Solving CMVC Release -extract problems c. Run the following database commands, where ${FAMILY} is replaced by the name of the CMVC family (used by MKDB). > connect to internal > delete from ${FAMILY}.users; > delete from ${FAMILY}.hosts; > delete from ${FAMILY}.config; > delete from ${FAMILY}.interest; > delete from ${FAMILY}.authority; > delete from ${FAMILY}.sequence; > delete from ${FAMILY}.components; > delete from ${FAMILY}.compmembers; > delete from ${FAMILY}.levels; > delete from ${FAMILY}.releases; > delete from ${FAMILY}.versions; > delete from ${FAMILY}.cfgrelproc; > delete from ${FAMILY}.cfgcomproc; > commit; > quit; d. Import customer database data: +---------------------------------------------------------------+ | | | $ $ORACLE_HOME/bin/imp password \ | | file=path-to-export-file fromuser=familyname \ | | touser=familyname buffer=204800 commit=y ignore=y | | | | $ORACLE_HOME Home directory for Oracle | | PASSWORD the password for Oracle (such as system/manager)| | FAMILY the name of the CMVC family | | FILENAME the name of the file with the imported database.| | | | .EXP is assumed. | | | +---------------------------------------------------------------+ Figure 4. Importing a database using Oracle 5. Replace the "mount" command, if necessary. 6. Start the daemons. 7. Edit the Host List in order to create an account where the client will work. You refer to the technical report "How to do migration tasks with CMVC", section "How to add a host list entry directly into the database?". 8. Recreate the problem. Debugging hints 27 28 Solving CMVC Release -extract problems 7.0 FILE EXTRACT 7.1 SPECIFYING THE FULL PATH NAME OF A CMVC FILE USING CMVC_TOP QUESTION: Specifying the full path name of a CMVC file using CMVC_TOP ANSWER: The material in this section is based on Chapter 18, Part, "TeamConnection Commands Reference". There are a number of ways to specify the full path of a CMVC file: o Specify the file name (which includes the path name and base name) and take the path from your current working directory. o Use the "-relative" flag to specify a path name that is outside your current working directory. In this case, the values for the "-relative" flag and the file name are concatenated. o If your workstation directory structure matches the way files are named, you can set the CMVC_TOP variable to the leading portion of your current working directory and specify only the base name in the actions of the "File" command. CMVC subtracts the contents of CMVC_TOP from the current working directory. This option might take some time to understand, but it can save you a great deal of typing when using line commands. The example that follows starts with the simplest case and moves to the more complex. Throughout the example, assume that the environment variables for CMVC_FAMILY, CMVC_RELEASE and CMVC_BECOME have been set. File extract 29 7.1.1 Working from your current working directory __________________________________________________ The main assumption for this example, is that the user is NOT using a directory/file structure that mimics the path names of the files stored in CMVC. 7.1.1.1 Initial checkout of txfile.c A user is working with a file named txfile.c. The path name and base name for this file is src/lib/txfile.c; the current working directory on the client workstation is /robot. There is no other file in this release named txfile.c. 1. The current working directory is: /robot 2. The user has not set CMVC_TOP yet. 3. The user enters the command: File -checkout txfile.c 4. The family server identifies that the base name of txfile.c is unique in the release: src/lib/txfile.c 5. The file is checked out in the workstation as: /robot/txfile.c 7.1.1.2 Initial checkout of msg.h Next, the user needs to check out src/lib/msg.h. Let's assume that other files named msg.h exist in the release, thus, the user must specify the entire path name. 1. The current working directory is: /robot 2. The user has not set CMVC_TOP yet. 3. The user enters the command: File -checkout src/lib/msg.h 4. The family server identifies that the base name of msg.h is not unique in the release. 5. The file is checked out in the workstation as: /robot/src/lib/msg.h 30 Solving CMVC Release -extract problems 7.1.1.3 Miscellaneous notes If you create a file without specifying "-relative" or setting the CMVC_TOP variable, then a file with the same name must exist in the current working directory. For example, you use the fol- lowing command: File -create src/lib/read.me In this case, read.me must exist in your current directory; CMVC does not search for the part in /src/lib. 7.1.2 Using the -relative flag _______________________________ 7.1.2.1 Check out again the file txfile.c from another directory The next day, the user's current working directory is /server. But the user wants to check out txfile.c to the /robot directory. The user specifies the "-relative" flag to do this: 1. The current working directory is: /server 2. The user has not set CMVC_TOP yet. 3. The user enters the command: File -checkout txfile.c -relative /robot 4. The family server identifies that the base name of txfile.c is unique in the release: src/lib/txfile.c 5. The file is checked out in the workstation as: /robot/src/lib/txfile.c 7.1.2.2 Check out again the file msg.h Similar scenario, but this time with the file msg.h which is not unique in the release, and thus, it is necessary to specify the complete path name. 1. The current working directory is: /server 2. The user has not set CMVC_TOP yet. 3. The user enters the command: File -checkout src/lib/msg.h -relative /robot File extract 31 4. The family server identifies that the base name of msg.h is not unique in the release. 5. The file is checked out in the workstation as: /robot/src/lib/msg.h 7.1.3 Using the CMVC_TOP variable __________________________________ A few days later, the user has checked out many files from the release into subdirectories of /robot, using a directory struc- ture that mimics the names of CMVC files and for convenience, the user sets the CMVC_TOP variable to /robot. Now when working with files, the user can specify just the base name of the files in his current working directory, as in this example: 1. The current working directory is: /robot/src/lib 2. The user has set CMVC_TOP to /robot 3. The user enters the command: File -checkout txfile.c 4. The family server identifies that the base name of txfile.c is unique in the release: src/lib/txfile.c 5. The file is checked out in the workstation as: /robot/src/lib/txfile.c The additional information added to the leading portion of the path is formed by subtracting the value of CMVC_TOP from the current working directory. In other words, the CMVC_TOP variable tells CMVC which portion of the current working directory path belongs to the client workstation. This command is completely equivalent to the following command if the CMVC_TOP variable is not set: File -checkout src/lib/server/tx/txfile.c -relative /robot 7.2 FILE PATH NAME WITH FORWARD OR BACKWARD SLASHES QUESTION: How CMVC handles the file path names with forward or backward slashes? ANSWER: 32 Solving CMVC Release -extract problems The CMVC client and server will take into account the usage of the slashes, both backward (for DOS, OS/2 and Windows) and forward (for UNIX). The file will be treated according to the target platform. That is, you can extract the file tmp\file.dos into a UNIX directory and the end result will be a file in tmp/file.dos. The reverse is also true, you can extract the file tmp/file.aix into a DOS directory and the end result will be a file in tmp\file.aix. You can use the GUI to specify the relative directory to store the code. In the UNIX GUI it is called "Set Directory", in the OS/2 and Windows GUI is called "Relative directory" from the first settings page. When using the line commands the CMVC_TOP environment variable can be used. For more details on this topic, please see page 51 in the CMVC Commands Reference, Chapter 10, Files. Actually, we improved the explanation in the TeamConnection Commands Refer- ence, page 98, Chapter 18, Parts (same concept). An example of the relative directory is shown next: if the user specifies the relative directory to C:\pages, and the file tmp\file.dos is extracted, then the file will be located in C:\pages\tmp\file.dos. Another point to consider is the difference between "path name" and "base name" when doing queries. The base name is the file name with NO directory information, such as file.dos, while the path name is the complete file name with the directory informa- tion, such as tmp\file.dos. 7.3 UNIX: FILE COMMAND ADDS (DOT) FILES AT EXTRACT OR CHECK-OUT QUESTION: UNIX: File command adds (dot) files at extract or check-out ANSWER: To preserve any potential changes that users could have made to the files on their workstation, the CMVC File command creates a copy of a file being extracted or checked out, and this copy is renamed with a prefix of . (dot) in order to keep it hidden during a normal "ls" query. In order to see these dot files, you have to issue the command "ls -a". File extract 33 7.4 INTEL: FILE COMMAND ADDS $ TO THE FILE SUFFIX AT EXTRACT OR CHECK-OUT QUESTION: File command adds $ to the file suffix at extract or check-out when using the client for OS/2 or for Windows. ANSWER: To preserve any potential changes that users could have made to the files on their workstation, the CMVC File command creates a copy of a file being extracted or checked out, and this copy is renamed with a suffix that begins with a $ (dollar). 7.5 RESTRICTION ON FILE EXTRACTS INTO DOS OR OS/2 FAT DRIVES QUESTION: Restriction on file extracts into DOS or OS/2 FAT drives ANSWER: Files from the CMVC server that do not conform to the 8.3 naming convention (file name longer than 8 characters or file extensions longer than 3 characters) cannot be extracted to a FAT drive. 34 Solving CMVC Release -extract problems 8.0 HANDLING PERFORMANCE PROBLEMS The first step to determine whether the problem is related to the performance of the UNIX system is to use the "vmstat" tool to monitor system behavior. Some possible performance problems are: o Running out of CPU cycles (becoming CPU bound) o Running out of real memory or paging space (thrashing) 8.1 USING VMSTAT TO FIND PERFORMANCE PROBLEMS The "vmstat" command can be used to determine both CPU utiliza- tion and real memory available in a UNIX system You generally run vmstat in one window, while running line commands in another window. You can also set this up as an "at" job, redirecting the output to a file. The goal is to make sure you catch the system behavior while the CMVC command and database operations are in progress. 8.1.1 vmstat usage ___________________ The vmstat command is invoked as follows: vmstat frequency µrepetitionº NOTES: 1. frequency is how often the system will be checked. You gen- erally use "1", for every second. 2. repetition is optional. It is the number of times to repeat the command. Without this parameter, vmstat will run until the process is killed. 3. Ignore the first line of output, that is an average of system performance from the first day the box was used. Handling performance problems 35 8.1.2 Interpreting the vmstat output _____________________________________ This is what to look for when running vmstat. You can find more detailed information in the AIX commands reference or the AIX 3.2 _______ Performance Monitoring and Tuning Guide: _______________________________________ COLUMN HEADING POSSIBLE INTERPRETATION AVM available memory. If this number gets too low, processes will be swapped out. This is one of the most expensive operations in UNIX. The relative cost of recovering memory is shown in paging activity and context switches. FRE free memory list. This is an arbitrary value, but when the number of pages becomes too low (again, an arbitrary value), the "initd" process runs garbage collection. This will slow your system for a short time. PO page out. These operations move pages containing dormant proc- esses to paging space. Depending on how efficiently paging space is set up, this is either slow or very slow. Ideally, you want paging space on several of your least active disks (so that the least active disk can receive the pages). Also, to improve performance it helps to have the disks on separate SCSI controller cards (since a single SCSI card can send data to only one device at a time). Ideally, this number is always 0. CS context switches. Each context switch is a change of the active process. It includes changing from user state to system state when an application requests access to a privileged device (for example, an application makes a SYSTEM() call to write the file). The more processes you run and the less efficiently these processes make use of system calls, the higher this number. Ideally, this number is at or below 100. R process run queue. This is the number of processes waiting for access to the CPU. Ideally this is never greater than 1 or 2. It is a gross indication that the system is not keeping up with the user's demands. 36 Solving CMVC Release -extract problems ID cpu idle. This the amount of time that the CPU has nothing to do. A healthy system has at least 20 percent idle. You would probably like about 50 percent idle so that you can handle the rough times. WA cpu wait on device. This number indicated the percentage of time the cpu is waiting for a device to complete a task. If this number is routinely high, you could have too much data on a single disk (so lots of writes have to be serial- ized) or have lots of delays on your network. SY cpu system utilization. A further indication of system problems is when "sy" is greater then "us" (cpu user process utilization). This is usually an indication that the system is not well configured or that the application was written ineffi- ciently (for example, using unbuffered write commands). 8.1.3 Following leads provided by vmstat _________________________________________ If you see any unusual activity with vmstat, also look at the error report log with the "errpt" command. Some of the system problems are very visible here, for example: o "paging space low, killing process ..." Paging space can be checked on AIX using "lsps -a". There should be roughly twice as much paging space as real memory (information available from SMIT), and it should be distrib- uted across disks. o "token ring error ..." Check netstat to see if there are network errors. Also, ask the system administrator to check for the physical integrity of the network components (such as adapter cards, cables, etc.). Handling performance problems 37 8.2 USING NETLS DEBUGGING TOOLS THAT USE CMVC LICENSES In some occasions, if the NetLS configuration is not correct, the CMVC clients may fail to execute properly because they cannot reach the NetLS server or they cannot obtain a valid license, or the performance of the execution of the CMVC command may seem to be too slow. To help with the diagnostics and the debugging of these NetLS configuration problems, the following executable files are pro- vided which can be used to contact the NetLS server and request a token for CMVC. Each program uses the same logic as a normal CMVC line command, but it will display a lot of information that will allow the user to better diagnose the problem with the NetLS configuration. Each program does not use the message catalog from CMVC, and thus, it is an independent program. Actually, each program does not require that CMVC should be installed at all. The testing NetLS programs in /usr/lpp/cmvc/bin (Client code) are: 1. test.netls.with.queues This program uses the NetLS processing done by CMVC in 2.2, 2.3.0.0 up to 2.3.0.20. 2. test.netls.no.queues This program uses the NetLS processing done by CMVC since 2.3.0.21, in which the previous waiting approach for NetLS was replaced by "softstop". That is, if all the licenses for CMVC are taken, the next user to request a license is not placed in a queue, instead, a token is given and an entry is added into the NetLS server log to indicate this situation. These tools provide time stamps that can help you to determine the overhead associated with getting a license from NetLS. If the overhead is large (more than 1 or 2 seconds), you may have do the procedure mentioned in the technical report "CMVC frequently asked questions: license management with NetlS/iFOR" to optimize the NetLS configuration. 38 Solving CMVC Release -extract problems APPENDIX A. RELEASE/LEVEL -EXTRACT ERROR MESSAGES The following are error messages commonly seen during Release/Level -extract, and suggestions on what to do. A.1 MESSAGES RELATED TO FILE SYSTEM SPACE AND FILE PERMISSIONS The following two error messages indicate that the file system runs out of space or does not have the proper file permissions. o msgErrno 0010-063 Error, %1$s, occurred when the CMVC server software attempted to process the %2$s() function. Usually this is a problem with fork, pipe, or other system command. For more information see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: CMVCD, NOTIFYD, CMVCARCHIVE/RESTORE, USER EXITS, SERVER ERRORS, DATABASE ISSUES , section "Error message: 0010-063". o msgExtractBadDir 0010-326 The CMVC server software cannot create directory %1$s relative to the destination extraction directory. Check that the parent directory permits the CMVC server soft- ware to write to it. Also check for the existence of the destination extraction directory and its access permissions. If the destination extraction directory exists, ensure that its permissions allow the CMVC server software to write to it. o msgErrno3 0010-350 The error (%1$s) occurred when the CMVC server software processed function %2$s() on the file with path name %3$s. Usually, open, read, or write, falures. Check that the path name, the file permissions, and the directory permissions are correct. o msgExtractChmodFailed 0010-380 The chmod request on file %1$s to a mode of %2$o failed during the extraction. Appendix A. Release/Level -extract error messages 39 The file is extracted, however, the permissions cannot be changed. Check if the server has permission to do that. You can change the permissions manually. o msgExtractCannotOverwrite 0010-499 File %1$s exists in the destination extraction directory with permissions that prevent the CMVC server software from overwriting it. Extract the files to an empty directory or change the access permissions of the file or destination directory to permit the CMVC server software to overwrite the file during the extraction. o msgClientNoFreeSpace 0010-677 The CMVC action was aborted while attempting to extract file %1$s . The file system does not have enough available space. o msgClientBadStatFile 0010-678 The CMVC action was aborted because the CMVC server could not access the extracted file %1$s. The file system does not have enough available space. o msgCrlfConvertFailed 0010-817 The CRLF conversion failed in %1$s. There are several reasons to cause this falure: - Cannot open temporary file (out of /tmp space). - Cannot open the target file for write. - The system calls fread() or fwrite() failed. A.2 MESSAGES RELATED TO THE FORMAT OF DATE AND TIME The following error messages are related to the format of the specified date. You can use -date yy/mm/dd to indicate that files modified in a release/level SINCE the specified date are to be extracted. The date must be in the format yy/mm/dd, where year, month, and day are two-digit character strings separated by a slash. o msgExtractDateBadFormat 40 Solving CMVC Release -extract problems 0010-328 The format for the date, %1$s, is not valid. The date must be in the format yy/mm/dd, where year, month, and day are two-digit character strings separated by a slash. For example, 91/04/28 o msgExtractDateBadMonth 0010-330 The value for month, %1$s, is not valid. The valid range for month is 01 to 12. o msgExtractDateBadDay 0010-331 The value for day, %1$s, is not valid. The valid range for day is 01 to 31. o msgExtractDateBadSubString 0010-461 The format of the date string, %1$s, is not valid. NOTE: text="CMVC 2.3.1 note". CMVC version 2.3.1 or newer support 4-digit year format, that is, the format of the date is, yyyy/mm/dd. A.3 MESSAGES RELATED TO NFS MOUNT AND FILE EXTRACTION The following messages are related to NFS mount when a real node is specified (that is, not using "NOMOUNT"); also, when files are extracted cross NFS mounted file systems. Contact the family administrator to resolve the NFS problems. o msgExtractVCErr 0010-025 An error occurred when the CMVC server software tried to extract files from the version control file repository. It could be an open file failure, a file permission problem, and an SCCS or PVCS problem (in this case, run some SCCS or PVCS command manually to see if it is working properly). o msgExtractMountErr 0010-325 The CMVC server software cannot mount directory %1$s from node %2$s. The CMVC server software requires mount and write access to the directory to perform a remote extraction. Contact your system administrator to see if the NFSD is running, and if necessary to manually mount/umount the direc- tory or to export the directory. o msgUmountBad Appendix A. Release/Level -extract error messages 41 0010-339 The CMVC server software cannot unmount the directory %1$s which was mounted from node %2$s. Contact your system administrator to see if the NFSD is running, and if necessary to manually mount/umount the direc- tory or to export the directory. o msgRemoveMountBad 0010-340 The CMVC server software cannot remove the directory %1$s. The extraction cleanup procedure cannot be completed. Contact your system administrator to see if the NFSD is running, and if necessary to manually mount/umount the direc- tory or to export the directory. o msgFileNetBadTransferToClient 0010-351 An error occurred when the CMVC server software tried to open a temporary file on the CMVC server prior to transferring a file from the CMVC server to the CMVC client. Check to see if the /tmp (temporary directory) is out of space. Contact the family administrator for further problem resolution. o msgExtractCannotUmount 0010-492 The unmount of directory %1$s failed. Contact your system administrator to see if the NFSD is running, and if necessary to manually mount/umount the direc- tory or to export the directory. o msgUndefinedVCType 0010-638 The version control type, %1$s, is not valid. Verify that the environment variable "CMVC_VCTYPE" is defined and exported properly. SCCS and PVCS are the only supported version control systems. o msgExtractBadHost 0010-673 Hostname %1$s is not a known host. Check for proper command format and that the hostname is known to the CMVC server's nameserver. 42 Solving CMVC Release -extract problems A.4 MESSAGES RELATED TO THE VERSION CONTROL SYSTEM The following messages are related to your version control system (SCCS or PVCS). For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZATION ISSUES. o msgVCGetFileCannotOpen 0010-510 File %1$s could not be retrieved. A temporary file could not be created because the CMVC server software does not have write access. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. o msgVCGetFailed 0010-511 An error occurred when the CMVC server software tried to get file %1$s. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. o msgVCGetFileDirExists 0010-595 A file and a subdirectory cannot have the same name. File %1$s cannot be extracted. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. o msgVCGetFileBadWrite 0010-604 File %1$s could not be retrieved. The CMVC server software can not create this file on the CMVC client, %2$s. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. o msgBinVersionInvalidSID 0011-106 An error occurred when the CMVC server software tried to get the file %s from the version control repository. An invalid version number was received. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. Appendix A. Release/Level -extract error messages 43 o msgBinVersionMergeFailed 0011-107 An error occurred when the CMVC server software tried to get the file from the version control repository. The server failed to merge the binary files, %s and %s. For more details, see the technical report: CMVC FREQUENTLY ASKED QUESTIONS: VERSION CONTROL AND DATABASE SYNCHRONIZA- TION ISSUES. o msgBinVersionMergeMallocFailed 0011-155 binmerge could not allocate temporary memory. It may be necessary to stop some processes or otherwise reduce memory consumption before retrying. o msgBinVersionMergeWriteFailed 0011-156 binmerge could not write to a temporary file. It may be necessary to free disk space before retrying. o msgBinVersionMergeOpenFailed 0011-157 binmerge could not open file %s. It may be necessary to free disk space or inodes. It is also possible that some versions of this file may be corrupted. o msgBinVersionMergeFileCorrupt 0011-158 binmerge detected a truncated data file: %s. It is likely that the current difference file is corrupt. 44 Solving CMVC Release -extract problems APPENDIX B. DETAILS ON USING NOMOUNT B.1 WHAT IS THE NEW FUNCTION TO AVOID USING NFS ON RELEASE/LEVEL EXTRACT? QUESTION: What is the new function to avoid using NFS on release/level extract? ANSWER: The extract function for Release and Level has been expanded to include an option to not perform an NFS mount to extract the files (that is, to extract to the CMVC server machine without performing an NFS mount). In this option the extraction will always be to a local directory in the CMVC server host. Because if NFS is not used, then the CMVC server will not be able to access remote directories for the extraction. The default function is to continue using NFS for mounting the target directories for the extraction. The specification for the new no-mount capability can be done as follows: o From the client, specify the keyword NOMOUNT as the destina- tion host (for example, -node nomount). For the -root attribute, specify a directory that is not available via NFS mount. This option allows users to selec- tively indicate which extractions will not use an NFS mount. o From the server, set the CMVC environment variable, CMVC_NFS_DISABLE=YES (the default is NO). This option allows family administrators to prevent the use of an NFS mount during extractions. The details of the new behavior are shown in the following sections. Appendix B. Details on using NOMOUNT 45 B.1.1 Details on the Mount option __________________________________ o If a user specifies a uid (the numeric user identification in /etc/passwd) and/or gid (the numeric group identification in /etc/passwd and /etc/group), all extracted files and directo- ries will receive that uid and/or gid. o The defaults for uid and gid are: - If the client userid (the user name you use when logging on, such as testclt) exists on the server, the server will look up the uid and gid on the server and use them. - If the id of the user exists on both the client and the server machines, but have different uids and gids on those machines, the extract will be done, but the userid on the client will not own the files. For example: testclt on client_wsn has uid 113 and gid 14 testclt on server_wsn has uid 110 and gid 10 testfam on server_wsn has uid 115 and gid 15 The extracted files will have uid 110 and gid 10 (from server) - If the client userid is NOT defined on the server, then the family uid and gid will be used (for example, uid 115, gid 15). o File permissions filtering: The setuid bit will always be filtered. For example: rwsrw-rw- becomes rwxrw-rw- rwSrw-rw- becomes rw-rw-rw- B.1.2 Details on the NOMOUNT option ____________________________________ o Uid/gid will be set to client id on the server, or family ID if the client id is not defined. This is the same as the default behavior for the mount option. o If the user attempts to set the uid or gid, they will receive a warning, and the specified values will be ignored. o User must specify a -node of "nomount" or "NOMOUNT" in order to get this optional behavior. 46 Solving CMVC Release -extract problems o The -root directory must be accessible to the userid of the client, that is, the uid/gid on the server must allow access. B.1.3 Details on CMVC_NFS_DISABLE=YES option _____________________________________________ o The family administrator must define and export the CMVC_NFS_DISABLE variable with the value either "YES" or "yes", in the family account. This must be done before the daemons are started. If this variable is not specified, then the default is NO. o Once CMVC_NFS_DISABLE is set to yes, all users will then receive the NOMOUNT behavior: - If the user specifies any valid -node other than "NOMOUNT" or "nomount", they will receive a warning that the -node is being ignored and told that their files will be extracted to the server. - If the "become id" (set with the -become attribute) does not exist in the server's /etc/passwd file, the family account id will be used for the extracted files (that is, the family account will own the files). If the "become id" does exist, that user id will own the files. - If the user account has a different uid on the server (such as, 123 on client-a and 125 on server-b), the uid on the server will be used (that is, 125). B.1.4 Implications of AFS/NIS/NIS+ ___________________________________ o Distributed file systems allow the user's space to be acces- sible to the server to extract directly to the server. Uid and gid must be common on all file systems. o Some configurations of NIS+ may cause problems with userid resolution. o Automounting schemes may force users to do an extra step so that their directories will be visible on the server (such as klog in AFS). o Automounting problems should be addressable through user exits. Appendix B. Details on using NOMOUNT 47 48 Solving CMVC Release -extract problems APPENDIX A. DESCRIPTION OF TOOLS AND HOW TO GET THEM A.1 DESCRIPTION OF TOOLS A.1.1 Sample mount script __________________________ This sample mount script is Used for debugging problems with behavior of mount during CMVC "Release -extract" command. NOTE: This is not to be used in a production environment. There are some return codes that are lost due to the interaction of this script with CMVC. This shell script is a front end for the real /etc/mount command that is invoked by other programs, such as the CMVC server. The objective is to provide a way to specify different parameters that can be tailored by the system administrator in case that the network is busy. See the header of the tool for the actual strings used by the CMVC server to build the command that will invoke the mount command. Notice that there are 2 versions of the parameters, one for OS/2 and another for UNIX, it is necessary to identify in this shell script which version is used. To use this sample script, do the following: 1. Rename the real mount command to something like "/etc/mountx" 2. Save this shell script as /etc/mount 3. Edit this script to use the proper positional parameter that is used by the CMVC server for the platform. See the section labeled "CHANGE HERE" inside the sample script. A.1.2 XFILRELC (eXtract all the FILes for a RELease, Committed) ________________________________________________________________ The XFILRELC tool, written in REXX, is equivalent to do the full extract of the latest committed level for a release. Syntax of tool: Appendix A. Description of tools and how to get them 49 USAGE: xFilRelC releaseName relativePathName ENVIRONMENT VARIABLES: CMVC_FAMILY µCMVC_BECOMEº DESCRIPTION: This sample command extracts all the files associated with a committed release and places them in the path specified by the relativePathName parameter. A.1.3 XFILLEVU (eXtract all the FILes for a LEVel, Uncommitted) ________________________________________________________________ The XFILLEVU tool, written in REXX, is equivalent to do a delta extract of the latest uncommitted level for a release. Syntax of tool: USAGE: xFilLevU releaseName levelName relativePathName ENVIRONMENT VARIABLES: CMVC_FAMILY µCMVC_BECOMEº DESCRIPTION: This sample command extracts the files associated with an uncommitted level in the specified release and places them in the path specified by relativePathName. A.1.4 XFILLEVC (eXtract all the FILes for a LEVel, Committed) ______________________________________________________________ The XFILLEVC tool, written in REXX, is equivalent to do a delta extract of a committed level for a release. Syntax of tool: USAGE: xFilLevC releaseName levelName relativePathName ENVIRONMENT VARIABLES: CMVC_FAMILY µCMVC_BECOMEº DESCRIPTION: This sample command extracts the files associated with a committed level in the specified release and places them in the path specified by relativePathName. 50 Solving CMVC Release -extract problems A.1.5 XFILTRK (eXtract atest FILe change for all the files in a ________________________________________________________________ TRacK) ______ The XFILTRK tool, written in REXX, extracts all the files associ- ated with a specific track. Syntax of tool: USAGE: xFilTrk releaseName trackName relativePathName ENVIRONMENT VARIABLES: CMVC_FAMILY µCMVC_BECOMEº DESCRIPTION: This sample command extracts all the files associated with a specific track and places the files in the path specified by relativePathName. A.1.6 XFILCOMP (eXtract latest FILe change for all files in a ______________________________________________________________ COMPonent) __________ The XFILCOMP tool, written in REXX, extracts all the files asso- ciated with a specific component. Syntax of tool: USAGE: xFilComp componentName relativePathName µcommittedº ENVIRONMENT VARIABLES: CMVC_FAMILY µCMVC_BECOMEº DESCRIPTION: This sample command extracts all the files associated with a specific component. The files are placed in a directory that represents the release name to which the version of the file is associated. This directory is created relative to the relativePathName parameter. Appendix A. Description of tools and how to get them 51 A.2 OBTAINING THE TOOLS The tools described in this technical report can be downloaded as follows: o From the IBM intranet (only for IBM employees). o From the Internet (open to everyone). A.2.1 IBM Intranet ___________________ A.2.1.1 Web Home Page You can access the CMVC Service/Development Home Page at: http://keithp.raleigh.ibm.com/&tilde.cmvcsupt From the index at the top of the page, select the section "Release extract tools". A.2.1.2 FTP You can download the code from our internal FTP site, by doing: 1. ftp keithp.raleigh.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd pub/cmvc/fixes/tr-tools 4. binary 5. get fileName 6. quit A.2.2 Internet _______________ A.2.2.1 Web Home Page Not available. 52 Solving CMVC Release -extract problems A.2.2.2 FTP You can download the code from our external FTP site, by doing: 1. ftp ftp.software.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd ps/products/cmvc/fixes/tr-tools 4. binary 5. get fileName 6. quit Appendix A. Description of tools and how to get them 53 54 Solving CMVC Release -extract problems APPENDIX B. BIBLIOGRAPHY For more information on how to use CMVC, you can consult the fol- lowing manuals and publications: SC09-1596-01 IBM CMVC Client Installation and Configuration SC09-1597-01 IBM CMVC User's Reference SC09-1631-02 IBM CMVC Server Administration and Installation SC09-1633-00 IBM CMVC Concepts SC09-1635-01 IBM CMVC Commands Reference The following Redbooks offer practical advice on CMVC: GG24-4178-00 Did you say CMVC? GG24-4345 CMVC: Customer's Perspective The following technical reports describe in detail useful hints on using CMVC: 29.2130 Diagnosing and solving Release -extract problems with CMVC 29.2169 How to use CMVC with National Language Support (NLS) and Double-Byte Character Sets (DBCS) 29.2180 How to do routine tasks in the OEM Platforms for CMVC 29.2183 Using CMVC for Products with Multiple National Language Versions (NLVs) 29.2232 How to do migration tasks with CMVC 29.2244 How to build and package the CMVC client for Windows 3.1 29.2245 How to build and package the CMVC client for OS/2 29.2253 Comparison between CMVC 2.3 and TeamConnection 2 29.2254 Migrating from CMVC 2.3 to TeamConnection 2 29.2268 CMVC frequently asked questions: license management with NetLS/iFOR 29.2269 How to build and package the CMVC Server/Client for UNIX B.1 HOW TO GET ELECTRONIC COPIES OF MANUALS AND TECHNICAL REPORTS Many of the manuals and technical reports mentioned in this docu- ment can be downloaded as follows: o From the IBM intranet (only for IBM employees). o From the Internet (open to everyone). B.1.1 IBM Intranet ___________________ Appendix B. Bibliography 55 B.1.1.1 Web Home Page You can access the CMVC Service/Development Home Page at: http://keithp.raleigh.ibm.com/&tilde.cmvcsupt From the index at the top of the page, select: o Technical Reports and related tools o Copies of ALL the archived versions for the forums CMVC and CMVC6000 o Documentation in PostScript files o Documentation in ASCII text files B.1.1.2 FTP You can download the code from our internal FTP site, by doing: 1. ftp keithp.raleigh.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd pub/cmvc/doc 4. binary 5. get fileName 6. quit B.1.2 Internet _______________ B.1.2.1 Web Home Page Not available. 56 Solving CMVC Release -extract problems B.1.2.2 FTP You can download the code from our external FTP site, by doing: 1. ftp ftp.software.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd ps/products/cmvc/doc 4. binary 5. get fileName 6. quit Appendix B. Bibliography 57 58 Solving CMVC Release -extract problems APPENDIX C. COPYRIGHTS, TRADEMARKS AND SERVICE MARKS The following terms used in this technical report, are trademarks or service marks of the indicated companies: +---------------------+-------------------------------------------+ | TRADEMARK, | COMPANY | | REGISTERED | | | TRADEMARK OR | | | SERVICE MARK | | +---------------------+-------------------------------------------+ | IBM | IBM Corporation | | AIX | | | OS/2 | | | PC-DOS | | | CMVC | | | TeamConnection | | | TCP/IP for DOS | | +---------------------+-------------------------------------------+ | PostScript | Adobe Systems Incorporated | +---------------------+-------------------------------------------+ | NetLS | Gradient Technologies, Inc. | | iForLS, iFor | | +---------------------+-------------------------------------------+ | HP-UX | Hewlett-Packard Company | +---------------------+-------------------------------------------+ | Info-ZIP | Info-ZIP Group | +---------------------+-------------------------------------------+ | Intel | Intel Corp. | +---------------------+-------------------------------------------+ | Microsoft | Microsoft Corporation | | Windows | | | Windows NT | | | Windows 95 | | | MS-DOS | | +---------------------+-------------------------------------------+ | Novell | Novell Inc. | | Netware | | +---------------------+-------------------------------------------+ | Oracle | Oracle Corp. | +---------------------+-------------------------------------------+ | OSF | Open Software Foundation, Inc. | | DFS | | | DCE | | +---------------------+-------------------------------------------+ | NFS | Sun Microsystems Inc. | | SunOS | | | Solaris | | +---------------------+-------------------------------------------+ | AFS | Transarc Corp. | +---------------------+-------------------------------------------+ Appendix C. Copyrights, Trademarks and Service marks 59 | UNIX | X/Open Co., Ltd. | +---------------------+-------------------------------------------+ END OF DOCUMENT 60 Solving CMVC Release -extract problems