CMVC FREQUENTLY ASKED QUESTIONS: CMVCD, NOTIFYD, CMVCARCHIVE/RESTORE, USER EXITS, SERVER ERRORS, DATABASE ISSUES Document Number TR 29.xxxx Angel Rivera, Lee Perlov, Clifford Meyers CMVC/TeamConnection Development IBM Software Solutions Research Triangle Park, North Carolina Copyright (C) 1997 IBM Corp. All rights reserved. ii CMVC FAQ: administration ABSTRACT This technical report provides answers to frequently asked questions made by CMVC users with respect to the following CMVC topics: o cmvcd daemons o notifyd daemon and mail notification o cmvcarchive and cmvcrestore o user exits o server errors o database issues ITIRC KEYWORDS o CMVC o cmvcd o notifyd o user exits o cmvcarchive o cmvcrestore ABSTRACT iii iv CMVC FAQ: administration ABOUT THE AUTHORS ANGEL RIVERA Mr. Rivera is an Advisory Software Engineer and team lead for the development of CMVC 2.3. He joined IBM in 1989 and since then has worked in the development and support of library systems. Mr. Rivera has an M.S. in Electrical Engineering from The Univer- sity of Texas at Austin, and B.S. in Electronic Systems Engi- neering from the Instituto Tecnologico y de Estudios Superiores de Monterrey, Mexico. LEE PERLOV Mr. Perlov is a Staff Software Engineer in the TeamConnection/CMVC development group. He started working for IBM in 1985 in Gaithersburg, Md, working in the Federal Systems Division on various projects for the United States intelligence community. He then moved to RTP to work on library development and support. Mr. Perlov received a B.S.Acc degree in Accounting from the Uni- versity of Florida in 1983. He also completed two years of grad- uate work in the Department of Computer Science at the University of Florida. CLIFFORD MEYERS Mr. Meyers is an advisory programmer and the technical team lead for ISSC Distributed Configuration Management Services. He joined IBM in 1985 as support for AIX/370 and AIX/ESA before accepting his current assignment in 1993. The ISSC organization in RTP provides consulting and support ser- vices for over 7,000 internal and external customers. Cliff was also involved in the development of the original Motif GUI for CMVC and all of the tools in the appendices of this docu- ment. +--- CLIFF, PLEASE UPDATE. -------------------------------------+ | | +---------------------------------------------------------------+ ABOUT THE AUTHORS v vi CMVC FAQ: administration CONTENTS ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . III ITIRC KEYWORDS . . . . . . . . . . . . . . . . . . . . . iii ABOUT THE AUTHORS . . . . . . . . . . . . . . . . . . . . . . V Angel Rivera . . . . . . . . . . . . . . . . . . . . . . . v Lee Perlov . . . . . . . . . . . . . . . . . . . . . . . . v Clifford Meyers . . . . . . . . . . . . . . . . . . . . . . v FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . IX 1.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Acknowledgements . . . . . . . . . . . . . . . . . . . 3 2.0 CMVCD DAEMONS . . . . . . . . . . . . . . . . . . . . . 5 2.1 What is the relationship between the CMVC client and the CMVC server? . . . . . . . . . . . . . . . . . . . . . 5 2.2 How does the CMVC server process the transactions? . . 6 2.3 What is the flow of a server transaction? . . . . . . 7 2.4 What are the interactions between the server and the database? . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.5 What is the purpose of the Sequence table? . . . . . . 8 2.6 How to debug problems with the CMVC daemons . . . . . 9 2.7 Server daemon fails to start when specifying multiple daemons . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.8 Do not change the file permissions nor the ownership of cmvc files . . . . . . . . . . . . . . . . . . . . . . 12 2.9 Stopping a CMVC family: use the stopCMVC script . . 12 2.10 How to deal with CMVC processes showing . 13 2.11 Conflict with stopCMVC with names of DB2 instance and CMVC family . . . . . . . . . . . . . . . . . . . . . . . 14 3.0 NOTIFYD AND MAIL NOTIFICATION . . . . . . . . . . . . 15 3.1 What is the interaction between the CMVC server and notifyd? . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2 Diagnosing and correcting typical mail problems . . 16 3.3 How can I manipulate the text of the message to new users ? . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.4 How to diagnose problems with notification (notifyd)? 17 3.5 Subdirectories queue and queue/messages in the CMVC family account . . . . . . . . . . . . . . . . . . . . . 18 4.0 CMVCARCHIVE AND CMVCRESTORE . . . . . . . . . . . . . 19 4.1 What important details should I know about cmvcarchive and cmvcrestore? . . . . . . . . . . . . . . . . . . . . 19 4.2 What additional details should I know about cmvcarchive? . . . . . . . . . . . . . . . . . . . . . . 20 4.3 What are the prerequisites for cmvcarchive? . . . . 21 4.4 What are the files created by cmvcarchive? . . . . . 22 Contents vii 4.5 What additional details should I know about cmvcrestore? . . . . . . . . . . . . . . . . . . . . . . 24 4.6 What are the prerequisites for the CMVC family to be restored? . . . . . . . . . . . . . . . . . . . . . . . . 24 4.7 If having problems with cmvcarchive, then migrate to CMVC 2.3.0.22 . . . . . . . . . . . . . . . . . . . . . . 25 4.8 CMVC is not reclaiming file system space when doing cmvcarchive . . . . . . . . . . . . . . . . . . . . . . . 25 4.9 cmvcarchive is taking a long time to run . . . . . . 26 4.10 Several problems with cmvcarchive . . . . . . . . . 26 4.11 How to find out a missing path id (due to cmvcarchive) . . . . . . . . . . . . . . . . . . . . . . 27 5.0 USER EXITS . . . . . . . . . . . . . . . . . . . . . . 31 5.1 What are user exits? . . . . . . . . . . . . . . . . 31 5.2 Showing a small user exit example . . . . . . . . . 32 5.3 What are the typical problems with user exits? . . . 33 5.4 Warning on using CMVC line commands inside CMVC user exits . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.5 New parameters for user exits in CMVC commands starting 2.3.0.22 . . . . . . . . . . . . . . . . . . . . 34 5.6 Apparent problem when passing more than 10 parameters to a user exit . . . . . . . . . . . . . . . . . . . . . 36 5.7 temp file for user exit 2 FileCheckIn not longer available . . . . . . . . . . . . . . . . . . . . . . . . 36 5.8 Are there any real examples of UserExits shipped with CMVC? . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.0 ERROR HANDLING AND AUDIT LOG . . . . . . . . . . . . . 39 6.1 What details should I know about error management? . 39 6.2 Where are the error messages from the CMVC daemon and databases? . . . . . . . . . . . . . . . . . . . . . . . 39 6.3 How to get more information from DB2 error numbers? 40 6.4 How to deal with the 0010-250 error message . . . . 41 6.5 CMVC error message: 0010-256 . . . . . . . . . . . . 42 6.6 Error message: 0010-063 with keyword bind() . . . . 43 6.7 CMVC error message 0010-063: Not owner, chown() function . . . . . . . . . . . . . . . . . . . . . . . . 45 6.8 Error message 0010-063 - Broken pipe, write () function . . . . . . . . . . . . . . . . . . . . . . . . 45 6.9 What details should I know about audit logging? . . 46 6.10 After running the sample cmvclog.cleanu, cmvcd does not log . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.0 DATABASE ISSUES . . . . . . . . . . . . . . . . . . . 47 7.1 Indexes that improve performance in levels and file changes . . . . . . . . . . . . . . . . . . . . . . . . . 47 7.2 Reorganization and rebinding a DB2 database . . . . 48 7.3 Running DB2 RUNSTATS is taking a very long time . . 49 7.4 Need to reboot the host if system hangs during db2stop 49 7.5 backupCMVC fails with DB2 error DB2106E . . . . . . 50 7.6 How to display the SQLCA reason code when there is a DB2 problem . . . . . . . . . . . . . . . . . . . . . . . 50 viii CMVC FAQ: administration 7.7 How to dump the transaction log in Sybase due to notify . . . . . . . . . . . . . . . . . . . . . . . . . 51 APPENDIX A. BIBLIOGRAPHY . . . . . . . . . . . . . . . . . 53 A.1 How to get electronic copies of manuals and technical reports . . . . . . . . . . . . . . . . . . . . . . . . . 53 A.1.1 IBM Intranet . . . . . . . . . . . . . . . . . . 53 A.1.2 Internet . . . . . . . . . . . . . . . . . . . . 54 APPENDIX B. COPYRIGHTS, TRADEMARKS AND SERVICE MARKS . . . 57 FIGURES 1. Updated parameter list for user exits . . . . . . . . 35 2. Sample Bourne shell using positional parameters . . . 37 3. Sample Korn shell using positional parameters . . . . 38 Contents ix x CMVC FAQ: administration 1 2 CMVC FAQ: administration 1.0 INTRODUCTION This technical report provides answers to frequently asked questions made by CMVC users with respect to the following CMVC topics: o cmvcd daemons o notifyd daemon and mail notification o cmvcarchive and cmvcrestore o user exits o server errors and audit log o database issues 1.1 ACKNOWLEDGEMENTS Many of the questions and answers that are compiled in this tech- nical report were obtained from the CMVC forum in the IBMPC con- ferencing disk, and from the CMVC6000 forum in the IBMUNIX conferencing disk. We want to thank the main participants in these electronic forums for their support! Introduction 3 4 CMVC FAQ: administration 2.0 CMVCD DAEMONS 2.1 WHAT IS THE RELATIONSHIP BETWEEN THE CMVC CLIENT AND THE CMVC SERVER? QUESTION: What is the relationship between the CMVC client and the CMVC server? ANSWER: o Because all CMVC client requests depend in some way on inter- action with the database, a connection between the CMVC server and the database must be made before work for a client can begin. o The CMVC server is implemented as one or more background server processes (cmvcd daemons). Each of these server proc- esses can service one client request at a time. The number of server processes that should be started depends on the number of clients that are likely to make requests at the same time. On the one hand, starting too many server processes wastes system resources and shared memory on the CMVC server. On the other hand, starting too few server processes may cause some client request rejections (a connection refused message). When you start the server processes, you indicate the number you want to start. The CMVC server will fork an additional child process for each of the parent server processes that have been requested. Therefore, if you query the number of cmvcd processes running for the CMVC family's account you will see double the number that you started. You can use the CMVC Activity Monitor to see the server proc- esses that are servicing requests. o When the parent server processes are initialized, they issue a series of TCP/IP socket calls to establish the server-side socket connection. Also during initialization, each child server process establishes a connection to the database. It then waits for client requests. cmvcd daemons 5 The operating system kernel decides which server process gets the client's socket connection. During server process initialization, each child server process connects to the database causing the creation of a shadow database process for each child server process. For example, type ps -u from the UNIX prompt on the CMVC server to see the server processes and the shadow data- base processes. The clients locate the server processes by determining the network address of the machine on which the server processes are running and the port number on which they are listening. 2.2 HOW DOES THE CMVC SERVER PROCESS THE TRANSACTIONS? QUESTION: How does the CMVC server process the transactions? ANSWER: The requests from a client are handled by the transaction processor in the server. A CMVC transaction is the accomplishment of one unit of work. One command line invocation (directly or indirectly via the GUI) result in one or more transactions. o A server process (cmvcd) will always complete its transaction regardless of whether the client connection is still intact. - The client connection can be broken by entering CTRL-C from the client's machine. - When the transaction completes, the packets of informa- tion will be discarded if the client-server connection is not available. - The client's request will still be effective since the server will have completed the requested unit of work. o A transaction can be terminated at the server side by 'recy- cling' a server process. - To recycle a server daemon process, enter 'kill -1 pid', where pid is the process id of the server daemon process. This will roll back the transaction within the database and prepare the server daemon process for the next request. 6 CMVC FAQ: administration - A transaction may complete before you have time to termi- nate or recycle it. Once a transaction is complete it can only be undone by using CMVC actions that would reverse the effect of the completed transaction. Some transactions can never be undone (for example, File Destroy). 2.3 WHAT IS THE FLOW OF A SERVER TRANSACTION? QUESTION: What is the flow of a server transaction? ANSWER: The following is the typical sequence of steps that the CMVC server takes when servicing a user request: 1. Make a monitor entry for the CMVC action. 2. Initialize the return arguments and message buffer. 3. Query the objects in the database. This includes validating the existence of the objects and related objects, obtaining object information and locking required objects. 4. Check the user's access authority to perform the action. 5. Test the validity of the action. For example, testing that the requested state change for an object is valid. 6. Make modifications, insertions, deletions to the database and in some cases the version control system. 7. Commit the changes to the database. 8. Notify the users of the action. 9. Make an audit entry for successful CMVC actions. 10. Return arguments and message buffer to the client. cmvcd daemons 7 2.4 WHAT ARE THE INTERACTIONS BETWEEN THE SERVER AND THE DATABASE? QUESTION: What are the interactions between the server and the database? ANSWER: o For the storage and fast retrieval of user data (other than files), the CMVC server makes use of an underlying relational SQL database. Each child server daemon process has a connection to the database and makes requests using embedded SQL. The CMVC server depends on a type of database transaction that begins implicitly when the first database request is made. The transactions is explicitly terminated by a commit or a rollback command. o The majority of the locking that occurs is done by the data- base by default. For example, when a database user updates a record, that record remains locked for that user until it is explicitly committed or rolled back. When default locking is not sufficient, table locks are used. For example, default locking is not sufficient when a data- base record is to be modified based on its current content since another user could modify it first and invalidate the action. Unique indexes on tables are used to ensure that duplicate records are not created. o Deadlock conditions are detected and resolved by the data- base. If the database detects a deadlock condition it will return an error to one of the deadlock participants. The CMVC server treats a deadlock error like any other database error. 2.5 WHAT IS THE PURPOSE OF THE SEQUENCE TABLE? QUESTION: What is the purpose of the Sequence table? ANSWER: Similar to most database applications, the information about CMVC objects is stored in a number of tables. Database identifiers 8 CMVC FAQ: administration are used to link the information in the various tables. These database identifiers are controlled by sequence numbers within the CMVC Server. The CMVC server has 3 sequence numbers which are stored in the Sequence table: o defect It controls the next number assigned to a defect or a feature. o source It controls the next number assigned to a file during cre- ation time. o general It controls the next number assigned to all other database objects. 2.6 HOW TO DEBUG PROBLEMS WITH THE CMVC DAEMONS QUESTION: How to debug problems with the CMVC daemons ANSWER: This section summarizes the steps that you can take to better diagnose the problems with the CMVC daemons, in case that they seem to suddenly die or behave in a unusual manner. First of all, you will need to enable the syslog facility, as explained in the technical report "CMVC frequently asked questions: handling OEM platforms and databases", section "Where are the error messages from the CMVC daemon and databases?" Due to the nature of the CMVC daemons, the messages generated by CMVC of by the database are logged in the system log or syslog. Thus, if a CMVC daemon suddenly dies, then it is very likely that there will be a message at the bottom of the syslog that will reveal more details on the cause of the problem. Then, it is proposed to perform the following sequence of events (probably off-shift to minimize the impact to other users): 1. Logon as the CMVC family administrator. 2. Stop the family daemons: cmvcd daemons 9 stopCMVC yourCMVCfamily 3. Run the "tail" command on the syslog to find out which is the timestamp of the last message in the syslog. This will serve as a point of reference in the syslog. 4. Start the cmvc daemons with just 2 daemons: cmvcd yourCMVCfamily 2 5. Do "ps -ef | grep cmvcd" to see that you have 4 entries: 2 are for the initial 2 parent daemons, and the other 2 are for the new 2 children daemons. 6. Start the "monitor 1" CMVC command (to refresh every second). This command will show you the status of each of the daemons, and if they are processing requests. 7. From another screen or terminal, issue several CMVC commands, such as "Report -view Users". 8. At the same time look at the monitor output, and it might be possible to see when a daemon is processing the request. In some cases the request is processed so fast that the monitor will not show it. However, one good indicator is the number of hits in the top portion of the monitor output, it should increment by 1 every time you issue the Report command. 9. Press any key when looking at the monitor output. This will stop it. 10. Do "ps -ef | grep cmvcd", Do you see still 4 entries? 11. Stop again the family daemons: stopCMVC yourCMVCfamily 12. Look at the tail of the syslog: are there any warning or error messages since the last time? If there are messages, then try to solve them. If there are database errors, then consult the documentation for your database. Some additional hints are shown below: o Whenever you explicitly specify 1 or more cmvcd daemons, they are brought up in background. However, if the above sequence fails to provide you with more clues, then you may try to bring up your family in the foreground: cmvcd familyName 0 10 CMVC FAQ: administration o Ensure that your family is properly defined in the /etc/hosts and /etc/services in server host. The family should be uniquely identified and the port number used by the family should not be used by another application. o It might be possible that another application is hard-coding the same port number used by your family. If you are using a port number that would be easier to specify such as a round number "3000", try to change it to another number such as "3001" or "3124". o If the cmvcd fails only when specifying multiple daemons, then you may take a look at 2.7, "Server daemon fails to start when specifying multiple daemons." 2.7 SERVER DAEMON FAILS TO START WHEN SPECIFYING MULTIPLE DAEMONS QUESTION: The CMVC server daemon fails to start when specifying multiple daemons but it works fine when specifying only one daemon. ANSWER: For more information on debugging problems with the CMVC family server daemon see 2.6, "How to debug problems with the CMVC daemons" on page 9. Because only "multiple daemons" is not working, you may look at the following o Check for processes that are getting killed by the system. Use the "errpt" command to see if there are processes getting killed. Hint: the output from "errpt -a" provides much more detail. o If you do not have your system set up to allow enough proc- esses for each user, use "smit" to change the system config- uration to allow for more concurrent processes (at least 30). o Ensure that you have enough licenses for the database. You need at least 1 license for each daemon that is run. The CMVC Server Administration manual recommends that you should have an extra for miscellaneous commands/queries that are run by the family administrator. o Ensure that you have enough space in /var, /tmp, and the file system that contains the family account (such as /home). cmvcd daemons 11 You want 3 times the largest possible file to be checked-in for /tmp, at least 2-3 Mb for mail, syslog, and other files that grow in /var, and tons of space for the family account. 2.8 DO NOT CHANGE THE FILE PERMISSIONS NOR THE OWNERSHIP OF CMVC FILES QUESTION: A customer modified the ownership of the executable of cmvcd in /usr/lpp/cmvc/bin from root:system to bin:bin (probably by mistake) and this action caused a LOT of problems that were dif- ficult to diagnose, such as messages in the syslog about not able to open the audit/log, error message 0010-063 with chown() when creating or checking a file, and problems with fchown() during a release extract. ANSWER: NOTE: Please, do not change the file permissions or the owner- ship of any of the CMVC executables. For your information, the cmvcd must have: -rwsr-xr-x 1 root system /usr/lpp/cmvc/bin/cmvcd* While the other executables must be: -rwxr-xr-x 1 bin bin /usr/lpp/cmvc/bin/Report* 2.9 STOPPING A CMVC FAMILY: USE THE STOPCMVC SCRIPT QUESTION: Stopping a CMVC family: use the stopCMVC script ANSWER: The recommended process for stopping a CMVC family is to use the stopCMVC script, which takes care of using "kill -15" against each family cmvcd daemon and the notifyd daemon, and clears up the shared memory, semaphores and process queues used by the family. NOTE: Please do not use "kill -9" to stop a cmvcd daemon because it does not give the daemon the opportunity to do a graceful shutdown, as it is done with "kill -15" In case that stopCMVC does not kill all the cmvcd processes, one likely cause is that a Release/Level -extract action is in 12 CMVC FAQ: administration progress or is hanging. This action and other similar actions need to do something as a user other than the family and they spawn a new process owned by root and frequently change to another user ID. Therefore, the primary reasons that many fami- lies are seeing cmvcd processes hanging around and owned by root is that they are: o Started by an OS/2 client. o Then, the OS/2 client process is killed by the user, the box hangs, etc. o The OS/2 NFS daemon either hangs or is killed as a result. o The server daemon hangs because it cannot perform a umount. o Because the killing of a client process does NOT kill the corresponding server process (CMVC does not have a mechanism for communicating a kill message), there is no way for the client user to get rid of this daemon. One suggestion is to run a series of "cron" jobs in this order: 1. As each family: run stopCMVC against each family 2. As root: grep for any cmvcd processes still remaining, then kill them using first "kill -15" and if that does not work then use "kill -9". 3. Do your normal backup for the CMVC family account and the corresponding database. You could use the backupCMVC script as a starting base. 4. Restart the cmvcd and notifyd daemons. 2.10 HOW TO DEAL WITH CMVC PROCESSES SHOWING QUESTION: I have restarted my families using cmvcd and after a while I checked the processes for the family by using "ps -fu familyName". The resulting list does no longer contain the cmvcd processes but a lot of entries like sfw 21240 1 0 0:00 What can I do ? ANSWER: If the parent process for all the defunct processes is 1, it may mean that inittab has not exited. Check in /etc/inittab for all cmvcd daemons 13 entries with 'wait' or 'once'. Then run 'who -a' and look for these processes. They should all say 'exit=0'. Investigate (or kill) any that have not exited. When you kill the hanging process, all the defunct ones will probably go away. 2.11 CONFLICT WITH STOPCMVC WITH NAMES OF DB2 INSTANCE AND CMVC FAMILY QUESTION: We are using CMVC with DB2. We created a new DB2 instance called 'cmvc23'. Next, I created CMVC family called 'cmvc23'(same DB2 instance). I ran "db2start", "cmvcd cmvc23 3" and "notifyd". There are no problems at this point. However, when I ran "stopCMVC cmvc23", besides killing the CMVC notifyd daemon and cmvcd daemons, all the DB2 instance's daemons were killed at same time. Is this a bug or not? ANSWER: Your problem is caused by the stopCMVC script and it occurs because the name of the DB2 instance is the same as the family name. The script stopCMVC script uses "ps -ef" to identify the CMVC daemons that are logged, it is possible to misidentify some other processes that have the "cmvc" string in them, such as a user "cmvcdemo", "cmvca" (for user CMVC "A"), "cmvc23", a DB2 instance called "cmvc", etc. You will have to set the DB2 instance with a different name to overcome the problem. 14 CMVC FAQ: administration 3.0 NOTIFYD AND MAIL NOTIFICATION 3.1 WHAT IS THE INTERACTION BETWEEN THE CMVC SERVER AND NOTIFYD? QUESTION: What is the interaction between the CMVC server and notifyd? ANSWER: The CMVC Server's notification subsystem uses the UNIX sendmail facility to deliver mail to users. This subsystem consists of the notifyd daemon (a background process) and its connection to a database shadow process. The server daemon processes initiate a notification during the processing of most CMVC successful actions. A message and a limited set of delivery instructions are placed in the CMVC family's $HOME/queue/messages and $HOME/queue direc- tories respectively. The notifyd daemon periodically reads the $HOME/queue directory looking for instruction files to process. After reading the instruction files, a query is made to the database to resolve the notification requests into a list of addresses suitable for the sendmail command. The actual message to be used is obtained from the $HOME/queue/messages subdirectory. Any workstation or host that is able to receive mail in SMTP (simple mail transfer protocol) format can receive mail from a CMVC family. The notifyd daemon instructs the sendmail command to place the mail into the sendmail queue. The mail will be delivered the next time the sendmail queue is read. After the notifyd daemon has processed the sendmail command, the message and instruction files are removed from the family's UNIX account on the CMVC server. notifyd and mail notification 15 3.2 DIAGNOSING AND CORRECTING TYPICAL MAIL PROBLEMS QUESTION: Diagnosing and correcting typical mail problems ANSWER: The following are typical mail problems: o The mail for specific users is not being routed either because of a name resolution timeout or because the user is unknown. The family administrator should periodically check the mail queue for the CMVC family's account. Any mail that is not being delivered will eventually be rerouted back to the originator after the retry expiry period is reached. - Read this undelivered mail messages. The header shows the intended destination. - If 'Deferred: A route the remote host is not available' is displayed in the header, then this means that the user was identified but the connection to the host is tempo- rarily unavailable. The only way to avoid this situation is to chose a route that is guaranteed to be available all the time. - If 'Host Unknown' is displayed in the header, then this means that the host is not known on the network. This situation must be corrected immediately otherwise junk mail will accumulate. The family administrator should contact the person for whom this mail was intended and modify the user's mail address in the CMVC User ID. If the user no longer exists then the CMVC notification list and User ID for this user should be deleted. Undelivered mail should be removed from the family's mail queue. o The mail is arriving but it is delayed. If user's are complaining that the mail that they receive is outdated, then the sendmail queue processing time interval should be checked. 16 CMVC FAQ: administration A reasonable time interval is 10-15 minutes for sendmail to wake up and process the queue. The default is 30 minutes. The value can be modified in the /etc/rc.tcpip file and the sendmail daemon on the server can be restarted. 3.3 HOW CAN I MANIPULATE THE TEXT OF THE MESSAGE TO NEW USERS ? QUESTION: How can I manipulate the text of the message to new users ? ANSWER: The current implementation of the notification of CMVC messages does not provide the flexibility to alter the actual text. However, you can use a user exit for user Create and send another (customized) message to the new user. You can use the sample user exit checkPubsImpactDefectOpen1 which is available in 2.3.0.22. 3.4 HOW TO DIAGNOSE PROBLEMS WITH NOTIFICATION (NOTIFYD)? QUESTION: How to diagnose problems with notification (notifyd)? ANSWER: o Check that the notifyd daemon is running. o Check if sendmail daemon is running in the CMVC server. o Check if the appropriate mail daemon/process is running in the CMVC client. If a client host is not powered on or if the mail daemons are not active, then the sendmail daemon from the CMVC server will give up the delivery of mail to that client (after a certain period). notifyd and mail notification 17 3.5 SUBDIRECTORIES QUEUE AND QUEUE/MESSAGES IN THE CMVC FAMILY ACCOUNT QUESTION: There are many file names like "i.F8MIc5" with old dates in the subdirectory $HOME/queue of the CMVC family. What are these files? Why are still there? Can they be deleted? ANSWER: The queue and queue/messages subdirectories of a CMVC family account contain files that the CMVC notifyd daemon will use to send notifications from users. We fixed in our latest levels of CMVC a bug in which certain CMVC actions such as file add, created these queue files with the wrong ownership, which caused the notifyd to ignore them and leaving them in the queue subdirectories. The workaround is to change the file ownership of these files to the CMVC family account, then the next time that the notifyd is run, it will process and delete them. 18 CMVC FAQ: administration 4.0 CMVCARCHIVE AND CMVCRESTORE 4.1 WHAT IMPORTANT DETAILS SHOULD I KNOW ABOUT CMVCARCHIVE AND CMVCRESTORE? QUESTION: What important details should I know about cmvcarchive and cmvcrestore? ANSWER: o They are executed from the CMVC server. The CMVC family administrator must log in to the CMVC fami- ly's account on the CMVC server to run the programs. o The executables are found in /usr/lpp/cmvc/bin/cmvcarchive and cmvcrestore. o They are interactive, menu-driven programs. o The prerequisites must be met before running the programs. o Feedback messages are displayed to monitor the program status and progress. o The messages "Archive complete." and "Restore complete." indicate the successful completion of the programs. o The time to complete an archive or restore varies according to the size of the CMVC objects to be handled and on the system resources. o They cannot run unless the cmvcd server daemons are stopped or are running in maintenance (read-only) mode. Use the -m flag when starting the cmvcd server daemons to start in maintenance mode in which the daemons will only accept Report and File -extract commands. o cmvcd, cmvcarchive and cmvcrestore make use of system semaphores to guarantee that they are the only programs accessing the database in update mode. Use IPCS and IPCS -S commands to check and ____ _____________________ clear semaphores when necessary. Under normal conditions, the semaphores will be set and cleared automatically. If the user receives a message indicating that the programs cannot be run (either because the cmvcd server daemons are running in regular mode or another cmvcarchive/cmvcrestore program is running), and if they are not running, then the semaphore has not been cleaned up from the last execution. In this case, use the IPCS -S command to clear _____________________ the semaphore whose id is owned by the CMVC family's account. cmvcarchive and cmvcrestore 19 IPC status from /dev/mem as of Mon Jun 14 11:40:34 EDT 1993 T ID KEY MODE OWNER GROUP Message Queues: Shared Memory: m 0 0x0d050207 --rw------- root system m 1 0x58067039 --rw-rw-rw- gray staff m 2 0x52564801 --rw-rw---- informix informix m 3 0x000000dc --rw------- cmvcdemo system Semaphores: s 0 0x4d07004b --ra-ra---- root system s 3 0x58067039 --ra-ra-ra- gray staff s 4 0x52564801 --ra-ra---- informix informix s 5 0x6c090c7b --ra------- cmvcdemo system 4.2 WHAT ADDITIONAL DETAILS SHOULD I KNOW ABOUT CMVCARCHIVE? QUESTION: What additional details should I know about cmvcarchive? ANSWER: o Archive options: - Estimate storage required for archive. Estimates the amount of local file system storage required to archive levels or releases. In many cases the value specified will overestimate the amount of storage required because an average size for defect and feature notes and file version remarks is used to calcu- late the estimate. - Archive levels of a release up to and including a speci- fied level. When archiving levels, the users indicate the level they want to archive and the cmvcarchive program will archive that level and all levels committed prior to that level. In a sense, this is a baseline in which all work com- pleted prior to that time is archived. - Archive one or more releases. o Archive to a local file system or a device. Local file systems must be empty and accessible to the cmvcarchive program. Devices must be raw devices (for example, in AIX /dev/rmt0). When archiving to a device, files containing data being archived will be written in either the home directory of the family or a user-specified working directory prior to being written to the device. 20 CMVC FAQ: administration When archiving to a local file system, the version control files are cleaned up during the archive operation. When archiving to a device, the version control files are cleaned up during the restore operation. o Optional removal of archived objects from CMVC. Removal of objects allows the user to reclaim database and file system space. Users should verify the archive VERY CAREFULLY prior to selecting the remove option. The remove option takes more time to complete than the normal archive. Archived objects are deleted from the database. Defects and Features associated with archived tracks are deleted if no other tracks are associated with them and all verification records have been marked. Level maps for archived levels are deleted from the $HOME/maps directory on the server. Version control files are deleted from the $HOME/vc tree if a release archive is performed and all common/shared files are archived. o Archive prerequisites must be satisfied. o Creates files containing data required for restore. o Recommendations: Backup your CMVC family before attempting to do a CMVC archive. Perform an archive without delete, verify the restore, then perform the archive with delete. 4.3 WHAT ARE THE PREREQUISITES FOR CMVCARCHIVE? QUESTION: What are the prerequisites for cmvcarchive? ANSWER: o General Prerequisites: - CMVC Server at V2R1 or later. - cmvcd not running or running in maintenance mode. o Level Archive Prerequisites: - Level exists in the complete state for the release. - Previously committed levels in complete state. - Common/shared files are not locked (PVCS systems). cmvcarchive and cmvcrestore 21 - Tracks that are levelMembers are complete. - Levels not being archived do not include any tracks that are being archived (that is, a track included in more than 1 level) o Release Archive Prerequisites: - Releases exist; levels for the releases are complete. - No files are locked for the releases. - All tracks are complete for the releases. - Sizing records for the releases are marked. 4.4 WHAT ARE THE FILES CREATED BY CMVCARCHIVE? QUESTION: What are the files created by cmvcarchive? ANSWER: The following files are created by cmvcarchive: o CMVCarchive.info It contains information related to the archive (for example, date and time, objects archived). It is used during the restore phase to validate prerequisites. o CMVCarchive.dbData It contains archived object data. It is used during the restore phase to populate the database. o CMVCarchive.export It is created when database=ORACLE. It contains export of the Notes and Versions database tables of the family; it is imported during restore. o CMVCarchive.notesData It is created when database=INFORMIX/SYBASE: - INFORMIX -> holds notes data for archived defects and features. - SYBASE -> holds bulk copy of the Notes database table. It is imported during restore. o CMVCarchive.versionsData It is created when database=INFORMIX/SYBASE: 22 CMVC FAQ: administration - INFORMIX -> holds versions data for the deltas of archived files. - SYBASE -> holds bulk copy of versions database table. It is imported during restore. o CMVCarchive.users It contains the list of CMVC logins in archived data. It is used upon restore to verify the first superuser of the family. o CMVCarchive.Vccleanup It is created when archiving to a device. It contains a list of deltas to be removed from the version control files upon restore. Format of file is: sourceId : fileType SID SID SID sourceID : fileType SID where - sourceID is 0-6 digit number representing the location of the file in the version control tree (for example, 5 = /vc/0/0/0/0/s.05) - fileType = 0 for text; 1 for binary - SID = delta that is to be removed from the sccs/pvcs file during the restore o CMVCarchive.PVCS_clean It is created in the family's home directory when PVCS is used by the family and the user selects the 'remove archived objects' option. The user will have to manually delete version control files listed in this file because the cmvcarchive program does not have the permission to do so. o CMVCarchive.CPIOlist It is created with archiving to a device. It contains a list of files to be cpio'ed to the device. This file will also be cpio'ed to the device. Also used during restore to determine where the files get deposited when extracted from the device. cmvcarchive and cmvcrestore 23 4.5 WHAT ADDITIONAL DETAILS SHOULD I KNOW ABOUT CMVCRESTORE? QUESTION: What additional details should I know about cmvcrestore? ANSWER: o Restore options: - Restore from local file system. - Restore from device. o All archived data is restored (that is, there is no selection mechanism). o Data is restored to a NEW CMVC family. ___ o Restore prerequisites must be satisfied. o The database and the family's file systems are populated with archived data. o The configurable fields are set. 4.6 WHAT ARE THE PREREQUISITES FOR THE CMVC FAMILY TO BE RESTORED? QUESTION: What are the prerequisites for the CMVC family to be restored? ANSWER: o CMVC Server at V2R1 or later. o cmvcd not running or running in maintenance mode. o The restored family will be new (mkfamily and mkdb executed only). o The first super user is not a member of the archived data (check CMVCarchive.users) o No configurable fields are set for the new family. o The $HOME/vc and maps directories are empty. o The $HOME/vc file system owned by the family id until after the cmvc restore. 24 CMVC FAQ: administration PVCS users must switch the owner of $HOME/vc to the PVCS owner AFTER the restore is done. 4.7 IF HAVING PROBLEMS WITH CMVCARCHIVE, THEN MIGRATE TO CMVC 2.3.0.22 QUESTION: If having problems with cmvcarchive, then migrate to CMVC 2.3.0.22 ANSWER: Due to many fixes to cmvcarchive/cmvcrestore done between the refreshes 2.3.0.18 and 2.3.0.22, if you are having problems with these functions then upgrade to latest refresh. 4.8 CMVC IS NOT RECLAIMING FILE SYSTEM SPACE WHEN DOING CMVCARCHIVE QUESTION: CMVC is not reclaiming file system space when doing cmvcarchive ANSWER: This is a clarification of the term "reclaiming space" when doing a cmvcarchive: In order to remove an archive from the vc tree, NO rows in the Files table can point to the archive (or archive directory in the case of binary files). The utility cmvcarchive will delete an archive if, and only if, ALL entries in the Files table that point to a given archive are specified. In other words, you have to delete all releases that point to an archive file at the same time in order to get rid of the archive. Because this is not an easy thing, few archives are actually deleted. cmvcarchive and cmvcrestore 25 4.9 CMVCARCHIVE IS TAKING A LONG TIME TO RUN QUESTION: I am running cmvcarchive and it is taking a long time to run. ANSWER: Archiving to tape is much slower that archiving to disk. We have had a few instances where tape archives have failed after long periods because of tape i/o problems. We do not know of any instances where the archive has just hung, only taken several days to run. Because it takes so long, we have added some code to warn you when deletion begins (if you are using that option), so you know when you have reached the point of no return. It also turns out that the beginning of deletion is about the halfway point. If you do not intent to keep all of the releases installed together at any one time, then archive them separately. This will also use less space, so that you might be able to archive to disk, then tar off the files to tape is a separate step. We wish that we could make things faster, but the queries and other processing are simply very time consuming. 4.10 SEVERAL PROBLEMS WITH CMVCARCHIVE QUESTION: 1. I recently performed an archive of a CMVC release and had a failure during the archive. One suspicion is that disk space was becoming an issue with the system. 2. The cmvcarchive stated that it needed 161 MB to archive. The file system had 250 MB of space so I proceeded. I suspect that the database files need additional space to archive also. 3. Because CMVC is down during the cmvcarchive process, what error logs are used?. 4. I am having problems when specifying multiple arguments to cmvcarchive, because I seem to be reaching a limit as to how many releases I can input or at least how many arguments I can specify. ANSWER: 26 CMVC FAQ: administration 1. The tool cmvcarchive performs a little of everything in CMVC. As a result, there are lots of things that can go wrong: o Running out of space in /tmp (for the SCCS commands). o Running out of space in the database transaction log (this is one thing cmvcarchive does inefficiently). o Running out of space in real memory (for the query). o Running out of space in the target directory. This is a difficult one to watch for without a background process, since cmvcarchive cleans up space as soon as it fails. However, the problems are displayed on the screen. 2. Our experience is that the size estimate has always been larger than necessary. This is not a guarantee, but we have not seen yet a lower than necessary estimate. 3. Errors are written to standard error, that is, the screen. 4. The problem that you are experiencing is actually in the shell, not with cmvcarchive. In order to circumvent the lim- itations of the number of characters entered at a prompt, you need to redirect the input from a file: a. Do the steps manually to determine what you normally enter. b. Put your answers in a file. Below is the contents of one such file: 2 2 plink 1 1 /tmp/plink c. Run your cmvcarchive as follows: cmvcarchive < answers 2>&1 | tee cmvcarchive.out 4.11 HOW TO FIND OUT A MISSING PATH ID (DUE TO CMVCARCHIVE) QUESTION: How to find out a missing path id (due to cmvcarchive) ANSWER: cmvcarchive and cmvcrestore 27 When running cmvcarchive there is an error about a missing path id. What should be done to overcome the problem? 1. Using SQL from UNIX command line: Create a file with name xxx.sql (.sql required by Oracle). Here is a query (and exit) that will find all occurrences of missing Path table records (again, Oracle syntax): select f.id, f.pathId, f.baseName, r.name from Files f, Releases r where f.releaseId = r.id and r.name = '&CMVC_RELEASE' and f.pathId not in (select p.id from Path p where p.id = f.pathId); quit; There are important notes to consider: o Using Oracle syntax, the user will be prompted for the value of CMVC_RELEASE when the query is run. o You can change the syntax of the above query to look for a list of releases by replacing: r.name = '&CMVC_RELEASE' With: r.name in (&CMVC_RELEASES) Of course, you will then need to put apostrophes (') around each release in the set. o Run the query: sqlplus user/password @xxx or sqldba lmode=y @xxx 2. Fix the problem: a. Logon as the CMVC family administrator b. Connect to the database, such as: sqlplus user/password Without this step, you need to add the familyname in front of each table (for example, "select * from Fam.Files") c. Issue the following insert command (one single line): 28 CMVC FAQ: administration insert into Path (id, name) values (21799, 'src/dubparms.h'); "src/" is enough of a path d. Verify that the file is now available in FileView with: Report -view FileView -where "pathId=21799 and releaseName=r020" e. Run an archive check. cmvcarchive and cmvcrestore 29 30 CMVC FAQ: administration 5.0 USER EXITS 5.1 WHAT ARE USER EXITS? QUESTION: What are user exits? ANSWER: User exits are user-written programs or shell scripts that execute in the CMVC server during the course of a CMVC action. User exits provide the ability to extend the functionality of CMVC to trigger non-CMVC processing during the processing of CMVC transactions. There is no support for user exits in the CMVC clients. Family administrators can configure user exits for most of the CMVC actions. o To establish user exits, the family administrator must modify the USEREXITS file found in the $HOME/CONFIG directory on the CVMC server. When modifying the USEREXITS file: - Read the comments in the file and follow the definition format. - Only define one user exit per exit id. If you define multiple user exits for the same CMVC action and exit id then only the last one in the file will take effect. - The format of the definition file is: CMVCaction ExitID UEprogram UEparameters #Comments o The user exits programs themselves must be placed in the $HOME/BIN directory on the CMVC server. When writing user exit programs or shell scripts: - Remember to return a 0 exit status if everything is OK and you want the CMVC action to continue. User exits 31 - Remember to return a non-zero exit status if everything is not OK and if you do not want the CMVC action to con- tinue (only for exit id's of 0 and 1) - If the user exit program is not behaving as you expected, try echoing or printing all of the input arguments that the user exit is receiving from the CMVC server. This will help you determine if you are interpreting the correct input arguments. - It is not recommended that you invoke another CMVC command from within your user exit program. This could lead to a deadlock situation. o If you created or modified user exits while the cmvcd daemons are running, then you will have to restart the cmvcd daemons in order to get them to acknowledge the user exits. The cmvcd server processes read and initialize the user exit points when they start up. 5.2 SHOWING A SMALL USER EXIT EXAMPLE QUESTION: Showing a small user exit example ANSWER: The following is an example of the userExits definition file in the $HOME/CONFIG directory on the CMVC server: # # FileAdd User Exits # FileAdd 0 Check_Date Monday, Feb 3, 1991 FileAdd 1 Check_Date Monday FileAdd 2 Check_Date 02/03/91 # The following is an example of the 'Check_Date' user exit. This particular example is a shell script that just echoes the input arguments that it receives. #!/usr/bin/ksh # # Check_Date User Exit # echo "This is the Check_Date CMVC/6000 User Exit echo "User Exit Name = $0" 32 CMVC FAQ: administration for i do echo $i done exit 0 5.3 WHAT ARE THE TYPICAL PROBLEMS WITH USER EXITS? QUESTION: What are the typical problems with user exits? ANSWER: Some situations that may cause problems with user exits are: o $HOME/bin is not in the CMVC family's PATH. o The user exit programs in $HOME/bin are not executable. o The user exit programs in $HOME/bin are not owned by the CMVC family. o The cmvcd daemons were not restarted after creating or modi- fying the $HOME/config/userExits file. o If using Bourne shell, the "shift" statement is not used to get the positional parameters greater than 9. 5.4 WARNING ON USING CMVC LINE COMMANDS INSIDE CMVC USER EXITS QUESTION: Warning on using CMVC line commands inside CMVC user exits ANSWER: Even though the CMVC Server manual implies in page 88 (V2.3 edition) that it is OK to use the CMVC Report command inside user exit, it is possible to cause deadlocks in these situations if there are no enough available CMVC daemons to process the Report command. Thus, to avoid these deadlocks, ensure that you have enough data- base licenses for the proper amount of CMVC daemons. User exits 33 5.5 NEW PARAMETERS FOR USER EXITS IN CMVC COMMANDS STARTING 2.3.0.22 QUESTION: New parameters for user exits in CMVC commands starting with CMVC 2.3.0.22. ANSWER: The user exit parameters passed in the following CMVC commands have been updated. Below is the new parameter, trackList, shown with the parameter before and after it in the updated list. The new parameter, trackList, is a string that contains the track names of all features/defects specified in the CMVC command. See "Appendix H: User Exit Parameters" of the "CMVC Server Admin- istration and Installation" manual for the previous version of the complete parameter list. Also, as stated in Appendix H, the new parameters for ExitID 3 are the same as ExitID 0. The new parameter has NOT been added to the end of the parameter list, but it has been inserted between existing parameters; for example, with the command File -create (action FileAdd), in user exit id 0, the new parameter "trackList" has been inserted between "remarks" and "fileMode"; thus, the list of parameters for this user exit before CMVC 2.3.0.22 was: 1 file path name 2 temp file on server 3 release name 4 component name 5 file type 6 remarks 7 fileMode 8 effective CMVC user ID 9 real Unix login 10 verbose flag But after CMVC 2.3.0.22 is: 1 file path name 2 temp file on server 3 release name 4 component name 5 file type 6 remarks 7 trackList 8 fileMode 9 effective CMVC user ID 10 real Unix login 11 verbose flag The user exits that are affected are shown below. 34 CMVC FAQ: administration CMVC Command CMVC Exit After Parm., New Parameter, Before Parm. Action ID File -create FileAdd 0 remarks, trackList, fileMode" File -create FileAdd 1 remarks, trackList, fileMode File -create FileAdd 2 remarks, trackList, fileMode File -checkin FileCheckIn 0 remarks, trackList, common flag File -checkin FileCheckIn 1 remarks, trackList, common release name File -checkin FileCheckIn 2 remarks, trackList, common release name File -delete FileDelete 0 force flag, trackList, common flag File -delete FileDelete 1 component name, trackList, effective CMVC user ID File -delete FileDelete 2 component name, trackList, effective CMVC user ID File -link FileLink 0 SID, trackList, effective CMVC user ID File -link FileLink 1 component name, trackList, effective CMVC user ID File -link FileLink 2 component name, trackList, effective CMVC user ID File -migrate FileMigrate 0 component name, trackList, SID File -migrate FileMigrate 1 component name, trackList, SID File -migrate FileMigrate 2 component name, trackList, SID File -recreate FileRecreate 0 force flag, trackList, common flag File -recreate FileRecreate 1 force flag, trackList, common flag File -recreate FileRecreate 2 force flag, trackList, common flag File -rename FileRename 0 force flag, trackList, common flag File -rename FileRename 1 force flag, trackList, common flag File -rename FileRename 2 force flag, trackList, common flag File -undo FileUndo 0 force flag, trackList, common flag File -undo FileUndo 1 force flag, trackList, common flag File -undo FileUndo 2 force flag, trackList, common flag Release -link ReleaseLink 0 date, trackList, effective CMVC user ID Release -link ReleaseLink 1 date, trackList, effective CMVC user ID Release -link ReleaseLink 2 date, trackList, effective CMVC user ID Figure 1. Updated parameter list for user exits User exits 35 5.6 APPARENT PROBLEM WHEN PASSING MORE THAN 10 PARAMETERS TO A USER EXIT QUESTION: Some CMVC family administrators report that there is a problem when passing more than 10 parameters to a user exit, some of them appear to be wrong, particularly the ones that correspond to the positional parameters 10, 11, and so on. ANSWER: The Bourne shell only supports positional parameters $0 through $9. In order to access parameters 10 or greater, you must use the "shift" command. Shift will move $2 to $1, $3 to $2, etc. ($0 is the name of the command being invoked and never shifts). Also, you can shift more than one position at a time. For example, you can use $1 through $9, then issue the command "shift 9" and process variables 10 through 19 using the variables $1 through $9 again. The Korn shell supports positional parameters greater than $9 using braces. For example, you can access the eleventh posi- tional parameter as ${11}. 5.7 TEMP FILE FOR USER EXIT 2 FILECHECKIN NOT LONGER AVAILABLE QUESTION: temp file for user exit 2 FileCheckIn not longer available ANSWER: Even though the CMVC manual describes that the user exit 2 for FileCheckIn passes the temporary file name, at this stage the real file is gone. However, using user exit 1 is fine, that is, the real file is still available. 5.8 ARE THERE ANY REAL EXAMPLES OF USEREXITS SHIPPED WITH CMVC? QUESTION: Are there any real examples of UserExits shipped with CMVC? ANSWER: 36 CMVC FAQ: administration #!/usr/bin/sh # # This is a Bourne shell example - positional parameters # ------------------------------------------------------------------ # Giving legible names to the input parameters # It is nice to list ALL the parameters, it helps with the # debugging and overall understanding. # NOTE: It is important to deal with the "shift" command to # handle the input positional parameters greater than 9 # ------------------------------------------------------------------ # echo "parameters passed: " # echo "1 is component name: " $1 # echo "2 is prefix: " $2 prefix=$2 # echo "3 is severity ID: " $3 # echo "4 is reference: " $4 # echo "5 is environment name: " $5 # echo "6 is remarks: " $6 # echo "7 is level name: " $7 # echo "8 is abstract: " $8 # echo "9 is release name: " $9 shift 9 # echo "10 is configurable field string: " $1 string=$1 # echo "11 is defect number: " $2 defectNumber=$2 Figure 2. Sample Bourne shell using positional parameters Some CMVC users would like to do some extra activities during the processing of CMVC commands at the CMVC server host and the users can exploit the feature called "user exits" (there are no user exits in the CMVC clients). In CMVC 2.3 we added in the directory /usr/lpp/cmvc/samples several samples of user exits. For more details, you can take a look at the file "server.samples" in the mentioned directory. User exits 37 #!/usr/bin/ksh # # This is a Korn shell example - positional parameters # ------------------------------------------------------------------ # Giving legible names to the input parameters # It is nice to list ALL the parameters, it helps with the # debugging and overall understanding. # ------------------------------------------------------------------ # echo "parameters passed: " # echo "1 is component name: " $1 # echo "2 is prefix: " $2 prefix=$2 # echo "3 is severity ID: " $3 # echo "4 is reference: " $4 # echo "5 is environment name: " $5 # echo "6 is remarks: " $6 # echo "7 is level name: " $7 # echo "8 is abstract: " $8 # echo "9 is release name: " $9 # echo "10 is configurable field string: " $10 string=$10 # echo "11 is defect number: " $11 defectNumber=$11 Figure 3. Sample Korn shell using positional parameters 38 CMVC FAQ: administration 6.0 ERROR HANDLING AND AUDIT LOG 6.1 WHAT DETAILS SHOULD I KNOW ABOUT ERROR MANAGEMENT? QUESTION: What details should I know about error management? ANSWER: When a cmvcd or a database error occurs, a message is displayed in the server's syslog (/usr/spool/syslog) or on the server console. The database performs a rollback of the command and the server process is reset so that it can receive additional client requests. The client does not know that the database error has occurred. It only knows that the connection to the server was lost (a message is displayed on the client side indicating that the connection is lost). Reports are a special case of database error since the user is able to type any SQL selection query. In this case, the database error is returned to the client so that the user can correct the error. When errors occur due to problems other than the database, a message is generated and returned to the client. The message is displayed through stderr. The CMVC server and line command errors are identified by the 0010-XXX message number. The CMVC GUI errors are identified by the 2059-XXX message number. 6.2 WHERE ARE THE ERROR MESSAGES FROM THE CMVC DAEMON AND DATABASES? QUESTION: Where are the error messages from the CMVC daemon and databases? ANSWER: The error messages from the CMVC daemon and the database are logged in the syslog. Error handling and audit log 39 You will need to enable the syslog facility, as explained in the technical report "CMVC frequently asked questions: handling OEM platforms and databases", section "Where are the error messages from the CMVC daemon and databases?" 6.3 HOW TO GET MORE INFORMATION FROM DB2 ERROR NUMBERS? QUESTION: Sometimes there are DB2 error numbers shown in the syslog or in CMVC error messages. Is it possible to get more on-line informa- tion about them? ANSWER: To obtain more on-line information about a DB2 error number, such as "-818", you may use the following DB2 command: db2 ? sql---- Where ---- are the 4 digits of the error number. In case that the number has only 3 digits, such as 818 or -818, then simply add a zero to the left of the number, such as: db2 ? sql0818 In this case, this is a sample output: 40 CMVC FAQ: administration SQL0818N A timestamp conflict occurred. Explanation: The timestamp generated by the precompiler at precompile time is not the same as the timestamp stored with the package at bind time. This problem can be caused by the following: o Precompile, compile, and link without doing an application bind. o Precompile and bind without doing the program compile and link. o Bind the application using a bind file that resulted from a different precompile of the program than the precompile that produced the object module linked into the application module. o Bind an application with the same name as an existing plan and execute the existing (old) application. The statement cannot be processed. User Response: Bind the application again, using the bind file for the program that matches the object module. Or, execute the program that corresponds to the package stored in the database. If installing the sample database, record the number and text of this message and then contact your technical service representative. sqlcode: -818 sqlstate: 51003 By the way, the CMVC utility "db2bind" can be used to bind the package files (*.bnd) to the database for CMVC. 6.4 HOW TO DEAL WITH THE 0010-250 ERROR MESSAGE QUESTION: CMVC client shows this error message: 0010-250 A connection cannot be established with family XXXX at node YYYY on port PPPP. The error is: Connection timed out. ANSWER: Error handling and audit log 41 This error message is ONLY issued when the CMVC client is trying to connect to the CMVC server via the TCP/IP sockets. Some possible workarounds are: o If this behavior is consistent: Probably the cmvcd server daemons are not running. Logon to the CMVC server and check the status by issuing: ps -ef | grep cmvcd Also you can use the monitor command: monitor 1 If the CMVC daemons are not longer running, check the errors in the syslog (in AIX, the default name is /var/spool/syslog) o If this behavior is intermittent: It might be possible that there are some problems with the underlying network and name server. You may issue the 'host' TCP/IP command in AIX and OS/2 (or the cmvchost command for SunOS, Solaris and HP-UX) for the server node (on the command line) before starting your client. For example: host serverName Then repeat the host command and this time use the IP address for this server which was shown in the previous command. If you do not get a good reply from 'host' for the machine name and IP address, the network still is having problems. 6.5 CMVC ERROR MESSAGE: 0010-256 QUESTION: CMVC error message: 0010-256 ANSWER: The 0010-256 error message is a generic (almost a catch-all) error message. The CMVC 2.3 Server manual says the following about this error, but there is additional information that was not included in the manual which is shown at the bottom of this note: 42 CMVC FAQ: administration 0010-256 An error occurred when the CMVC server software tried to process function . It is possible that the network or the CMVC server is experiencing problems. EXPLANATION: The CMVC client software and the CMVC server software may be incompatible. Or your database file system may be full. This message is displayed on the CMVC client and on the CMVC server console, or entered into the syslog file if the syslogd daemon is running on the CMVC server. USER RESPONSE: o Verify that the version of the CMVC client software is the same as the version of the CMVC server software and that the port number designated to the CMVC family is unique on the network. o Check the amount of free space in the file systems; verify that your database file system has sufficient space for your CMVC server operations. o Message 0010-257 is also displayed on the CMVC client and on the console of the CMVC server, or entered in the syslog file if the syslog daemon is running. o Please check for: - Is your $HOME/vc tree full? - Is your database full? - Is your system table full? - Is the /tmp full? o Take a look at additional messages recorded in the syslog (see last page of the Chapter 14 from the CMVC 2.3 Server manual. The syslog recorded by syslogd usually gives you good hints. If you have not got it set up yet, you need to set it up and run the CMVC command again to catch the error in the syslog. 6.6 ERROR MESSAGE: 0010-063 WITH KEYWORD BIND() QUESTION: Error Message: 0010-063 Error, Address already in use, occurred when the CMVC Server/6000 attempted to process the bind() function. Error handling and audit log 43 ANSWER: 1. Situation A: dealing with a new family When creating and starting a new family, if this error appears, the most likely cause is that the port number for the new family is already being used by another service. Please take a look at the /etc/services file and ensure that the port number for the new family is unique and not in use by another service. Try to avoid "nice and round" numbers such as "3000", and try to use "odd" numbers such as "3125". We have encountered a couple of situations where a port number with a "nice and round" number was hard-coded in an application instead of being defined in the /etc/services" file. 2. Situation B: dealing with existing families, part 1 We have seen these errors when there is a conflict with the shared memory. We have added some cleaning up code in the stopCMVC sample in CMVC 2.3.0.18. In the meantime, please do the following steps: a. Ensure that all the CMVC daemons are down: stopCMVC familyName b. Ensure that there are no "orphan" cmvcd daemons: ps -ef | grep cmvcd If there are "orphan" cmvcd daemons, kill them with "kill -15". c. Invoke the CMVC Activity Monitor which shows the activity of the CMVC daemons (which at this point there should be none) and performs the cleaning of the shared memory (which is what we are trying to accomplish in this step). monitor 1 Press enter to terminate the monitor command. d. Restart your family: cmvcd yourFamily 2 e. Verify that the daemons are running: ps -ef | grep cmvcd p. In this example there should be 4 entries with cmvcd. 44 CMVC FAQ: administration 3. Situation C: dealing with existing families, part 2 When using an existing family and the situation above is not applicable, then this error may happen when stopping and restarting the cmvcd daemons; the most likely explanation is: What was probably happening was that you tried to restart the daemons, but the nfsmounthelper process was still attempting to do the NFS write. Therefore, the port or socket was already reserved for the communications between server and client. a. If you have to re-ipl the server to fix the problem because you cannot restart the daemons and could not unmount the file system in /tmp. b. Then you might have had reasonable luck stopping NFS, perhaps via /etc/nfs.clean. c. Then "kill -1 1" to kick off the orphaned NFS process that got attached to init when you stopped NFS. d. Then restart NFS. If using AIX, this can be accomplished by either through SMIT or /etc/rc.nfs. 6.7 CMVC ERROR MESSAGE 0010-063: NOT OWNER, CHOWN() FUNCTION QUESTION: When trying to create or checking a file, the following error gets placed in the syslog: 0010-063 Error, Not owner, occurred when the CMVC server attempted to process the chown() function, ... ANSWER: It appears that on some installations, /usr/lpp/cmvc/bin/cmvcd is owned by bin.bin instead of root.system. If you have this problem, please apply the following command (as user root): chown root.system /usr/lpp/cmvc/bin/cmvcd 6.8 ERROR MESSAGE 0010-063 - BROKEN PIPE, WRITE () FUNCTION QUESTION: The following message is received several times during a week in the /var/spool/syslog file: Error handling and audit log 45 0010-063 Error, Broken pipe, occurred when the CMVC server software attempted to process the write() function. ANSWER: One possibility is when the user closes the CMVC GUI using the UNIX pull down and selecting the Close item instead of using the function within the CMVC file->closed selection. The UNIX Close function will cause the pipe to break. 6.9 WHAT DETAILS SHOULD I KNOW ABOUT AUDIT LOGGING? QUESTION: What details should I know about audit logging? ANSWER: Successful CMVC actions are logged in an audit log, which is found in the $HOME/AUDIT directory on the CMVC server. The name of the log file is LOG Unsuccessful transactions are not logged. The audit log can be used as a debugging aide to retrace the sequence of events that have successfully completed. Regular maintenance of the audit log is recommended as it will grow infinitely. See the CMVC Administration and Installation manual for maintenance tips. Regularly check the amount of free space in the file system that contains the audit log and allocate new storage if required. 6.10 AFTER RUNNING THE SAMPLE CMVCLOG.CLEANU, CMVCD DOES NOT LOG QUESTION: After I ran the /usr/lpp/cmvc/samples/cmvclog.cleanu command, my cmvcd does not log entries into the new $HOME/audit/log anymore. ANSWER: After you run the sample shell script to clean up your audit log file, remember to bring down then up your cmvcd to have it log entries into the new log file. 46 CMVC FAQ: administration 7.0 DATABASE ISSUES 7.1 INDEXES THAT IMPROVE PERFORMANCE IN LEVELS AND FILE CHANGES QUESTION: Indexes that improve performance in levels and file changes ANSWER: One of the CMVC support groups has provided us with additional indexes that they used to improve performance for Level and Files related commands. These new indexes were added to CMVC in 2.3.0.18. These indexes are NOT automatically added to existing CMVC databases, they are automatically added only to new databases. Thus, you may need to manually add these indexes. o Added in Level 017 -> GA of 2.3.0.18 (Level check and commit) create index XChaFilTrkVerTyp on Changes (fileId, trackId, versionId, type); commit; create index XLLvlTrk on LevelMembers (levelId, trackId); commit; create index XTrackIdStat on Tracks (id, state); commit; o Added in Level 006 -> GA of 2.3.0.0 (File information) create index XFilPid on Files (pathId); commit; create index XFilesSrcId on Files (sourceId); commit; Database issues 47 7.2 REORGANIZATION AND REBINDING A DB2 DATABASE QUESTION: Reorganization and rebinding a DB2 database ANSWER: After any major changes done to the CMVC database (such as migrating from a previous version/release of CMVC, performing massive file creations, or just after a period of heavy use), it is important to perform the following process for a DB2 database in order to update the statistics for the indexes, in that way, the performance of the database will not be affected: 1. Ensure that the CMVC family daemons are down. 2. Run REORGCHK. 3. Run REORG. You can run the CMVC "db2reorg" command discussed below. 4. Run RUNSTATS. 5. Finally, bind the DB2 plans to the CMVC executables by running the CMVC shell script "db2bind". For more information on the commands REORGCHK, REORG and RUNSTATS, see the DB2/6000 Administration Guide manual, Chapter "Performance Considerations", Section "Reorganizing Tables and Running Statistics". The following shell scripts are included that facilitate the maintenance of DB2 databases used with CMVC families: o db2reorg You can invoke the CMVC shell script "db2reorg" to invoke the DB2 REORG command for the CMVC family. This shell script was added in 2.3.0.18. The CMVC db2reorg shell script provides the following help: db2reorg Purpose: Run the DB2 REORG command for the CMVC family db2reorg Usage: db2reorg CMVCfamily workPath Notes: 1. CMVCfamily is typically the name of the family account 2. workPath is typically a temporary directory, such as /tmp o db2bind is needed to synchronize DB2 plans with database and CMVC executable files. 48 CMVC FAQ: administration The CMVC db2bind shell script is needed to synchronize (bind) the DB2 plans with the DB2 database for the CMVC family and the CMVC executables. This means that whenever there is an update to the CMVC executables, it is necessary to invoke this shell script. The CMVC db2bind shell script provides the following help: db2bind Purpose: Bind CMVC executables to DB2 family instance db2bind Usage: db2bind Notes: 1. Assumes $CMVC_HOME points to cmvc directory (e.g. /usr/bin/cmvc) 2. is typically the name of the family account 3. Database output is stored in db2bind.out 7.3 RUNNING DB2 RUNSTATS IS TAKING A VERY LONG TIME QUESTION: When running DB2 RUNSTATS, it takes a very long time, even for a CMVC family that is very small. Is this OK? ANSWER: The most likely reason for this problem is that you are executing RUNSTATS (or related DB2 utilities such REORG) when the CMVC family daemons are still running. You should stop the CMVC family before you use these DB2 utili- ties. Also, you need to use the CMVC db2bind shell script after you run these DB2 utilities. 7.4 NEED TO REBOOT THE HOST IF SYSTEM HANGS DURING DB2STOP QUESTION: The CMVC family was successfully stopped but when issuing "db2stop" the system hangs. ANSWER: If after a successful shutdown of the CMVC family by using stopCMVC you try to stop DB2 by using "db2stop" and the system hangs, then you may need to reboot your system in order to reset it. Database issues 49 7.5 BACKUPCMVC FAILS WITH DB2 ERROR DB2106E QUESTION: I am using the backupCMVC script to backup our families. When I run the script the first family in the list backs up OK, however, when it comes to the next family backup the DB2 command fails with the following error: DB21016E The command line processor encountered a system error while sending the command to the back end process. ANSWER: You can add the following statement in the backupCMVC, at the end of the 'for i in ${CMVC_family_list}' loop: su - ${i} -c "db2 terminate" That breaks the connection for one database and you can continue on with the second database. 7.6 HOW TO DISPLAY THE SQLCA REASON CODE WHEN THERE IS A DB2 PROBLEM QUESTION: How to display the SQLCA reason code when there is a DB2 problem ANSWER: There are some DB2 systems errors, such as SQL0980 (shown as SQL -980), that provide more information, such as a reason code, on a data structure called SQLCA. The DB2 Messages and Problem Determination Guide, says in Table 4, Appendix B, Reason Codes, that for the SQL0980 message, the reason code is in the second array element of the SQLERRD field in the SQLCA. Th SQL Reference manual, page 239, Appendix B, SQLCA, describes the SQLERRD as an array of 6 integers, and the 2nd element is reserved for internal return code. This data structure is not available to CMVC, but the end user needs to do the following in order to get that extra information. 50 CMVC FAQ: administration The SQLCA can be viewed from the DB2 command line processor by setting the DB2OPTIONS=-a in the profile of the CMVC family account and then running: db2 list command options The SQLCA will be shown at the end of the output and the 2nd field of the SQLERRD structure will contain the reason code. An example is shown below: SQLCA Information sqlcaid : SQLCA sqlcabc: 136 sqlcode: 0 sqlerrml: 0 sqlerrmc: sqlerrp : sqlerrd : (1) 0 (2) 0 (3) 0 (4) 0 (5) 0 (6) 0 sqlwarn : (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) sqlstate: 7.7 HOW TO DUMP THE TRANSACTION LOG IN SYBASE DUE TO NOTIFY QUESTION: The Sybase log segment fills up because CMVC notifyd is doing a BEGIN TRANSACTION and is not rolling it back or committing it. The open transaction prevents Sybase from truncating the syslog when a dump tran database_name to the device is issued. This results in the logsegment filling up over time which causes cmvcd to hung daemons for Sybase 10 or terminate daemons for Sybase 4.9. ANSWER: When notifyd attaches to Sybase a begin transaction is issued. The transaction is kept open during the entire life span of notifyd, and this pending transaction prevents Sybase from trun- cating the transaction log at the time of doing a transaction dump. Because the transaction log is not truncated, then the disk space is consumed until all the space for the log device has been used up. This workaround is a significant deviation from normal practices. The work around is to run a cron job which: 1. Kill notifyd for a given family 2. Perform transaction dump For details, see "How to dump the transaction log in Sybase?" the section about Sybase in the technical report "CMVC fre- Database issues 51 quently asked questions: handling OEM platforms and data- bases". 3. restart notifyd This will allow Sybase to dump the transaction log. The only exposure is that during the interval when notifyd is down there will be no mail notification; to reduce this exposure window, it is recommended to perform this workaround at times of low user activity during the day. 52 CMVC FAQ: administration APPENDIX A. BIBLIOGRAPHY For more information on how to use CMVC, you can consult the fol- lowing manuals and publications: SC09-1596-01 IBM CMVC Client Installation and Configuration SC09-1597-01 IBM CMVC User's Reference SC09-1631-02 IBM CMVC Server Administration and Installation SC09-1633-00 IBM CMVC Concepts SC09-1635-01 IBM CMVC Commands Reference The following Redbooks offer practical advice on CMVC: GG24-4178-00 Did you say CMVC? GG24-4345 CMVC: Customer's Perspective The following technical reports describe in detail useful hints on using CMVC: 29.2130 Diagnosing and solving Release -extract problems with CMVC 29.2169 How to use CMVC with National Language Support (NLS) and Double-Byte Character Sets (DBCS) 29.2180 How to do routine tasks in the OEM Platforms for CMVC 29.2183 Using CMVC for Products with Multiple National Language Versions (NLVs) 29.2232 How to do migration tasks with CMVC 29.2244 How to build and package the CMVC client for Windows 3.1 29.2245 How to build and package the CMVC client for OS/2 29.2253 Comparison between CMVC 2.3 and TeamConnection 2 29.2254 Migrating from CMVC 2.3 to TeamConnection 2 A.1 HOW TO GET ELECTRONIC COPIES OF MANUALS AND TECHNICAL REPORTS Many of the manuals and technical reports mentioned in this docu- ment can be downloaded as follows: o From the IBM intranet (only for IBM employees). o From the Internet (open to everyone). A.1.1 IBM Intranet ___________________ Appendix A. Bibliography 53 A.1.1.1 Web Home Page You can access the CMVC Service/Development Home Page at: http://keithp.raleigh.ibm.com/&tilde.cmvcsupt From the index at the top of the page, select: o Technical Reports and related tools o Copies of ALL the archived versions for the forums CMVC and CMVC6000 o Documentation in PostScript files o Documentation in ASCII text files A.1.1.2 FTP You can download the code from our internal FTP site, by doing: 1. ftp keithp.raleigh.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd pub/cmvc/doc 4. binary 5. get fileName 6. quit A.1.2 Internet _______________ A.1.2.1 Web Home Page Not available. 54 CMVC FAQ: administration A.1.2.2 FTP You can download the code from our external FTP site, by doing: 1. ftp ftp.software.ibm.com 2. login as 'anonymous' and for password give your email address. 3. cd ps/products/cmvc/doc 4. binary 5. get fileName 6. quit Appendix A. Bibliography 55 56 CMVC FAQ: administration APPENDIX B. COPYRIGHTS, TRADEMARKS AND SERVICE MARKS The following terms used in this technical report, are trademarks or service marks of the indicated companies: +---------------------+-------------------------------------------+ | TRADEMARK, | COMPANY | | REGISTERED | | | TRADEMARK OR | | | SERVICE MARK | | +---------------------+-------------------------------------------+ | IBM | IBM Corporation | | AIX | | | OS/2 | | | CMVC | | | DB2/6000, DB2 | | | TeamConnection | | +---------------------+-------------------------------------------+ | PostScript | Adobe Systems Incorporated | +---------------------+-------------------------------------------+ | HP-UX | Hewlett-Packard Company | +---------------------+-------------------------------------------+ | Informix | Informix Inc. | +---------------------+-------------------------------------------+ | PVCS | InterSolv, Inc. | +---------------------+-------------------------------------------+ | Windows | Microsoft Corporation | +---------------------+-------------------------------------------+ | Oracle | Oracle Corp. | +---------------------+-------------------------------------------+ | NFS, | Sun Microsystems Inc. | | SunOS, | | | Solaris, | | +---------------------+-------------------------------------------+ | Sybase | Sybase Inc. | +---------------------+-------------------------------------------+ | UNIX | X/Open Co., Ltd. | +---------------------+-------------------------------------------+ END OF DOCUMENT Appendix B. Copyrights, Trademarks and Service marks 57