Collecting troubleshooting data (MustGather) for IBM Support
When you encounter a problem in CICS Transaction Server for z/OS (CICS TS), before you contact IBM Support, ensure that you collect troubleshooting data, also known as MustGather. CICS Support can use troubleshooting data to identify and resolve your specific problem. Different symptoms require different data. MustGather documents expedite the troubleshooting process and save you time.
General information
Gather the following general information for every problem record:
- A complete description of the problem that includes the following questions:
- When did the problem first occur?
- Is the problem a one time failure or recurring?
- Was software or hardware maintenance applied?
- Did the failure occur while doing a specific task?
- Is the failure occurring in more than one address space?
- CICS product version, release, and maintenance level
- Operating system version, release, and maintenance level
- Related products version, and release levels
- Your valid contact phone number and email address
Component-specific information
After you collect general information, gather information that is specific to your component. From the following list, click the problem type or component to view a listing of specific documentation that the support team requires to diagnose your problem:
- Abend 878 or 80A
- CICS-MQ adapter or CICS-MQ bridge
- CICSPlex SM
- Coupling facility data table server
- DB2
- Dynamic Scripting for CICS® TS 5.1 or later
- Dynamic Scripting for CICS TS 4.2 or CICS TS 4.1
- Event processing
- Explorer errors or incorrect output
- Explorer UI not available or not stable
- File control (non-RLS)
- File control (RLS)
- IMS Database Control (DBCTL)
- information centers
- Intersystem communication (ISC)
- IP interconnectivity (IPIC)
- Java (pooled JVM)
- Java (JVM server)
- Logger
- Language environment (LE) abend
- LIBRARY management
- Mobile extensions
- Multiregion operation (MRO)
- Policies
- Program checks or abends
- SFR (Service Flow Runtime)
- Shared data tables (SDT)
- Short on storage (SOS)
- Storage violation
- Terminal hang
- Temporary storage servers
- Waits or loops
- Web services, XML, and JSON transformation
Troubleshooting data for Abend 878 or 80A problems
If the available MVS storage is not enough to satisfy an MVS GETMAIN request, you might have an abend 878 or an abend 80A problem in CICS TS. To determine the type of storage that is requested, see the explanation of system completion codes 878 and 80A. Abend 0F9 and 40D might occur when MVS does not have enough storage to create the dump for the abend 80A or 878.
- Required data:
- The MVS system dump that is taken when an out of storage condition occurs.
- If you do not have an MVS system dump, set the following SLIP traps to get a SLIP dump of the
abend 878 or
80A:
SLIP SET,COMP=878,JOBNAME=cicsjob,A=SVCD, SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN),ENDSLIP SET,COMP=80A,JOBNAME=cicsjob,A=SVCD, SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN),ENDwhere cicsjob is the name of your CICS job.
- If you see gradual storage growth of MVS storage in the CICS region (excluding CICS DSA
storage), you can take an MVS system dump of the region when storage usage is high. To take an MVS system dump, use the MVS command followed by a reply:
where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob), SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN)- dumpname
- Specifies the name of the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name of your CICS region.
- The MVS system trace that is included in the MVS system dump, unless your installation has changed the IBM-supplied system tracing options. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. The failing GETMAIN might affect an innocent party that is not the cause of the storage leak. The large MVS trace can help identify the real cause. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- The CICS message log that shows the abend 878 or 80A.
Troubleshooting data for CICS-MQ adapter or CICS-MQ bridge problems
- Required data:
- The CICS message log and the MVS system log (SYSLOG) that contain the messages leading up to the problem.
- The MVS system dump that is taken at the point of failure. If you receive an abend and system dumping is permitted for the dump code, a system dump is taken when the error is detected. Otherwise, you can request a dump by following the instructions in the first step under the MVS system dump section in Using dumps for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. For more information, see Using trace
for CICS problem determination on z/OS.
Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. Also turn on level 1 tracing for all CICS components if possible.
Non-exception trace entries for the MQ adapter are controlled by Resource Manager Interface (RI) and Resource Manager Adapter (RA) level 1 and level 2 tracing. Standard trace is often sufficient, but level 1-2 tracing for the RI and RA components might expedite problem resolution. Bridge facility (BR) standard trace is often sufficient for a CICS-MQ bridge problem, but level 1-2 tracing for the BR component might expedite problem resolution.
- Optional data:
- The WebSphere MQ for z/OS MSTR job log that contains the messages leading up to the problem. The job log is named xxxxMSTR where xxxx is the WebSphere MQ subsystem identifier (ssid).
- Locrec report from the SYS1.LOGREC data set. For more information, see Directions to get Logrec report.
- If you need a larger trace, consider using the CICS auxiliary trace or the GTF Trace with the MVS system dump. Use the same level of tracing as documented for CICS internal trace. For more information, see Using trace for CICS problem determination on z/OS.
Troubleshooting data for CICSPlex SM problems
- Required data:
- The complete CICS job output for the CMAS and the MAS or WUI server that experience the
problem.
For problems that are related to a CMAS, in addition to the output for the CMAS experiencing the problem, gather the output from all other CMASs in the CICSplex. The requirement for gathering documentation from all CMASs is due to the way CMAS to CMAS communications works.
For problems that are related to a MAS or WUI server region, gather the output for the MAS or WUI server and for the CMAS to which the MAS or WUI server connects.
This output must include the job log, JES messages, JCL, MSGUSR DD output, and the EYULOG DD output. Do not limit the size of the output by excluding anything. CPSM problems are often caused by events that occurred minutes, hours, or even days before you notice anything unusual.
- The complete MVS system log from the LPAR where the problem occurs. Do not shorten or cut down the size of the log in any way. If you attempt to cut out and send only what you think is relevant, you might inadvertently cut out critical information.
- The MVS system dump that is taken at the point of failure. For more
information, see Using dumps for CICS problem determination on
z/OS.
If your CICS region hangs or loops and you do not get a CICS or CPSM dump or message when problem occurs, dump the CPSM environment with the IEADMCCP command. This command is supplied with SYS1.SAMPLIB. For more information about dumping CPSM, see Using the MVS DUMP command. If you do not automatically get a dump but receive a CICS or CPSM message when the problem occurs, set an MVS SLIP trap to take an SVC dump of the appropriate address spaces and data spaces.
For example, if you always get an EYUXL0105E message in your MAS when a certain problem occurs, set your SLIP trap as follows:
SL SET,MSGID=EYUXL0105E,ID=CPSM,ACTION=SVCD,MATCHLIM=1, R y,JL=(cmas,mas,EYUX###),DSPNAME=('EYUX###'.*), R y,SDATA=(ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ),END- y
- This parameter specifies the reply identifier that displays on the console.
- cmas
- This parameter specifies the name of the CMAS that the MAS connects to.
- mas
- This parameter specifies the job name for the region you see the problem message.
- EYUX###
- In both the JL and DSPNAME parameters, this parameter
specifies the ESSS address space and must reflect the release of CICS that you are running:
- For CICS TS 4.1, use EYUX410 in both parameters.
- For CICS TS 4.2, use EYUX420 in both parameters.
- For CICS TS 5.1, use EYUX510 in both parameters.
- For CICS TS 5.2, use EYUX520 in both parameters.
- For CICS TS 5.3, use EYUX530 in both parameters.
- For CICS TS 5.4, use EYUX540 in both parameters.
Ensure that you dump the CMAS and EYUX### address spaces even if you see a problem only in one MAS.
If you take a dump by using the MVS console DUMP command, make sure that you take a dump of the ESSS and its data spaces too. The following example is a sample DUMP command that you can use:
DUMP COMM=(DumpTitle) R xxx,JOBNAME=(cmas,mas1,mas2,EYUX###),DSPNAME=('EYUX###'.*),CONT R xxx,SDATA=(ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ),ENDYou can add extra MAS names to the JOBNAME parameter as shown in the example above.
- Unformatted (raw) CICS auxiliary trace data sets (DFHAUXT and DFHBUXT) from the CMAS and the MAS
or WUI server. For more information, see Using trace
for CICS problem determination on z/OS.
For problems that are related to a CMAS, in addition to the unformatted (raw) AUX trace files for the CMAS, gather the unformatted (raw) AUX trace files from all other CMASs in the CICSplex. The requirement for gathering documentation from all CMASs is because of the way CMAS to CMAS communications works.
- When you have a routing failure and you want to know why, capture the CPSM WLM trace that is generated from the routing region. Also, include the dump and the auxiliary trace from both the CMAS and the routing region. For more information, see Using trace for CICS problem determination on z/OS.
- The complete CICS job output for the CMAS and the MAS or WUI server that experience the
problem.
Troubleshooting data for coupling facility data table server problems
- A DFHCFnnnn message, which indicates you have a CFDT problem. For more information, see CICS messages.
- Various types of waits that occur when CICS accesses a CFDT. See Investigating file control waits for resource types that start with CFDT and possible causes for the waits.
- Retained locks that arise from failures during updates to recoverable CF data table records.See Coupling facility data table retained locks for causes of the retained locks.
- Required data:
- The CICS message log and the MVS system log (SYSLOG).
- The SYSPRINT log for the CFDT server. Most of the same information is visible in the MVS system log, but the SYSPRINT log might contain statistics and some other informational messages, for example, the options that were used during start-up.
- If you receive a message and can re-create the problem, take a SLIP dump of the CICS region or
regions and of the CFDT list structure, at the time that the message occurs. Set the SLIP by using
the following MVS command and then re-create the problem:
where:SLIP SET,MSGID=DFHCFnnnn,ACTION=SVCD, STRLIST=(STRNAME=DFHCFLS_poolname,ACC=NOLIM, (LISTNUM=ALL,ADJ=DIRECTIO,EDATA=UNSERIALIZE)), JOBLIST=(cf-server,XCFAS,cicsjob), SDATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,SUM,SWA,TRT, COUPLE,WLM,GRSQ,LPA),MATCHLIM=1,ID=CIC1,END- DFHCFnnnn
- Specifies the message you receive.
- poolname
- Specifies your CFDT pool.
- cf-server
- Specifies your CFDT server.
- cicsjob
- Specifies the job name for your CICS region.
A dump of the pool is rarely needed by IBM, so you can exclude STRLIST=(...(...)) from the SLIP command if you are concerned about dump size.
If the pool structure becomes full unexpectedly, a dump with STRLIST=(...(...)) can help determine what fills up the pool. If you want to dump the pool only, see Dumping a CFDT list structure for instructions. You can also use the server command DISPLAY TABLE=name to determine the number of records in a specific table. For more information, see DISPLAY and PRINT command options.
- If there is a wait or hang, take an MVS system dump of the CICS region or regions and the CFDT
list structure as soon as you notice the hang or wait. For more
information, see Using dumps for CICS problem determination on
z/OS.Use the following MVS command followed by the reply to
capture the dump:
where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob), STRLIST=(STRNAME=DFHCFLS_poolname,ACC=NOLIM, (LISTNUM=ALL,ADJ=DIRECTIO,EDATA=UNSERIALIZE)), SDATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,SUM,SWA,TRT, COUPLE,WLM,GRSQ,LPA),END- dumpname
- Specifies the name you want to give to the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name for your CICS region.
- poolname
- Specifies your CFDT pool.
A dump of the pool is rarely needed by IBM, so you can exclude STRLIST=(...(...)) from the dump command if you are concerned about dump size.
If a request is made to wait because of a lock conflict, the CFDT server provides detailed information about what it is waiting for to the CICS wait exit. The server provides this information by completing fields in the parameter list structure for the original request and setting the "lock busy" response during the wait. If the CICS transaction later times out, this information is displayed in message DFHFC7130. There is no way of seeing this information before the transaction times out.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 for the FC component. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
If you receive a wait and can reproduce the problem, consider using the CICS auxiliary trace or the GTF trace with the MVS system dump. The MVS system dump alone is unlikely to show anything about system activity in the period that leads up to the wait because the trace table probably wraps before you have a chance to respond. But the CICS auxiliary trace and the GTF trace can provide you with enough trace. For more information, see Using trace for CICS problem determination on z/OS..
Troubleshooting data for DB2 problems
A CICS task might wait for the CICS DB2 task to complete its work in DB2. A CICS task might also wait for an open thread TCB to become available in CICS. See CICS DB2 waits for a more possible cause. You might also have a DB2 problem in CICS TS that is not related to a wait.
- Required data:
- For General DB2 problems, you must collect the following data:
- The MVS system dump of both CICS and DB2 that is taken when the CICS DB2 problem occurs. For more
information, see Using dumps for CICS problem determination on
z/OS. Enter this MVS command followed by the reply to capture
the dump:
where:DUMP COMM=(dumpname) R yy, JOBNAME= (*MASTER*,cicsjob,XCFAS,ssnmIRLM,ssnmMSTR,ssnmDBM1), SDATA= (RGN,CSA,SQA,LPA,LSQA,SWA,PSA,ALLNUC,XESDATA,TRT,GRSQ,SUM)- dumpname
- Contains both the CICS job name and DB2 subsystem.
- yy
- Specifies the number of the message.
- cicsjob
- Specifies the CICS region.
- ssnm
- Specifies the subsystem name of the DB2 members.
- The MVS system trace that is included in the MVS system dump when the MVS internal system trace is on. You can enter MVS TRACE commands TRACE ST,128,BR=OFF and TRACE MT,256K to ensure that the system trace is large enough. For more information, see Using trace for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Turn on level 1-2 tracing for RA and RI components. When possible, turn on level 1 tracing for all other CICS components. For more information, see Using trace for CICS problem determination on z/OS.
- The MVS system dump of both CICS and DB2 that is taken when the CICS DB2 problem occurs. For more
information, see Using dumps for CICS problem determination on
z/OS. Enter this MVS command followed by the reply to capture
the dump:
- For DB2 wait problems, in addition to the previous material, collect the following data:
- As detailed in DB2 Information APAR II14016 with regard to MVS SYSLOG, to enable reference to
the DSNV404I response output that results from entering the following DB2 commands at
the point of the wait:
If possible, enter the following commands:-DISPLAY THREAD(*) DETAIL-DISPLAY DATABASE(*) USE/LOCKS LIMIT(*) -DISPLAY UTILITY (*)
- As detailed in DB2 Information APAR II14016 with regard to MVS SYSLOG, to enable reference to
the DSNV404I response output that results from entering the following DB2 commands at
the point of the wait:
- Optional data:
- If possible, enter the following MVS Display
commands:
D A,ALL (or D A,ssnm* or D A,IRL*) D GRS,CONTENTION D OPDATA - The CICS message log.
- The CICS auxiliary trace or the GTF Trace up until the time when the problem occurs. This might be helpful when you want to identify the cause of the problem, and it is necessary to keep a record of events that lead to the problem over a long period of time.
- If possible, enter the following MVS Display
commands:
Troubleshooting data for Dynamic Scripting problems in CICS TS 5.1 or later
A CICS Dynamic Scripting problem in CICS occurs for many reasons. For information about the most common problems, see Troubleshooting and support for CICS Dynamic Scripting Feature Pack V2.0.
- Required data:
- Trace and dumps under the JVM server working
directory:
<workdir/javacore* [any java core dumps] <workdir/Snap* [any java snap dumps] <workdir/p8trace* [any p8 trace files] - If available, the JVM server standard out, standard error, and trace files. These files are
created in the JVM server working directory and named in the following
way:
where applid is the application identifier (APPLID) of the CICS region and JVMServer is the name of the JVMSERVER resource.applid.JVMServer.Dyyyymmdd.Thhmmss.dfhjvmout applid.JVMServer.Dyyyymmdd.Thhmmss.dfhjvmerr applid.JVMServer.Dyyyymmdd.Thhmmss.dfhjvmtrcNote: The JVM server working directory is defined in the WORK_DIR parameter in the JVM server profile. - If available, the CICS transaction dump and the CICS system dump.
- The contents of your Liberty server profile directory. This directory is specified in the WLP_OUTPUT_DIR parameter in the JVM server profile. By default, it is created in the JVM server working directory <workdir/$APPLID/$JVMSERVER/wlp/usr/servers/server_name. Replace the symbols with runtime values.
- Trace and dumps under the JVM server working
directory:
- Optional data:
- If IBM Support requests more detailed logging, consider the following options:
- Run CICS transaction CETR to activate tracing and logging information by setting AP and SJ component level to ALL. Rerun your applications. More logging and trace information will be dumped in dfhjvmtrc.
- Provide the contents in <workdir>/$APPLID/$JVMSERVER/*.xapictrace [any xapic trace files] if the SJ component level is set to 3 or higher.
- Provide the contents of your JVM profile directory. This directory is defined in the JVMPROFILEDIR system initialization parameter in your CICS region.
- Provide JESMSGLG and MSGUSR for messages that are issued by CICS.
Troubleshooting data for Dynamic Scripting problems in CICS TS 4.2 or CICS TS 4.1
A Dynamic Scripting problem in CICS occurs for many reasons. For information about the most common problems, see Troubleshooting and support for CICS Dynamic Scripting Feature Pack V1.1 and Troubleshooting and support for CICS Dynamic Scripting Feature Pack V1.0.
- Required data:
- If available, the JVM server standard out and standard error files. These files are created in
the JVM server working directory. For command-line interface (CLI) JVMs, this is the directory from
which you enter the command. For application JVMs, this is the root directory of the application.
The files are named applid.JVMServerID.dfhjvmerr and applid.JVMServerID.dfhjvmout, where applid is the application identifier (APPLID) of the CICS region and JVMServerID is the name of the JVMSERVER resource.
- If available, the JVM server standard out and standard error files. These files are created in
the JVM server working directory. For command-line interface (CLI) JVMs, this is the directory from
which you enter the command. For application JVMs, this is the root directory of the application.
- Optional data:
If IBM Support requests more detailed logging of the interaction between Dynamic Scripting and the CICS region, consider the following options:
- Enter export CICS_DEBUG=ON to turn on the environment variable CICS_DEBUG in the z/OS UNIX shell before you start the application or run the problematic CLI command. As a result, extra logging is sent to the console and the log files. For more information, see Troubleshooting the command-line interface.
- Provide the contents of your JVM profile directory. This directory is defined in the JVMPROFILEDIR system initialization parameter in your CICS region. For more information about the JVMPROFILEDIR parameter, see JVMPROFILEDIR for CICS TS 4.2 and JVMPROFILEDIR for CICS TS 4.1.
Troubleshooting data for Event processing problems
If you have Event processing problems, see Event processing problem determination procedures to collect documentation that is listed for your problem before you call IBM Support.
To compliment the documentation, use the following guidelines:
- Required data:
- Details of the event bindings and EP adapters that are used to capture and emit the events. These can be exported from the Event Binding Editor within CICS Explorer or Rational Developer for z Systems and sent to IBM as files.
- The Version and release of any IBM product that is used to consume the events, such as IBM Operational Decision Manager, IBM Business Monitor, or Cognos Real-Time Monitoring.
- Relevant messages from the CICS console log. These messages typically include DFHEPnnnn messages and DFHECnnnn messages. For more information, see CICS messages.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 2 tracing for the EC and EP components and level 1 tracing for all other CICS components. If this is not possible, turn on a minimum of level 1 tracing for the EC, EP, EI, SM, PG, and RL components. For more information, see Using trace for CICS problem determination on z/OS.
- The MVS system dump that is taken when the problem occurs. The way that you get the dump depends on whether or not you get a message, transaction abend, or no output when the problem occurs. For more information, see Using dumps for CICS problem determination on z/OS.
- If you are unable to get a system dump, get a CICS auxiliary trace that leads up to the error. If possible, turn on level 2 tracing for the EC and EP components, and level 1 tracing for all other CICS components. If this is not possible, turn on a minimum of level 1 tracing for the EC, EP, EI, SM, PG, and RL components. If the problem involves System Events, also turn on level 1 tracing for FC, XM, and RA.
- If you use EXEC CICS SIGNAL EVENT from within an application program, take a copy of the source code for the program that you added this code to in order to emit an event.
Troubleshooting data for errors or incorrect output of IBM Explorer for z/OS or CICS
If you receive incorrect output, an error in the log, or a pop-up error message from the CICS Explorer, IBM Explorer for z/OS, or any of the CICS Tools plug-ins, gather the following documentation before you call IBM Support.
- Required data:
- Use the Explorer-integrated capability to create a zip file that contains service data:
- Select from the Explorer menu.
- Select where to save the .zip file.
For more instructions, see Preparing error data for IBM service in IBM Knowledge Center for CICS Explorer.
- If you think the Explorer shows an incorrect output, create screen captures of the Explorer screen as image files (.bmp or .jpg). These images might provide useful information for IBM Support.
- When you get an error, gather the following additional data based on the type of the connection
you use:
- For a CICSPlex SM connection, the complete job log (including CICS TDQ output) of the WUI server
and the CMAS to which the WUI server connects.
If you receive a bad response from the CICS management client interface (CMCI) or the CICSPlex SM application programming interface (API), for example FAILED or EXCEPTION response, collect the unformatted auxiliary trace data sets from the WUI server and the CMAS to which the WUI server connects.
- For a single server (SMSS) version in a stand-alone CICS region, the complete job log (including TDQ output).
- For other types of connections, the required documentation varies. For example, for CICS Performance Analyzer, a copy of the input file that is used on the request is useful.
- For a CICSPlex SM connection, the complete job log (including CICS TDQ output) of the WUI server
and the CMAS to which the WUI server connects.
- Use the Explorer-integrated capability to create a zip file that contains service data:
Troubleshooting data for the z/OS or CICS Explorer when the user interface is not available or not usable
- Required data:
- If the UI is displayed, create screen captures of the Explorer screen as image files (.bmp or .jpg). These images might provide useful information for IBM Support.
- If the Explorer user interface is usable, use the Explorer-integrated capability to create a zip
file containing service data:
- Select from the Explorer menu.
- Select where to save the .zip file.
For more instructions, see Preparing error data for IBM service in IBM Knowledge Center for CICS Explorer.
- If the Explorer user interface is not usable, collect the logs and other Explorer working files
(such as settings) from one of the following directories. Use an archiving utility (a program that
creates .zip, .rar, .tar, or
.tar.gz files) to collect directory content.
- If you use the default on Windows, use the location C:\Documents and Settings\<username\.cicsexplorer.
- If you use the default on Linux, use the location /home/<username/.cicsexplorer.
- If you use an alternate workspace, which is described in the "Starting multiple instances of CICS Explorer" Help, use that location.
- Optional data:
- If the UI is up but unresponsive, follow these instructions to take a Java Thread Dump. The dump
might provide useful information for IBM Support.
- If you use the Explorer on Linux, perform the following procedures:
- Find out the process identifier of the CICS Explorer by using any system tool. For example,
enter the following command from a shell (command
line):
$ ps -ef | grep cicsexplorer - From a shell, invoke kill -QUIT <process_id>. A file named javacore.YYYYMMDD.<other>.txt will be created in the Explorer directory.
- Find out the process identifier of the CICS Explorer by using any system tool. For example,
enter the following command from a shell (command
line):
- If you use the Explorer on Windows, perform the following procedures:
- Ensure that you start the Explorer with the -consolelog parameter, so that the Explorer opens a shell (command line) window only for output besides a graphical UI window.
- If a dump is not taken, press Ctrl+Break on the command line to request a dump.
- From the
consolelog
window, you will see a message that includes the location and name of the dump:
Send the message to IBM.JVMDUMP010I Java dump written to C:\...\CICS Explorer\javacore.20090610.092732.6908.0001.txt.
- If you use the Explorer on Linux, perform the following procedures:
Troubleshooting data for file control (VSAM non-RLS) wait problems
Many problems in file control are caused by waits for CICS or VSAM resources.
- Required data:
- The CICS message log and the MVS system log (SYSLOG).
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 for the FC component. For more information, see Using trace for CICS problem determination on z/OS.
- An MVS system dump of the CICS region that is taken as soon as you notice the hang or wait. To take an MVS system dump, use the MVS command followed by a reply:For more information, see Using dumps for CICS problem determination on z/OS.
where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob), SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN)- dumpname
- Specifies the name of the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name of your CICS region.
- Optional data:
- If feasible, save the data set by using IDCAMS ALTER NEWNAME to rename the data set and save it in place (note that the example in the link has a DEFINE before the ALTER). If this is not feasible, run IDCAMS EXAMINE ITEST DTEST against the data set and any alternate indexes that are associated with the data set before you continue. For more information, see z/OS DFSMS Access Method Services Commands.
- If you can reproduce the problem, consider using the CICS auxiliary trace or the GTF Trace with the MVS system dump. The MVS system dump alone is unlikely to show anything about system activity in the period that leads up to the wait because the trace table probably wraps before you have a chance to respond. For more information, see Using trace for CICS problem determination on z/OS..
Troubleshooting data for file control (VSAM RLS) problems
- Required data:
- The CICS message log for all CICS regions that use RLS and the system log (SYSLOG) for all MVS images that use RLS.
- The CICS internal trace that is included in the MVS system dump and the SLIP dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 for the FC component. For more information, see Using trace for CICS problem determination on z/OS.
- If you experience hang situations with CICS and RLS, take MVS system dumps of SMSVSAM address
spaces and dataspaces, Master address spaces, XCF address spaces and dataspaces, and CICS address
spaces. For more
information, see Using dumps for CICS problem determination on
z/OS. To capture a dump on each MVS system in
your sysplex, use the following MVS command with the REMOTE option when the hang occurs:
where:DUMP COMM=(dumpname) R nn,JOBNAME=(SMSVSAM, XCFAS,*MASTER*,cicsjob*),CONT R nn,DSPNAME=('XCFAS'.*,'SMSVSAM’.*),CONT R nn,SDATA=(COUPLE,ALLNUC,LPA,LSQA,PSA,RGN,SQA,SUM, TRT,CSA,GRSQ,XESDATA),CONT R nn,REMOTE=(SYSLIST=*('XCFAS','SMSVSAM‘,’cicsjob*’, ’*MASTER*’),DSPNAME,SDATA),END- dumpname
- Specifies the name you want to give to the dump.
- nn
- Specifies the unique reply identifier for each reply.
- cicsjob*
- Specifies the first part of the job name for your CICS regions. The asterisk is a wildcard that
instructs MVS to dump all regions that start with cicsjob.Note: The incident token in message IEA611I or IEA911I is the same on all MVS systems.
To avoid replying with so many dump options, you can specify the suffixes of one or more IEADMCxx parmlib members. For more information, see IEADMCxx (DUMP command parmlib)
- Customize an IEADMCxx member to include the dump command. The following sample is for CICS and
RLS
problems:
/* ** Start of Specifications for IEADMCCR ************************ */ /* */ /* $MAC(IEADMCCR) COMP(SCDMP) PROD(HBB7705): Dump CICS and RLS */ /* */ /* PROPRIETARY STATEMENT: */ /* */ /* LICENSED MATERIALS - PROPERTY OF IBM */ /* THIS MACRO IS "RESTRICTED MATERIALS OF IBM" */ /* 5694-A01 (C) COPYRIGHT IBM CORP. 2001 */ /* */ /* STATUS: HBB7705 */ /* */ /* External classification= none */ /* End of external classification: */ /* */ /* DUMP command Parmlib member */ /* */ /* Function: Provides DUMP command options to dump */ /* Logger, RLS and CICS. */ /* */ /* */ /* To use: Copy this member to Parmlib. */ /* Modify cicsjob to be the first part of the jobname for */ /* your CICS regions. */ /* */ /* To execute: Specify the following DUMP command: */ /* */ /* DUMP PARMLIB=xx */ /* - or - */ /* DUMP TITLE=(dump_title_text),PARMLIB=xx */ /* */ /* If a dump title is specified on the DUMP command using the */ /* TITLE option, the specified text will be used instead of the */ /* TITLE line in the parmlib member. */ /* * /* ================================================================ * /* * /* Change activity: * /* * /* 10/24/05 created as sample for CICS MustGather * /* * /* ** End of Specifications for IEADMCCR ************************** * TITLE=(DUMP RLS AND CICS), JOBNAME=(SMSVSAM, XCFAS,*MASTER*,cicsjob*), DSPNAME=('XCFAS'.*,'SMSVSAM’.*), REMOTE=(SYSLIST=*('XCFAS','SMSVSAM‘,’cicsjob*’,’*MASTER*’), DSPNAME,SDATA) SDATA=(COUPLE,ALLNUC,LPA,LSQA,PSA,RGN,SQA,SUM,TRT,CSA,GRSQ, XESDATA) - Copy the member into SYS1.PARMLIB.
- Enter DUMP TITLE=(dump-name),PARMLIB=xx to request a dump by using the dump command in IEADMCxx. To request multiple parmlib members, specify PARMLIB=(xx,yy).
- If you receive RLS messages such as DFHFC0987, take a SLIP dump. For more information, see Using trace
for CICS problem determination on z/OS. Use the following SLIP command to capture a dump of
the SMSVSAM address space and the CICS address spaces for message DFHFC0987:
where:SLIP SET,MSGID=DFHFC0987,A=SVCD, JOBLIST=(SMSVSAM,cicsjob*), DSPNAME =('SMSVSAM'.*), REMOTE=(JOBLIST,DSPNAME),ID=id,END- DFHFC0987
- Specifies the message you receive.
- id
- Specifies the dump identifier.
- cicsjob*
- Specifies the first part of the job name for your CICS regions. The asterisk is a wildcard that
instructs MVS to dump all regions that start with cicsjob.Note: The MSGID parameter is valid for z/OS V1.2 or later. The incident token in message IEA611I or IEA911I is the same on all MVS systems.
- Optional data:
If you can reproduce the problem, consider using the CICS auxiliary trace or the GTF Trace with the MVS system dump. The MVS system dump alone is unlikely to show anything about system activity in the period that leads up to the wait because the trace table probably wraps before you have a chance to respond. But the CICS auxiliary trace and the GTF trace can provide you with enough trace.
Troubleshooting data for IMS problems
IMS problems might occur because a CICS task waits for the CICS IMS task to complete its work in IMS or because a CICS task waits for an open thread TCB to become available in CICS.
- Required data:
- If possible, enter the following IMS command from the console, SDSF, or by using the CICS CDBM
transaction:
/DISPLAY ACTIVE - The MVS system dump of both CICS and IMS that is taken when the IMS wait occurs. For more
information, see Using dumps for CICS problem determination on
z/OS. Enter this MVS command followed by the reply to capture
the dump:
where:DUMP COMM=(dumpname) R yy, JOBNAME= (*MASTER*,cicsjob,XCFAS,ssnmIRLM,ssnmCTRL,ssnmDBRC,ssnmDLI), SDATA= (RGN,CSA,SQA,LPA,LSQA,SWA,PSA,ALLNUC,XESDATA,TRT,GRSQ,SUM)- dumpname
- Contains both the CICS job name and the IMS subsystem.
- yy
- Specifies the number of the message.
- cicsjob
- Specifies the CICS region.
- ssnm
- Specifies the subsystem name of the IMS members.
- The MVS system trace that is included in the MVS system dump when MVS internal system trace is on. You can enter MVS TRACE command TRACE ST,128,BR=OFF and TRACE MT,256K to ensure that the system trace is large enough. For more information, see Using trace for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Turn on level 1-2 tracing for the RI component. When possible, turn on level 1 tracing for all other CICS components. For more information, see Using trace for CICS problem determination on z/OS.
- If possible, enter the following IMS command from the console, SDSF, or by using the CICS CDBM
transaction:
- Optional data:
- If possible, enter the following MVS Display
commands:
D A,ALL (or D A,ssnm* or D A,IRL*) D GRS,CONTENTION D OPDATAFor more information, see Display command.
- The CICS message log.
- The CICS auxiliary trace or the GTF Trace up until the time when the problem occurs. This might be helpful when you want to identify the cause of the problem, and it is necessary to keep a record of events that lead to the problem over a long period of time.
- If possible, enter the following MVS Display
commands:
Troubleshooting data for IBM Knowledge Center problems
- Required data:
- A brief description of what you attempt to do when you experience the problem.
- Messages or errors that you receive, for example, on screen or in the /logs directory of the locally-installed IBM Knowledge Center.
- The way in which you attempt to use the CICS TS documentation in IBM Knowledge Center:
- On the internet, accessing IBM Knowledge Center for CICS TS.
- Locally, using one of the downloadable versions.
- On z/OS, using the IBM KC for z/OS.
- The browser and browser version that you use to access the documentation.
- If you use a local version of the IBM KC, the operating system of the machine where the KC is installed. If you installed the KC as a WAR file, the version of Liberty where the KC is installed.
Troubleshooting data for intersystem communication (ISC) problems
- Required data:
- The CICS message log for any DFHZCnnnn message. If you have problems with connecting or have a failure when you run a transaction, the console and CSNE transient data queue usually have messages. If the other region is also CICS, make sure that you have the message log from that region too. For more information on DFHZCnnnn messages, see CICS messages.
- The MVS system dump that is taken at the point of failure. For more information, see Using dumps for CICS problem determination on z/OS. If you cannot re-create the problem, take a dump by using the MVS dump command, or add the pertinent DFHZCxxxx message to the CICS dump table. If you cannot determine which side of the ISC connection causes the problem, dump both regions.
- The CICS internal trace, which runs within CICS regions that participate in the conversation. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Also, turn on standard level 1 tracing for all components and level 1-2 for the TC component. Use the CETR transaction to turn on internal trace, set internal trace size, and set component trace levels. For more information, see Using trace for CICS problem determination on z/OS..
- Optional data:
- If you can re-create the problem, the CICS auxiliary trace might be useful. Ensure that you run the trace on all the CICS regions that are involved in the ISC problem.
- If you want to see all of the trace entries in one file, you can use the GTF Trace. The GTF Trace is useful when the CICS regions are on different MVS regions. You can write the trace to a GTF trace data set on each MVS, and then merge the files to see the flow of trace in the connected regions. If the CICS regions are in a single MVS, you can write all the trace to a single GTF trace data set.
- The CICS transaction dump that is taken at the point of failure.
- The z/OS Communications Server exit tracing (CICS VTAM exit tracing) that is written to the GTF trace data set. To get this tracing, GTF tracing must be active. Each VTAM exit in CICS contains a trace point that captures the relevant terminal control information. This tracing is often used instead of a VTAM buffer trace.
- The z/OS Communications Server (VTAM) buffer trace that allows you to look at all the data that is passed across a VTAM connection. You can use this trace if you are concerned with the data stream being passed between the regions.
- Get a SLIP dump if the java system does not automatically create a dump or if level 2 instructs you to dump on a specific condition.
Troubleshooting data for IP interconnectivity (IPIC) problems
- Required data:
- The CICS message log that includes DFHISnnnn messages. If you have problems with connecting or have a failure when you run a transaction, the console and CISO transient data queue usually have messages. If the other region is also CICS, make sure that you have the message log from that region too. For more information, see CICS messages.
- The MVS system dump that is taken at the point of failure. If you can re-create the problem, take a dump or add the pertinent DFHISxxxx message to the dump table. If you cannot determine which side of the connection causes the problem, dump both regions. For more information, see Using dumps for CICS problem determination on z/OS.
- The CICS internal trace, which runs within CICS regions that participate in the conversation. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Also, turn on standard level 1 tracing for all components and level 1-2 for the IS, WB, and SO components. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- If you can re-create the problem, the CICS auxiliary trace might be useful. Ensure that you run the trace on all the CICS regions that are involved in the IPIC problem.
- If you want to see all of the trace entries in one file, you can use the GTF Trace. The GTF Trace is useful when the CICS regions are on different MVS regions. You can write the trace to a GTF trace data set on each MVS, and then merge the files to see the flow of trace in the connected regions. If the CICS regions are in a single MVS, you can write all the trace to a single GTF trace data set.
- The CICS transaction dump that is taken at the point of failure.
- Get a SLIP dump if the java system does not automatically create a dump or if level 2 instructs you to dump on a specific condition.
Troubleshooting data for pooled JVM (Java) problems in CICS TS 4.1 or CICS TS 4.2
A pooled Java Virtual Machine (JVM) problem in CICS occurs for different reasons. Before you gather the following documentation and call IBM Support, run the Hello World and Hello CICS World sample programs to verify that Java has been installed and set up correctly in your CICS region. For more information, see Checking your pooled JVM setup with the examples.
- Required data:
- The CICS message log that contains DFHSJxxxx, DFHIIxxxx, and DFHEJxxxx messages. You can also cut and paste the messages into your problem record. For more information, see CICS messages.
- The JVM profile and the properties file if applicable. Properties files are optional.
- If you use the Shared Class Cache, collect the DFHJVMCD profile.
- The output from specifying the PRINT_JVM_OPTIONS=YES option in the JVM profile. When you specify this option, all the options passed to the JVM at startup, including the contents of the class paths, are printed to SYSPRINT. The output is written every time a JVM is started with this option in its profile, so add the option to the appropriate JVM profile, wait for a JVM to be started with the profile (or enter the PERFORM JVMPOOL command to manually start a JVM with the profile), and then immediately remove the option from the profile.
- If a Java exception occurs, collect the standard error (stderr) file that contains the Java stack trace. The output from Java applications is written to an HFS file that is named by the STDERR option in the JVM profile for the JVM. The output file is located in the directory that is named by the WORK_DIR option in the JVM profile.
- If you have hang or loop problems with CICS and Java, take an MVS system
dump of the CICS region and OMVS ASID. For more
information, see Using dumps for CICS problem determination on
z/OS. Use the
following MVS command to capture the dump:
Where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob),CONT R yy,DSPNAME=('OMVS'.*),ASID=(omvs),CONT R yy,SDATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,ALLNUC,GRSQ)- dumpname
- Specifies the name you want to give the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name of your CICS region.
- omvs
- Specifies the ASID of OMVS. To see your OMVS ASID in the BPXO042I message, enter the MVS command D OMVS. For more information, see BPX messages.
If an abend occurs because of a failure in the JVM or in the Language Environment (LE) enclave that the JVM runs, an MVS system dump of CICS taken at the point of the abend is of limited use. In this case, the MVS system dump is unlikely to contain the JVM storage. This is because that the LE enclave might have ended by the time CICS notices the failure and takes the dump. However, it is useful to provide any of the accompanying JAVADUMP, SYSDUMP (TDUMP), and CEEDUMP dumps with the MVS system dump.
You can control the behavior of JAVADUMP, SYSDUMP, and CEEDUMP dumps by using the JAVA_DUMP_OPTS environment variables in your JVM profile. By default CICS TS sets JAVA_DUMP_OPTS to ONANYSIGNAL(JAVADUMP,CEEDUMP,SYSDUMP),ONINTERRUPT(NONE), which provides the desired dumps.
- JAVADUMP is a formatted dump that is taken by the JVM. The dump summarizes the internal state of the JVM when it crashes unexpectedly. The output file is located in the directory named by the WORK_DIR option in the JVM profile.
- SYSDUMP (TDUMP) is a standard SVC dump that is taken by the JVM when the JVM crashes. The
following example shows messages that are written to STDERR when the JVM requests a
SYSDUMP.
JVMDG217: Dump Handler is Processing Signal 4 - Please Wait. JVMHP002: JVM requesting System Transaction Dump JVMHP012: System Transaction Dump written to PLOWMAN.JVM.TDUMP.COMKZCES.D050419.T231519 JVMDG303: JVM Requesting Java core file JVMDG304: Java core file written to /u/plowman/cics630/workdir/JAVADUMP.20050419.231606.83886118.txt JVMDG215: Dump Handler has Processed Exception Signal 4.You can use MVS Interactive Problem Control System (IPCS) to format the dump. For more information, see Using IPCS to format and analyze CICS dumps: Overview
To override the default dump name, use the environment variable JAVA_DUMP_TDUMP_PATTERN in the JVM profile. For more information, see APAR II13292 and Java Diagnostics Guide.
- CEEDUMP is a formatted application level dump that is taken by the LE service CEE3DMP after any
SYSDUMP processing. The following example shows the LE
traceback information in CEEDUMP. For more
information, see Using trace for CICS problem determination on
z/OS.
TRACEBACK: DSA ADDR PROGRAM UNIT PU ADDR PU OFFSET ENTRY E ADDR E OFFSET STATEMENT LOAD MOD SERVICE STATUS 2E63E178 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/threads_utils.c 2EB098A0 +000006AA ThreadUtils_CoreDump 2EB098A0 +000006AA 1662 *PATHNAM h020723 CALL 2E63E068 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c 2EAEC108 +000004B0 userSignalHandler 2EAEC108 +000004B0 376 *PATHNAM h020723 CALL 2E63DFB8 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c 2EAEC670 +000000B6 intrDispatch 2EAEC670 +000000B6 642 *PATHNAM h020723 CALL 2E63DF00 2DED9938 +0000001A @@GETFN 2DED9890 +000000C2 CEEEV003 CALL 2E63DE80 CEEVASTK 2DCCC420 +00000126 CEEVASTK 2DCCC420 +00000126 CEEPLPKA CALL 2E32BCC0 2DFF2B80 +00000736 __zerros 2DFF2B80 +00000736 CEEEV003 DRIVER9 CALL 2E329128 CEEHDSP 2DC16848 +00000C0C CEEHDSP 2DC16848 +00000C0C CEEPLPKA CALL 2E3286A8 /u/sovbld/hm131s/hm131s-20020723/src/zip/sov/ZipEntry.c 31F6EB38 +00000252 Java_java_util_zip_ZipEntry_initFields 31F6EB38 +00000252 99 *PATHNAM h020723 EXCEPTION 2E3285C8 java/util/zip/ZipEntry.java 32CEE95C +000000E4 java/util/zip/ZipEntry.initFields(J)V 32CEE95C +000000E4 0 CALL
- The Java version that you run. Determine your Java version in one of the following ways:
- See message DFHSJ0901I that is written at JVM startup when you specify the following option in
the JVM profile. For more information, see CICS
messages
-Dcom.ibm.cics.ShowJavaVersion=true - From OMVS, enter command java -version.
- Take the Java version from the JAVADUMP if one is available.
The following table lists CICS releases and supported Java releases. Ensure that you use a supported release of Java.
CICS TS release Supported Java (SDK) release 4.2 6.0.1 (64-bit) 4.1 6 or 6.0.1 (31-bit) Go to Service for Java on z/OS for a list of service summaries for different Java version. The summaries contain tables that list the SDK service levels and their corresponding PTFs. Install the latest service level applicable to your CICS TS to take advantage of improved service, performance, and stability.
- See message DFHSJ0901I that is written at JVM startup when you specify the following option in
the JVM profile. For more information, see CICS
messages
- Optional data:
- If the original cause of the failure happens much earlier than the
exception, take a SYSDUMP (TDUMP) that is taken when you receive an earlier Java message. Specify
the following options in the JVM profile to capture a dump for a specific Java
message:
IBM_XE_COE_NAME=AnyValidJava/message JAVA_DUMP_OPTS=ONEXCEPTION(SYSDUMP),ONERROR(SYSDUMP) JAVA_DUMP_TDUMP_PATTERN=\ "DUMP.JVM.TDUMP.&APPLID;.&JVM_NUM;.LATEST"Replace AnyValidJava/message with any valid Java message, for example, java/lang/OutOfMemoryError.
To continue a parameter on a second line, end the first line with a backslash and a blank continuation character (\ ) , as in JAVA_DUMP_TDUMP_PATTERN=\ .
- The JVM trace that is written to the CICS internal trace and is included in the MVS system dump
when tracing is active. Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. When
possible, turn on level 1 tracing for all CICS components. For more information, see Defining and activating traces for pooled JVMs. For more information, see Using trace
for CICS problem determination on z/OS.Note: The JVM trace can also be written to the CICS auxiliary trace or the GTF Trace.
- If the original cause of the failure happens much earlier than the
exception, take a SYSDUMP (TDUMP) that is taken when you receive an earlier Java message. Specify
the following options in the JVM profile to capture a dump for a specific Java
message:
Troubleshooting data for JVM server (Java) problems in CICS TS 4.2 or later
- Required data:
- The CICS job log that is sent in tersed binary. This contains various CICS messages that are relevant to the JVM server and any entries in the SYSOUT and SYSnnnn DD cards.
- The JVM profile contents. Ensure that you identify the JVM profile correctly. The JVMSERVER
resource defines the file name and the location of the JVM profile. You can use the EXEC
CICS INQUIRE JVMSERVER or the CEMT INQUIRE JVMSERVER command to inquire
on the JVMSERVER resource.
To find the file name of the JVM profile:
Look at the JVMPROFILE attribute in the JVMSERVER definition. For CICS TS 5.2 or later, the file name has an extension of .jvmprofile. For more information, see JVMSERVER attributes.
To locate the directory that contains the JVM profile:- For CICS TS 4.2, this is the directory on zFS that is specified in your JVMPROFILEDIR SIT parameter. For more information, see JVMPROFILEDIR in IBM Knowledge Center for CICS TS 4.2.
- For CICS TS 5.1 or later, this is the directory on zFS that is specified by the
PROFILEDIR attribute in the JVMSERVER definition.
- For a JVM server that is defined in a local CICS region, the directory is the one specified by the JVMPROFILEDIR SIT parameter for the region. For more information, see JVMPROFILEDIR.
- For a JVM server that is defined in a CICS bundle, this is the CICS bundle subdirectory where the JVM profile is stored.
- The STDERR, STDOUT, and JVMTRACEfiles. Refer to the JVM server options in the JVM profile to determine their location. For more information, see JVM server options. By default, they are located in the WORK_DIR directory on zFS, where WORK_DIR is specified in your JVM profile. The STDERR file contains the Java stack trace when a Java exception occurs. The STDOUT file contains user application written information. And the JVMTRACE file contains any CICS trace information that is produced by the JVM server.
- JVM trace that is written to the CICS internal trace and is included in the MVS system dump when
tracing is active. Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. Turn on level
1-2 tracing for the AP and EI components, and level to 1-5 tracing for the SJ component. Set levels
of tracing to be at least 1 for all other component. For more information, see Using trace
for CICS problem determination on z/OS.Note: Besides the trace that is written to the internal trace table, trace entries for each JVM server are written to a unique file on zFS. For more information, see Activating and managing tracing for JVM servers.
- If the JVM server profile defines a Liberty server, for example, by specifying the
WLP_INSTALL_DIR parameter, create a dump of the Liberty server by using the
CICS generated wlpenv script and the Liberty server dump command, for example, ./wlpenv
server dump.
For more information about the wlpenv script, see Troubleshooting Liberty JVM servers and Java web applications. The dump contains both Liberty configuration files, such as server.xml and installedApps.xml, and output files, such as messages.log, trace.log, and ffdc.
- If you have hang or loop problems with CICS and Java, take an MVS system
dump of the CICS region and OMVS ASID. For more
information, see Using dumps for CICS problem determination on
z/OS. Use the
following MVS command to capture the dump:
Where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob),CONT R yy,DSPNAME=('OMVS'.*),ASID=(omvs),CONT R yy,SDATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,ALLNUC,GRSQ)- dumpname
- Specifies the name you want to give the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name of your CICS region.
- omvs
- Specifies the ASID of OMVS. To see your OMVS ASID in the BPXO042I message, enter the MVS command D OMVS. For more information, see BPX messages.
If an abend occurs because of a failure in the JVM or in the Language Environment (LE) enclave that the JVM runs, an MVS system dump of CICS taken at the point of the abend is of limited use. In this case, the MVS system dump is unlikely to contain the JVM storage. This is because that the LE enclave might have ended by the time CICS notices the failure and takes the dump. However, it is useful to provide any of the accompanying JAVADUMP, SYSDUMP (TDUMP), and CEEDUMP dumps with the MVS system dump.
You can control the behavior of JAVADUMP, SYSDUMP, and CEEDUMP dumps by specifying the-Xdump option in your JVM profile. For more information, see -Xdump. For example, to collect information on a java.lang.OutOfMemoryError exception, set the -Xdump option as follows:-Xdump:java+heap+system:events=systhrow,filter=java/lang/OutOfMemoryError- JAVADUMP is a formatted dump that is taken by the JVM. The dump summarizes the internal state of the JVM when it crashes unexpectedly. The output file is located in the directory named by the WORK_DIR option in the JVM profile.
- SYSDUMP (TDUMP) is a standard SVC dump that is taken by the JVM when the JVM crashes. The
following example shows messages that are written to STDERR when the JVM requests a
SYSDUMP.
JVMDG217: Dump Handler is Processing Signal 4 - Please Wait. JVMHP002: JVM requesting System Transaction Dump JVMHP012: System Transaction Dump written to PLOWMAN.JVM.TDUMP.COMKZCES.D050419.T231519 JVMDG303: JVM Requesting Java core file JVMDG304: Java core file written to /u/plowman/cics630/workdir/JAVADUMP.20050419.231606.83886118.txt JVMDG215: Dump Handler has Processed Exception Signal 4.You can use MVS Interactive Problem Control System (IPCS) to format the dump. For more information, see Using IPCS to format and analyze CICS dumps: Overview
To override the default dump name, use the environment variable JAVA_DUMP_TDUMP_PATTERN in the JVM profile. For more information, see APAR II13292 and Java Diagnostics Guide.
- CEEDUMP is a formatted application level dump that is taken by the LE service CEE3DMP after any
SYSDUMP processing. The following example shows the LE
traceback information in CEEDUMP. For more
information, see Using trace for CICS problem determination on
z/OS.
TRACEBACK: DSA ADDR PROGRAM UNIT PU ADDR PU OFFSET ENTRY E ADDR E OFFSET STATEMENT LOAD MOD SERVICE STATUS 2E63E178 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/threads_utils.c 2EB098A0 +000006AA ThreadUtils_CoreDump 2EB098A0 +000006AA 1662 *PATHNAM h020723 CALL 2E63E068 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c 2EAEC108 +000004B0 userSignalHandler 2EAEC108 +000004B0 376 *PATHNAM h020723 CALL 2E63DFB8 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c 2EAEC670 +000000B6 intrDispatch 2EAEC670 +000000B6 642 *PATHNAM h020723 CALL 2E63DF00 2DED9938 +0000001A @@GETFN 2DED9890 +000000C2 CEEEV003 CALL 2E63DE80 CEEVASTK 2DCCC420 +00000126 CEEVASTK 2DCCC420 +00000126 CEEPLPKA CALL 2E32BCC0 2DFF2B80 +00000736 __zerros 2DFF2B80 +00000736 CEEEV003 DRIVER9 CALL 2E329128 CEEHDSP 2DC16848 +00000C0C CEEHDSP 2DC16848 +00000C0C CEEPLPKA CALL 2E3286A8 /u/sovbld/hm131s/hm131s-20020723/src/zip/sov/ZipEntry.c 31F6EB38 +00000252 Java_java_util_zip_ZipEntry_initFields 31F6EB38 +00000252 99 *PATHNAM h020723 EXCEPTION 2E3285C8 java/util/zip/ZipEntry.java 32CEE95C +000000E4 java/util/zip/ZipEntry.initFields(J)V 32CEE95C +000000E4 0 CALL
- The Java version that you run. Determine your Java version in one of the following ways:
- See message DFHSJ0901I that is written at JVM startup when you specify either of the following
options in the JVM profile, based on your CICS versions:
- For CICS TS 5.1 or earlier:
-Dcom.ibm.cics.ShowJavaVersion=true - For CICS TS 5.2, CICS TS 5.3, and CICS TS 5.4:
DISPLAY_JAVA_VERSION=YES
- For CICS TS 5.1 or earlier:
- From OMVS, enter command java -version.
- Take the Java version from the JAVADUMP if one is available.
The following table lists CICS releases and supported Java releases. Ensure that you use a supported release of Java.
CICS TS release Supported Java (SDK) release 5.3 7, 7.1 or 8 (64-bit) 5.2 7, 7.1 or 8 (64-bit) 5.1 7, 7.1 or 8 (64-bit) 4.2 6.0.1 (64-bit) 4.1 6 or 6.0.1 (31-bit) Go to Service for Java on z/OS for a list of service summaries for different Java version. The summaries contain tables that list the SDK service levels and their corresponding PTFs. Install the latest service level applicable to your CICS TS to take advantage of improved service, performance, and stability.
- See message DFHSJ0901I that is written at JVM startup when you specify either of the following
options in the JVM profile, based on your CICS versions:
- Optional data:
- The output from specifying the PRINT_JVM_OPTIONS=YES option in the JVM profile. When this option is specified, all the options that are passed to the JVM at startup, including the contents of the class paths, are printed to SYSPRINT. After you obtain this information, immediately remove the option from the profile.
- If the original cause of the failure happens much earlier than the
exception, take a SYSDUMP (TDUMP) that is taken when you receive an earlier Java message. Specify
the following options in the JVM profile to capture a dump for a specific Java
message:
IBM_XE_COE_NAME=AnyValidJava/message JAVA_DUMP_OPTS=ONEXCEPTION(SYSDUMP),ONERROR(SYSDUMP) JAVA_DUMP_TDUMP_PATTERN=\ "DUMP.JVM.TDUMP.&APPLID;.&JVM_NUM;.LATEST"Replace AnyValidJava/message with any valid Java message, for example, java/lang/OutOfMemoryError.
To continue a parameter on a second line, end the first line with a backslash and a blank continuation character (\ ) , as in JAVA_DUMP_TDUMP_PATTERN=\ .
Troubleshooting data for logger problems
A log stream failure or a failure within the MVS logger function occurs for many reasons. For more information, see Some conditions that cause CICS log manager error messages.
- Required data:
- The CICS message log back to the startup of the CICS region. The CICS messages contain the MVS logger return and reason codes for the failure. The messages also include log stream names, attributes, and other helpful information. For more information, see z/OS MVS Programming: Assembler Services Reference (IAR-XCT).
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 tracing for the LG and RM components. If this is not possible, turn on a minimum of level 1-2 or 1 for the LG component. For more information, see Using trace for CICS problem determination on z/OS.
- The MVS system dump that is taken automatically by CICS when a failure is detected. A dump of the MVS logger address space is produced automatically when an MVS IXGBRWSE or IXGDELET request fails because the MVS logger cannot find a specific log stream block identifier. For more information, see Using dumps for CICS problem determination on z/OS.
- If a dump of the XCF and MVS logger address spaces is not taken and you suspect that the problem
is a result of something within MVS logger, take a SLIP dump of the MVS system logger address space
and the coupling facility structure by setting a SLIP trap and performing a diagnostic run of CICS.
For more information, see Setting a SLIP trap. For instructions on
how to perform a diagnostic run of CICS, see Dealing with a corrupt system
log.
Alternatively, use the MVS dump command to take a dump before you perform an initial start. See Obtaining MVS logger and coupling facility dumps for the dump command. To avoid replying with so many dump options, you can specify the suffixes of one or more IEADMCxx parmlib members by following these instructions. For more information, see IEADMCxx.
- Customize an IEADMCxx member to include the dump command. The following sample is for CICS and
Logger problems:
/* ** Start of Specifications for IEADMCCL ************************ */ /* */ /* Dump CICS and Logger */ /* */ /* PROPRIETARY STATEMENT: */ /* */ /* LICENSED MATERIALS - PROPERTY OF IBM */ /* THIS MACRO IS "RESTRICTED MATERIALS OF IBM" */ /* 5694-A01 (C) COPYRIGHT IBM CORP. 2001 */ /* */ /* STATUS: HBB7705 */ /* */ /* External classification= none */ /* End of external classification: */ /* */ /* DUMP command Parmlib member */ /* */ /* Function: Provides DUMP command options to dump */ /* Logger and CICS. */ /* */ /* */ /* To use: Copy this member to Parmlib. */ /* Modify cics_jobname and structure to include your */ /* CICS regions and structure name. */ /* */ /* To execute: Specify the following DUMP command: */ /* */ /* DUMP PARMLIB=xx */ /* - or - */ /* DUMP TITLE=(dump_title_text),PARMLIB=xx */ /* */ /* If a dump title is specified on the DUMP command using the */ /* TITLE option, the specified text will be used instead of the */ /* TITLE line in the parmlib member. */ /* * /* ================================================================ * /* * /* Change activity: * /* * /* 10/31/05 created as sample for CICS MustGather for Logger * /* 01/24/10 verified that the command has not changed /* * /* ** End of Specifications for IEADMCCL ************************** * TITLE=(DUMP LOGGER AND CICS) JOBNAME=(IXGLOGR,XCFAS,cics_jobname), DSPNAME=('IXGLOGR'.*,'XCFAS'.*), STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM), REMOTE=(SYSLIST=*('XCFAS','IXGLOGR'),DSPNAME,SDATA), SDATA=(COUPLE,ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA) - Copy the member into SYS1.PARMLIB.
- Enter DUMP TITLE=(dump-name),PARMLIB=xx to request a dump by using the dump command in IEADMCxx. Request multiple parmlib members by specifying PARMLIB=(xx,yy).
- Customize an IEADMCxx member to include the dump command. The following sample is for CICS and
Logger problems:
- The output from the DFHJUP utility. Use this utility to print or copy the contents of the failing log stream before you restart the CICS region or reformat the log stream. CICS tries to trim DFHLOG and DFHSHUNT at each keypoint and log data might be lost as a result of the trim. Therefore, it is important to run DFHJUP right after a problem occurs. For more information, see Using DFHJUP to read log streams.
- Optional data:
- The output from the IXCMIAPU utility with LIST and DETAIL=YES specified. For sample JCL, see Checking log stream status.
- The output from entering the MVS commands D LOGGER and D LOGGER,L. For more information, see Displaying the system logger and its log streams
- A copy of the offload data sets that are taken by using IDCAMS REPRO.
Troubleshooting data for LE 40xx abend problems
When Language Environment (LE) detects an unrecoverable error in an LE-enabled application, LE terminates a transaction with an EXEC CICS ABEND and an abend code numbered from 4000 to 4095. LE abends might occur when CICS is under stress (SOS) as described in 4xxx LE/370 abend codes. For more details on LE abend codes, see Language Environment abend codes.
If multiple abends occur within the same time frame (seconds), the first abend is often the cause of subsequent abends. Therefore, ensure that you send the diagnostic information for the initial abend to IBM Support, unless the information is not available or you are otherwise requested.
- Required data:
- The CICS message log that contains the messages leading up to the abend and the abend message. Message CEE1000S contains the abend code and the reason code that is associated with the abend. LE writes message CEE1000S to the system console. You can also find the message in CEEMSG. For more information, see Language Environment runtime messages.
- The MVS system dump that is taken at the point of failure. Create a transaction dump for all LE
40xx abends, except abend 4038. The transaction dump often does not contain enough information to
diagnose the problem. Therefore, if the problem is re-creatable, request a system dump by entering
the following
command:
where xx is the transaction abend code. Ensure that MAX is 1 greater than CUR to capture a dump. Check this by entering CEMT INQ TRD(40xx).CEMT SET TRD(40xx) SYS MAX(1) ADDA CICS system dump of an abend 4038 is not helpful because it is taken at the time of the last termination, not at the point of detection. If you receive an abend 4038, specify the following LE options:
This produces a CICS transaction dump for an abend 4039.TERMTHDACT(UADUMP) ABTERMENC(ABEND) TRAP(ON)If you do not get a dump when the abend occurs, see Step 3 under the MVS system dump section Using dumps for CICS problem determination on z/OS and Language Environment abend and condition handling for more instructions.
To format the system dump in IPCS, you can use one of the following IPCS commands:- The LE verb exit LEDATA. Enter the following
command:
The output from LEDATA basically contains the same data as an LE dump in CEEMSG. For more information on the command, see Formatting and analyzing system dumps in the z/OS Language Environment Debugging GuideVERBX LEDATA ASID(00nn) CAA(xxxxxxxx) DSA(xxxxxxxx) ALL - The IPCS verb exit DFHPDxxx. Enter the following IPCS
command:
where nnnnn is the task number of the abending task.VERBX DFHPDxxx 'APS=<taskid=nnnnn>'This command instructs CICS routines to call the LE verb exit and generates the same output as the verb exit LEDATA. But it eliminates the need to determine the DSA address (register 13) and the CAA address (register 12). It also formats the LE Traceback, CAA, PCB, MEML, RCB, EDB, runtime options, heap control blocks, stack storage control blocks (User and Lib), DSA, and Condition Management control blocks.
- The LE verb exit LEDATA. Enter the following
command:
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. If your LE
TERMTHDACT parameter is set to QUIET or MSG,
a smaller CICS trace (4096K) is sufficient. When possible, turn on level 1 tracing for all CICS
components and level 1-2 for the CICS application (AP) domain. By turning on level 2 tracing for the
AP domain, you enable LE feature trace (FT). For more information on the
TERMTHDACT parameter, see TERMTHDACT. For more information on
considerations for setting TERMTHDACT options, see Generating a Language
Environment dump with TERMTHDACT.
For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- LE traceback if requested by IBM Support. For more information, see Using trace for CICS problem determination on z/OS.
- If you need a larger trace, consider using the CICS auxiliary trace or the GTF Trace with the MVS system dump. For more information, see Using trace for CICS problem determination on z/OS. Ensure that you turn on the levels of tracing as documented in the CICS internal trace section in that topic.
Troubleshooting data for dynamic LIBRARY management problems
Unexpected errors of dynamic LIBRARY management might occur under the following circumstances:
- Installing a dynamic LIBRARY
- Using SET LIBRARY to change a dynamic LIBRARY
- Loading programs
- Using SET PROGRAM NEWCOPY
Also, if an error results in a system dump that is taken by the CICS Loader, the error might be a dynamic LIBRARY management problem.
- Required data:
- The CICS message log that contains the messages that lead up to the failure, including all the DFHLDnnnn messages that report the problem. For more information, see CICS messages.
- LIBRARY audit trail
messages
that are sent to the CSLB transient data queue, which by default is an indirect queue to CSSL. The
most recent instances of audit trail messages are very useful:
- DFHLD0555 and DFHLD0556 that give the dynamic LIBRARY configuration at the time of the error
- DFHLD0505, DFHLD0506, and DFHLD0507 that give details for each LIBRARY when it is installed
- The MVS system dump that is taken at the point of failure. If a dump is not created by CICS when the problem occurs and you can re-create the problem, take a system dump manually, or add the pertinent DFHLDnnnn message to the dump table. If system dumping is permitted for the dump code and system dumping is not disabled, a system dump is taken when the error is detected. If you do not get a dump when the abend occurs, see number 3 under the MVS system dump section in Using dumps for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 tracing for the LD component. If this is not possible, turn on a minimum of level 1-2 for the LD component, and turn on level 1 for LM and SM when possible. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- Output from the CEMT INQUIRE LIBRARY command. The output shows the LIBRARY resources that are installed in the system and the details for each LIBRARY. This command is particularly helpful if audit trail information is not available.
- Output from the CEMT INQUIRE PROGRAM command for programs that are affected by the problem. The output shows the information in the LIBRARY and LIBRARYDSN parameters.
- The CICS auxiliary trace might be useful if you can re-create the problem. As with the CICS internal trace, when possible, turn on level 1 tracing for all CICS components and level 1-2 tracing for the LD component. If this is not possible, turn on a minimum of level 1-2 for the LD component and if possible, level 1 for LM and SM. For more information, see Using trace for CICS problem determination on z/OS.
Troubleshooting data for CICS TS Feature Pack for Mobile Extensions V1.0 in CICS TS 4.2 or CICS TS 5.1
If you are a new user to the CICS TS Feature Pack for Mobile Extensions, before you gather the following documentation and contact IBM Support, run the sample application that is distributed with the feature pack to verify your installation. For more information, see Verifying the operations of the CICS TS Feature Pack for Mobile Extensions in IBM Knowledge Center for CICS TS 5.1 or in IBM Knowledge Center for CICS TS 4.2 depending on the release you use.
- Required data:
- For problems with the CICS JSON assistants, including problems with DFHJS2LS or DFHLS2JS,
provide the following documentation:
- A description of the problem.
- The source document. For DFHLS2JS, the document includes the language structure (copybook). For DFHJS2LS, the document includes the JSON schema that is used as input.
- A list of the parameters that you specified.
- The log file that is generated by DFHJS2LS or DFHLS2JS.
- For problems with the configuration of a pipeline that is used to handle JSON web service requests, provide the pipeline configuration file that is used. This file includes problems with installing and manipulating pipelines. For more information about the pipeline configuration file in CICS TS 5.1, see Creating the CICS infrastructure for a JSON service provider in IBM Knowledge Center for CICS TS 5.1. For the pipeline configuration file in CICS TS 4.2, see Creating the CICS infrastructure for a JSON service provider in IBM Knowledge Center for CICS TS 4.2.
- For JSON web services runtime problems or problems in using the linkable interface to transform
JSON, provide the following documentation:
- An MVS system dump if one is taken. For more
information, see Using dumps for CICS problem determination on
z/OS. If you
receive message DFHPI1007 or DFHPI1008 and a dump is not taken, request a dump when you receive the
message by enter the following
command:
where xxxxx is the CICS message without the DFH prefix.CEMT SET SYD(xxxxx) SYS MAX(1) ADDAnd then re-create the problem or wait for it to occur.
- The CICS internal trace that is included in the MVS system dump when tracing is active. For more information, see Using trace
for CICS problem determination on z/OS.
Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. When possible, turn on the
following trace levels:
- Level 1-2 for the ML, PG, PI, SO, and WB components. ML level 2 trace might affect performance.
- Level 1-2 for the AP component if you have a code page conversion problem.
- Level 1 for all other CICS components.
- An unformatted CICS auxiliary trace if no dump is available. For more information on CICS auxiliary trace and the trace levels you need to turn on, see the CICS internal trace section in Using trace for CICS problem determination on z/OS.
- An MVS system dump if one is taken. For more
information, see Using dumps for CICS problem determination on
z/OS. If you
receive message DFHPI1007 or DFHPI1008 and a dump is not taken, request a dump when you receive the
message by enter the following
command:
- For problems with the CICS JSON assistants, including problems with DFHJS2LS or DFHLS2JS,
provide the following documentation:
Troubleshooting data for MRO problems
- Required data:
- The CICS message log of both regions in the MRO conversation.
- The MVS system dump that is taken at the point of failure. Setting system or transaction dump
codes by using CEMT is preferred. Ensure that appropriate dump codes are set in both CICS regions
within the conversation.For more
information, see Using dumps for CICS problem determination on
z/OS.Note: Messages or abends might be different on each side of the conversation.
- The CICS internal trace of CICS regions that participate in the conversation. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Also, turn on standard level 1 tracing for all components and level 1-2 for the IS component. Use the CETR transaction to turn on internal trace, to set internal trace size, and to set component trace levels. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- If you can re-create the problem, the CICS auxiliary trace might be useful. Ensure that you run the trace on all the CICS regions that are involved in the MRO problem. For more information, see Using trace for CICS problem determination on z/OS.
- If you want to see all of the trace entries in one file, you can use the GTF Trace. The GTF Trace is useful in an XCF environment, when the CICS regions are on different MVS regions. You can write the trace to a GTF trace data set on each MVS, and then merge the files to see the flow of trace in the connected regions. If the CICS regions are in a single MVS, you can write all the trace to a single GTF trace data set. For more information, see Using trace for CICS problem determination on z/OS.
- The CICS transaction dump that is taken at the point of failure. For more information, see Using dumps for CICS problem determination on z/OS.
- Get a SLIP dump if the java system does not automatically create a dump or if level 2 instructs you to dump on a specific condition.
Troubleshooting data for policy problems in CICS TS 5.1 or later
Policy problems occur for different reasons. To identify the source of errors that affect policies, for CICS TS 5.1, see Troubleshooting for policies; for CICS TS 5.2 or later, seeTroubleshooting platforms, applications, and policies. After that, gather complimentary data by using the following guidelines.
- Required data:
- Relevant messages from the CICS console log. These messages typically include any DFHMPnnnn message. For more information, see CICS messages.
- The CICS internal trace that is included in the MVS system dump when tracing is active. For more information, see Using trace
for CICS problem determination on z/OS.
Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. When possible, turn on level
2 tracing for the MP component and level 1 tracing for all other CICS components. If this is not
possible, turn on level 1 tracing by following these rules:
- For problems during the installation of a policy, turn on a minimum of level 1 tracing for MP, PI, SM, PG, and RL components.
- For runtime problems, turn on a minimum of level 1 tracing for MP, EI, ME, and SM components. If the problem involves Time policies, also turn on level 1 tracing for the DS component. For policies with an action of event, also turn on level 1 tracing for EC, EP, and PG components.
- The MVS system dump that is taken when the problem occurs. The way that you get the dump depends on whether or not you get a message, transaction abend, or no output when the problem occurs. For more information, see Using dumps for CICS problem determination on z/OS.
- If you are unable to get a system dump, take a CICS auxiliary trace that leads up to the error.
For more information, see Using trace
for CICS problem determination on z/OS. If possible, turn on level 2 tracing for the MP
components, and level 1 tracing for all other CICS components. If this is not possible, turn on
level 1 tracing by using the following rules:
- For problems during the install of a policy, turn on a minimum of level 1 tracing for MP, PI, SM, PG, and RL components.
- For runtime problems, turn on a minimum of level 1 tracing for MP, EI, ME, and SM components. If the problem involves Time policies, also turn on level 1 tracing for the DS component. For policies with an action of event, also turn on EC, EP, and PG components.
Troubleshooting data for program check or abend problems
- Required data:
- The CICS message log that contains the messages leading up to the abend with the abend code and
the abend message.
If multiple abends occur within the same time frame (seconds), the first abend is often the cause of subsequent abends. Therefore, ensure that you send the diagnostic information for the initial abend to IBM Support, unless the information is not available or you are otherwise requested.
- The MVS system dump that is taken at the point of failure. If system dumping is permitted for the dump code and system dumping is not disabled, a system dump is taken when the error is detected. See number 3 under the MVS system dump section in Using dumps for CICS problem determination on z/OS if you do not get a dump when the abend occurs.
- The CICS message log that contains the messages leading up to the abend with the abend code and
the abend message.
- Optional data:
- The CICS transaction dump that is taken at the point of failure. The transaction dump often does
not contain enough information to diagnose the problem. Therefore, if you have only one transaction
dump and the problem is re-creatable, you can request a system dump by entering one of the following commands:
where xxxx is the transaction abend code.CEMT SET TRD(xxxx) SYS MAX(1) ADDEnsure that MAX is 1 greater than CUR to capture a dump. Check this by entering CEMT INQ TRD(xxxx).
where xxxxx is the CICS message without the DFH.CEMT SET SYD(xxxxx) SYS MAX(1) ADDEnter this command to request a system dump when you receive CICS messages at the time of the error. It is common to suppress dumps for dump code AP0001. Make sure that you do not suppress dumps for AP0001 by entering CEMT INQ SYS(AP0001)
and ensuring that SYS is specified.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components. For more information, see Using trace for CICS problem determination on z/OS.
- The CICS transaction dump that is taken at the point of failure. The transaction dump often does
not contain enough information to diagnose the problem. Therefore, if you have only one transaction
dump and the problem is re-creatable, you can request a system dump by entering one of the following commands:
Troubleshooting data for SFR problems
If you are a new user to the SFR, you can use SupportPac CH1C to test whether program links that use a CICS channel and Link3270 bridge invocations can be made from within a service flow. These tests help you ensure that the CICS Service Flow Runtime is installed and configured correctly. After that, gather the following documentation for a CICS SFR problem before you call IBM Support.
- Required data:
- The CICS message log that contains the SFR error messages, see Messages and codes issued by CICS Service Flow Runtime V3.2for more information.
- If you experience a problem with a Link3270 server adapter, see Steps 4 through 6 underTroubleshooting Link3270 server adapters for instructions on how to turn on vector logging, populate the vector log file, and dump the vector log file.
- The MVS system dump that is taken at the point of failure. The CICS internal trace is included
in the MVS system dump if tracing is active. Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. Activate the CICS auxiliary trace,
and turn on level 1 tracing for all components, with an exception of ALL for EI, AP, BA, BR, EM, PI,
SH, TC, and WB. Then re-create the error and capture a system dump by using the CEMT P
SNAP command. For more information, see Using dumps
for CICS problem determination on z/OS and Using trace
for CICS problem determination on
z/OS.
For information on application domain (AP) trace points that are written by CICS SFR, see Trace points.For information on trace points that are written by CICS Business Transactions Services (BTS), see Trace entries . BTS consists of business application manager (BA), event manager (EM), and scheduler services (SH).
- The non-compiled generated service flow that is produced by the Rational Developer for System z modeler. This is the generated COBOL code that is the code equivalent of what the user modeled on the modeler.
- Optional data:
- The MVS system dump that is taken at the point of failure. For more information, see Using dumps for CICS problem determination on z/OS.
- An archive file that contains
the Service Flow Projects. Perform the following steps from IBM Rational Developer for System z that
contains the Service Flow Projects:
- Click .
- In the Export window, select .
- In the "Archive file" wizard, select the base Service Flow Project and
all its associated projects. The following examples are possible project names:
- serviceprojectname
- serviceprojectname.Interface
- serviceprojectname.Nonterminal
- serviceprojectname.OutboundWebService
- serviceprojectname.Terminal
- Complete the "Archive file" wizard to produce the archive file that contains the Service Flow Projects.
Troubleshooting data for file control shared data table (SDT) problems
- Required data:
- The CICS message log for all CICS regions that use the SDT and the system log (SYSLOG) for all MVS images that use the SDT.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 for the FC component. For more information, see Using trace for CICS problem determination on z/OS.
- The MVS system dump of the CICS address space and the data spaces for the shared data tables.
For more information, see Using dumps for CICS problem
determination on z/OS and Data spaces for data
tables. Use the following MVS command at the time of the problem to capture the dump:
where:DUMP COMM=(CICS & SDT) R nn,JOBNAME=(cicsjob), DSPNAME=('cicsjob'.DFHDTnnn),END- nn
- Specifies the reply identifier.
- cicsjob
- Specifies your CICS job name.
- DFHDTnnn
- Specifies the data space name.
If the total size of the data tables exceeds 2GB, the data records for each table might be spread across more than one data space. For example, DFHDT003, DFHDT004 and so on.
To dump multiple data spaces, you can follow a partial data space name with a wildcard of asterisk. For example, specify DSPNAME=('cicsjob'.DFHDT*) to dump all data spaces that start with DFHDT. However, use wildcards with care because if there are many data spaces the dump data set might be huge. For more information, see Dump information for data tables.
To determine the data space name or names of your CICS region, enter MVS command D J,jobname, where jobname is your CICS job name. For example, a DSPNAME=DFHDT001 in the message output indicates that the data space name is DFHDT001.
Troubleshooting data for storage violation problems
Storage violation occurs for an overlay of storage that should not be changed, for example, an overlay of storage check zones (SCZ) that surround task storage. In normal operation, CICS sets up four task-lifetime storage subpools for each task. Each element in the subpool starts and ends with a check zone that includes the subpool name. At each FREEMAIN execution and at the end of task, CICS checks the check zones and abends the task if either check zone has been overwritten.
To diagnose a storage violation problem, see Dealing with storage violations. After that, gather complementary data by using the following guidelines.
- Required data:
- The CICS message log that contains the messages DFHSM0102 or DFHSM0103, which indicate that CICS has detected a storage violation.
- The MVS system dump that is taken automatically when CICS generates messages DFHSM0102 and DFHSM0103. You can use the SIT parameter DUMP=NO or the system dump table with the system dump codes SM0102 or SM0103 to suppress the dumps. For more information, see Using dumps for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Also, turn on level 1 tracing for all CICS components. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- If you want to identify the transaction that causes the storage overlay or if you need to keep a record of system actions over a long period of time, take the CICS auxiliary trace or the GTF Trace. For more information, see Using trace for CICS problem determination on z/OS.
- If you receive message DFHSM0103 as a result of the storage checker, take an MVS system dump.
You can force storage chain checking by using the CHKSTSK or the
CHKSTRM SIT parameter, or by using the CSFE DEBUG transaction. Set your options
in one of the following ways:
- CHKSTSK=CURRENT
- Validates SCZs for task storage.
- CHKSTRM=CURRENT
- Validates SAAs.
For more information, see How you can force storage chain checking.
Troubleshooting data for terminal hang problems
Terminals hang with x-clock or other symptoms for many reasons, for example, application problems, emulators, or failure to follow SNA protocol.
- Required data:
- The CICS message log that contains DFHZCnnnn messages for the terminal. Messages are written to the console or the CSNE transient data destination. If terminals are involved in transaction routing, collect the CICS message log for both the terminal owning region (TOR) and the application owning region (AOR). For more information, see CICS messages.
- The MVS system dump that is taken after the terminal is hung so control blocks such as the RPL and TCTTE can be analyzed. If terminals are involved in transaction routing, take an MVS system dump of both the TOR and the AOR. For more information, see Using dumps for CICS problem determination on z/OS.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. Also, turn on level 1 tracing for all CICS components. If the problem is re-creatable, set the level of tracing to 1-2 for the AP, IS, and TC components. For more information, see Using trace for CICS problem determination on z/OS.
- Optional data:
- If you want to identify a transaction that causes the terminal to hang and you need to keep a record of system actions over a long period of time, take the CICS auxiliary trace or the GTF Trace. For more information, see Using trace for CICS problem determination on z/OS.
- If the hang occurs at bind time (you never see the CICS good morning message), take a z/OS Communications Server (VTAM) buffer trace of the logon application identifier (applid) to see the bind flow. Also, take a z/OS Communications Server exit trace (CICS VTAM exit trace), and a ZCP trace. For more information on the trace, see Using trace for CICS problem determination on z/OS, z/OS Communications Server exit tracing, and Specifying transaction and terminal-related trace options.
- If a specific DFHZCxxxx message appears to be the cause of the hang, you can add the message to
the CICS system dump table. The following command is an
example:
This produces a system dump at the time of the DFHZCxxxx message. The dump contains the trace that leads up to the failure. Ensure that MAX is 1 greater than CUR to capture a dump. Check this by entering CEMT INQ SYD(ZCxxxx).CEMT SET SYD(ZC2403) MAX(1) ADD
Troubleshooting data for temporary storage (TS) server problems
If you receive a message, see DFHXQnnnn messages for the explanation and user response. For more information, see CICS messages. If a user task waits on a resource type of TSSHARED, TSPOOL, or TSQUEUE, see Investigating temporary storage waits.
- Required data:
- The CICS message log and the MVS system log (SYSLOG).
- The SYSPRINT log for the TS server. Although most of the same information is also visible in the MVS system log, the SYSPRINT log might contain statistics and some other informational messages, for example, the options that are used during start-up.
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components and level 1-2 for the TS component. For more information, see Using trace for CICS problem determination on z/OS.
- If you receive a message and can re-create the problem, take a SLIP dump of the CICS region or
regions. Also, take the TS list structure when you receive the message. Set the SLIP by using the
following MVS command, and then re-create the problem as follows. For more
information, see Using dumps for CICS problem determination on
z/OS.
where:SLIP SET,MSGID=DFHXQnnnn,ACTION=SVCD, STRLIST=(STRNAME=DFHXQLS_poolname,ACC=NOLIM, (LISTNUM=ALL,ADJ=DIRECTIO,EDATA=UNSERIALIZE)), JOBLIST=(ts-server,XCFAS,cicsjob), SDATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,SUM,SWA,TRT, COUPLE,WLM,GRSQ,LPA),MATCHLIM=1,ID=CIC1,END- DFHXQnnnn
- Specifies the message you receive.
- poolname
- Specifies your TS pool.
- ts-server
- Specifies your TS server.
- cicsjob
- Specifies the job name for your CICS region.
If the pool structure becomes unexpectedly full, a dump with the STRLIST=(...(...)) helps determine what fills up the pool. If you only want to dump the pool, see Dumping a shared temporary storage list structure for more instructions.
- If there is a wait or hang, take an MVS system dump of the CICS region or regions. Also, take
the TS list structure when you notice the hang or wait. For more
information, see Using dumps for CICS problem determination on
z/OS. Use the following MVS command followed by the reply to
capture the dump of the list structure:
where:DUMP COMM=(dumpname) R yy,JOBNAME=(ts-server,XCFAS,cicsjob), STRLIST=(STRNAME=DFHXQLS_poolname,ACC=NOLIM, (LISTNUM=ALL,ADJ=DIRECTIO,EDATA=UNSERIALIZE)), SDATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,SUM,SWA,TRT, COUPLE,WLM,GRSQ,LPA),END- dumpname
- Specifies the name you want to give the dump.
- yy
- Specifies the reply identifier.
- ts-server
- Specifies your TS server.
- cicsjob
- Specifies the job name for your CICS region.
- poolname
- Specifies your TS pool.
A dump of the pool is rarely needed by IBM, so you can exclude STRLIST=(...(...)) from the dump command if you are concerned about dump size.
- Optional data:
If you receive a wait and can reproduce the problem, consider using the CICS auxiliary trace or the GTF trace with the MVS system dump. The MVS system dump alone is unlikely to show anything about system activity in the period that leads up to the wait because the trace table probably wraps before you have a chance to respond. But the CICS auxiliary trace and the GTF trace can provide you with enough trace. For more information, see Using trace for CICS problem determination on z/OS..
Troubleshooting data for waits or loops
A wait or hang occurs for different reasons. For example, the region might be at the MXT limit, so no new tasks can be attached. Or the region might wait for an ECB to be posted. A loop is the repeated execution of some code.
If you suspect that the problem is due to another IBM product or subsystem like MVS system logger, DB2, VSAM, or RLS, see the MustGather document for that particular component. Otherwise, gather the following documentation for the wait or loop before you call IBM Support:
- Required data:
- The CICS internal trace that is included in the MVS system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or later. When possible, turn on level 1 tracing for all CICS components. For more information, see Using trace for CICS problem determination on z/OS.
- An MVS system dump of the CICS region that is taken as soon as you notice the hang or wait. For more
information, see Using dumps for CICS problem determination on
z/OS. Use the following MVS command followed by the reply to
capture the dump:
where:DUMP COMM=(dumpname) R yy,JOBNAME=(cicsjob), SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN)- dumpname
- Specifies the name you want to give the dump.
- yy
- Specifies the reply identifier.
- cicsjob
- Specifies the job name of your CICS region.
- The CPU utilization for the CICS region. To display the active CICS job, enter S.DA when you use SDSF under TSO. Enter PF11 to display the percent of CPU time that is used on behalf of this address space during the most recent interval measured.
- Optional data:
- The CICS message log that contains the last message created by CICS and the time the message was created.
- If you can reproduce the problem, consider using the CICS auxiliary trace or the GTF Trace with the MVS system dump. The MVS system dump alone is unlikely to show anything about system activity in the period that leads up to the wait because the trace table probably wraps before you have a chance to respond. For more information, see Using trace for CICS problem determination on z/OS..
Troubleshooting data for web services, XML, and JSON transformation problems
If you are a new user to web services, run the web services sample application that is distributed with CICS. For more information on sample applications, see The CICS catalog manager example application. The CICS sample requires that you use WebSphere Application Server or something similar as the client platform. If you have neither of those, you can test CICS web services by using free Eclipse software to ensure that web services are set up correctly in your CICS region. For more information, see Testing CICS Web services with Explorer. After that, gather the following documentation for your web services problem and contact IBM Support.
- Required data:
- For CICS web services assistant, the CICS XML assistant, or the CICS JSON assistant that has
problems with DFHWS2LS, DFHLS2WS, DFHSC2LS, DFHLS2SC, DFHJS2LS, or DFHLS2JS, collect the following
documentation. For more information, see The CICS web services
assistant, The CICS XML assistant, and The CICS JSON assistant.
- A description of the problem.
- The source document. For DFHLS2WS, DFHLS2SC, and DFHLS2JS, the source document includes the language structure (copybook). For DFHWS2LS, DFHSC2LS, or DFHJS2LS, the document includes the Web Services Description Language (WSDL) file, XML schema, or JSON schema that is used as input to the assistant, and all external WSDL and schema documents that are included in it.
- A list of the parameters that you specified.
- The log file that is generated by DFHWS2LS, DFHLS2WS, DFHSC2LS, DFHLS2SC, DFHJS2LS, or DFHLS2JS.
- For problems with the configuration of a pipeline that is used to handle web service requests, collect the pipeline configuration file that is used. This file includes problems such as handler programs not being called or problems with installing and manipulating pipelines by using CEMT. For more information, see Pipeline configuration files.
- For web services runtime problems, collect the following documentation:
- An MVS system dump if one is taken. For more
information, see Using dumps for CICS problem determination on
z/OS. If you
receive message DFHPI1007 or DFHPI1008 and a dump is not taken, enter the following command when you
receive the message to request a
dump:
where xxxxx specifies the CICS message without the DFH. And then re-create the problem or wait for it to occur.CEMT SET SYD(xxxxx) SYS MAX(1) ADD - The CICS internal trace that is included in the MVS system dump when tracing is active. For more information, see Using trace
for CICS problem determination on z/OS.
Ensure that the internal trace table size is big enough to contain
sufficient data for diagnosis; for example, you can use a table size of 20480K in CICS TS 4.2 or
later. When possible, turn on the
following trace levels:
- Level 1-2 for the PI, SO, and WB components.
- Level 1-2 for the AP component if you have a code page conversion problem.
- Level 1-2 for the ML component. ML level 2 trace might affect performance.
- Level 1-2 for the PG component.
- Level 1 for all other CICS components.
- An unformatted CICS auxiliary trace if no dump is available. For more information on the CICS internal trace and the trace levels you need to turn on, see the CICS internal tracesection in Using trace for CICS problem determination on z/OS.
- An MVS system dump if one is taken. For more
information, see Using dumps for CICS problem determination on
z/OS. If you
receive message DFHPI1007 or DFHPI1008 and a dump is not taken, enter the following command when you
receive the message to request a
dump:
- For CICS web services assistant, the CICS XML assistant, or the CICS JSON assistant that has
problems with DFHWS2LS, DFHLS2WS, DFHSC2LS, DFHLS2SC, DFHJS2LS, or DFHLS2JS, collect the following
documentation. For more information, see The CICS web services
assistant, The CICS XML assistant, and The CICS JSON assistant.