Using trace to find out why tasks are waiting on interval control
Follow this procedure to use trace to find out why tasks are waiting on interval control.
Before you begin
About this task
As far as tracing selectivity is concerned, you need to capture level 2 trace entries made by dispatcher domain, timer domain, and interval control program. The sort of trace entries that you can expect in normal operation are shown in the examples below. They show the flow of data and control following execution of the command EXEC CICS DELAY INTERVAL(000003). A similar set of trace entries would be obtained if TIME had been specified instead of INTERVAL, because TIME values are converted to corresponding INTERVAL values before timer domain is called.
Procedure
- Use the CETR transaction to set up the following tracing
options:
- Specify special tracing for the level 2 trace points for components DS (dispatcher domain), TI (timer domain), and IC (interval control program).
- Select special tracing for the task causing the problem, by specifying special tracing both for the transaction and for the terminal where it is to be run.
- Set the master system trace flag off, to turn off all standard tracing. This helps minimize the number of trace entries not connected with the problem.
- Make sure that auxiliary tracing is active
- Set the transaction running. When the problem appears, format the auxiliary trace data set and either print it or view it online.
- Analyze the trace, using the examples as a guide.
- Figure 1 shows
the first two entries that you get following execution of the EXEC
CICS DELAY INTERVAL(000003) command. Figure 1. Trace entries following EXEC CICS DELAY INTERVAL(000003) invocation
AP 00E1 EIP ENTRY DELAY REQ(0004) FIELD-A( 0034BD70 ....) FIELD-B(08001004 ....) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-8413F43E TIME-16:31:58.0431533750 INTERVAL-00.0000166250 =000602= AP 00F3 ICP ENTRY WAIT REQ(2003) FIELD-A(0000003C ....) FIELD-B(00000000 ....) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-84760B88 TIME-16:31:58.0432681250 INTERVAL-00.0000370000 =000605=- Trace point AP 00E1 is on ENTRY to the EIP DELAY
routine. The function is stated in the trace header, and the fact
that this trace is made on ENTRY can be deduced from the value shown
in the request field, REQ(0004).
The rightmost two bytes of FIELD B give the EIBFN value, in this case X'1004'. This shows that this is an interval control DELAY request.
The value shown against TASK is the trace task number, and it is unique to the task while the task is in the system. Its purpose is to show which trace entries relate to which tasks. The task number in this example is 00163. As long as the task is in the system, and either running or suspended, trace entries having this task number always relate to it. Use the task number for your task to identify the trace entries associated with it.
- Trace point AP 00F3 is on ENTRY to the ICP WAIT
routine. The function is given explicitly in the trace header, and
both the function and the fact that this represents ENTRY to the routine
can be deduced from the request field, REQ(2003).
The value of FIELD A, X'0000003C', is an important one for problem determination. It shows the interval that has been specified, in this case three seconds. Check the value shown here for your own task, to make sure it is what you expect it to be.
- Trace point AP 00E1 is on ENTRY to the EIP DELAY
routine. The function is stated in the trace header, and the fact
that this trace is made on ENTRY can be deduced from the value shown
in the request field, REQ(0004).
- Look next for an entry with point ID DS 0004 showing
your task being suspended, as in Figure 2. You might
see TI domain trace entries preceding it that show entry and exit
for FUNCTION(REQUEST_NOTIFY_INTERVAL), but these do not always appear.
There might also be some intervening entries, but they are unlikely
to be of relevance to the problem. Figure 2. Trace entries showing interval calculation and task suspension
TI 0100 TISR ENTRY - FUNCTION(REQUEST_NOTIFY_INTERVAL) DOMAIN_TOKEN (00E70000 , 00000000) STCK_INTERVAL (00000002DC6C1000) PERIODIC_NOTIFY(NO) NOTIFY_TYPE(TIMER_TASK) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-8476352A TIME-16:31:58.0442390000 INTERVAL-00.0000155000 =000614= 1-0000 00600000 00000006 00000000 00000000 B3B00000 00000000 01000000 00000000 *.-..............................* 0020 00000000 00000000 00000000 00E70000 00000000 00000000 00000000 00000002 *.............X..................* 0040 DC6C1000 00000000 02020000 00000000 00000000 00000000 00000000 00000000 *.%........................ ......* TI 0101 TISR EXIT - FUNCTION(REQUEST_NOTIFY_INTERVAL) RESPONSE(OK) TIMER_TOKEN(03B9B058 , 0000001B) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-8476352A TIME-16:31:58.0738898750 INTERVAL-00.0296188125* =000617= 1-0000 00600000 00000006 00000000 00000000 B3B00000 00000000 01000100 00000000 *.-..............................* 0020 00000000 00000000 00000000 00E70000 00000000 03B9B058 0000001B 00000002 *.............X..................* 0040 DC6C1000 00000000 02020000 00000000 00000000 00000000 00000000 00000000 *.%........................ ......* DS 0004 DSSR ENTRY - FUNCTION(SUSPEND) SUSPEND_TOKEN(01040034) RESOURCE_NAME(1477) RESOURCE_TYPE(ICWAIT) PURGEABLE(YES) DEADLOCK_ACTION(INHIBIT) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-847645CE TIME-16:31:58.0739336250 INTERVAL-00.0000437500 =000618= 1-0000 00580000 00000014 00000001 00000000 B7050000 00000000 04000100 00000000 *................................* 0020 00000000 01040034 F1F4F7F7 40404040 C9C3E6C1 C9E34040 0000001B 00000002 *........1477 ICWAIT ........* 0040 DC6C1000 00000000 02010003 00000000 00000000 00000000 *.%...................... *- Trace point TI 0100, if shown, is on ENTRY to the
REQUEST_NOTIFY_INTERVAL function of timer domain. This is stated explicitly
in the trace header.
The value shown in the header for STCK_INTERVAL is derived from the machine store clock value calculated for the DELAY interval specified on the EXEC CICS DELAY command. You can find out how store clock values are related to times in hours, minutes, and seconds in the z/Architecture Principles of Operation.
If you do the calculation, you find that the value shown is not exactly equal to the interval you specified. An extra microsecond is added, to account for the case where the interval is specified as 0.
In this example, 3 seconds is exactly equal to a store clock interval of X'00000002DC6C0000'. You can see that the actual store clock value is quoted in the trace entry as X'00000002DC6C1000', which is 3 seconds plus 1 microsecond.
The TIME field of the trace entry shows the time at which the entry was made, in the format hh:mm:ss. The value in this example (ignoring the fractions of a second) is 16:31:58. It follows that the task is due to be resumed when the time is 16:32:01, because the interval is 3 seconds.
- Trace point TI 0101, if shown, is on EXIT from the REQUEST_NOTIFY_INTERVAL function of timer domain. You can see from RESPONSE(OK) in the header that the function completed normally.
- Trace point DS 0004 is on ENTRY to the dispatcher
task SUSPEND/RESUME interface.
The SUSPEND_TOKEN field in the trace header is significant. It shows the unique suspend token being used for this SUSPEND/RESUME dialog, and it is referred to explicitly again in a later trace entry showing that the task has been resumed. In this example, the suspend token is X'01040034'.
Any subsequent dispatcher trace entry that shows the suspend token for your task is connected with the suspension or resumption of the task.
Field RESOURCE_TYPE(ICWAIT) in the trace header shows that the resource type associated with this suspend is ICWAIT. ICWAIT is the resource type that is returned on CEMT INQ TASK for tasks that are waiting on interval control.
- Trace point TI 0100, if shown, is on ENTRY to the
REQUEST_NOTIFY_INTERVAL function of timer domain. This is stated explicitly
in the trace header.
- Get some trace entries recording system activity during the period when your task is suspended. There are likely to be relatively few at the level of tracing detail you have specified, but you need to look further on in the trace to find the next entries of interest.
- Add 3 seconds (or whatever interval you specified) to
the time shown on the last trace entry you looked at, and turn forward
to the trace entries made at around that time. Now look for an entry
made from trace point DS 0004. This does not show the task
number for your task, but it does show its suspend token. When
you have found it, go back one entry. You should find there a trace
entry made from trace point AP F322. This and
the following two trace entries of interest are shown in Figure 3. Figure 3. Trace entries showing your task being resumed
AP F322 APTIX RESUMED - SYSTEM TASK APTIX RESUMED TASK-00006 KE_NUM-0009 TCB-009F3338 RET-84773724 TIME-16:32:01.1016870625 INTERVAL-00.0001065000 =000670= 1-0000 01000000 D7C5D5C4 D5D6E3D7 01107739 00E70000 00000000 03B9B058 0000001B *....PENDNOTP.....X..............* 0020 01080002 00D40000 00000000 03B9B000 00000001 01050002 00000000 00000000 *.....M..........................* 0040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 *................................* 0060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 *................................* DS 0004 DSSR ENTRY - FUNCTION(RESUME) SUSPEND_TOKEN(01040034) TASK-00006 KE_NUM-0009 TCB-009F3338 RET-847646D4 TIME-16:32:01.1019761875 INTERVAL-00.0000278125 =000674= 1-0000 00580000 00000014 00000001 00000000 B4000000 00000000 05000100 00000000 *................................* 0020 00000000 01040034 00000000 00E70000 00000000 03B9B058 0000001A 000026EE *.............X..................* 0040 D9AC1000 00000000 00000000 0001632C 00000000 00000000 *R....................... * DS 0005 DSSR EXIT - FUNCTION(RESUME) RESPONSE(OK) TASK-00006 KE_NUM-0009 TCB-009F3338 RET-847646D4 TIME-16:32:01.1019959375 INTERVAL-00.0000197500 =000675= 1-0000 00580000 00000014 00000001 00000000 B4000000 00000000 05000100 00000000 *................................* 0020 00000000 01040034 00000000 00E70000 00000000 03B9B058 0000001A 000026EE *.............X..................* 0040 D9AC1000 00000000 00000000 0001632C 00000000 00000000 *R....................... *- Trace point AP F322 is used to report that system
task APTIX has been resumed. APTIX has the job of “waking up” your
task on expiration of the specified interval.
The task number for APTIX is, in this case, X'00006', and this value is shown on the trace entry.
- Trace point DS 0004 is on entry to the dispatcher
SUSPEND/RESUME interface. This function is stated explicitly in the
header. TASK-00006 indicates that the trace entry is for system task
APTIX.
SUSPEND_TOKEN(01040034) shows that APTIX is requesting dispatcher domain to resume the task that was suspended for the specified interval. You will recall that a suspend token of X'01040034' was given to your task when it was first suspended.
- Trace point DS 0005 is on exit from the dispatcher
SUSPEND/RESUME interface.
The trace entry shows RESPONSE(OK), indicating that the task whose suspend token was X'01040034' has successfully been resumed. However, note that this does not necessarily mean that the task has started to run—it has only been made “dispatchable”. For example, it still needs to wait for a TCB to become available.
- Trace point AP F322 is used to report that system
task APTIX has been resumed. APTIX has the job of “waking up” your
task on expiration of the specified interval.
- Now look forward in the trace, and locate a trace entry
made from trace point AP 00F3 and showing your task number. This and the next entry conclude the DELAY request for your
task. They are shown in Figure 4. Figure 4. Trace entries showing satisfactory conclusion of the DELAY request
aAP 00F3 ICP EXIT NORMAL REQ(0005) FIELD-A(01000300 ....) FIELD-B(03BD6EE0 ..>.) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-84760B88 TIME-16:32:01.1023045625 INTERVAL-00.0000154375 =000688= AP 00E1 EIP EXIT DELAY OK REQ(00F4) FIELD-A(00000000 ....) FIELD-B(00001004 ....) TASK-00163 KE_NUM-0007 TCB-009F3338 RET-8413F43E TIME-16:32:01.1024153750 INTERVAL-00.0000235625 =000691=- Trace point AP 00F3 is on EXIT from interval control program. Field REQ(0005) shows that this is so, and it also shows that the response was normal. Anything other than a normal response would result in a value other than X'00' for the first byte of the REQ field.
- Trace point AP 00E1 is on EXIT from the EXEC interface
program. This is shown by bits 0–3 of the second byte of the REQ value, X'F4'.
The values shown for FIELD A and FIELD B show that no exception condition was detected.
- Figure 1 shows
the first two entries that you get following execution of the EXEC
CICS DELAY INTERVAL(000003) command.
Results
When you look at your own trace table, be concerned principally with finding the point at which the processing went wrong. Also, watch for bad parameters. If you do find one, it could mean that an application has a coding error, or some field holding a parameter has been overlaid, or an error has occurred in CICS code.
Checking your application code is the easiest option you have. If you find that it is correct and you suspect a storage violation, see Dealing with storage violations. If you think the error is in CICS code, contact the IBM® Support Center.
dfhs1c0072.html |
Last updated: Monday, 5 June 2017