Storage shortages

Diagnosing storage shortage problems can be difficult. However, with a dump and an idea of where the shortage occurs, the problem source can be identified.

When you contact the IBM® Support Center, have available as much relevant information as possible. Remember to check for any changes that were made at the site. Many applications use VTAM's or TCP/IP's services. Changes that are made for one application can adversely affect another application.

Storage shortages in VTAM

Storage shortages can manifest themselves in any of the following ways:

VTAM® messages
A VTAM message might be received at the console, indicating a storage shortage. This message normally indicates the z/OS® subpool where the shortage occurred. Using this subpool number, the area of storage that experiences the shortage can be determined. For example, subpool 231 (SP231) is CSA, while subpool 17 (SP17) is in the address space private area.

The most common storage shortage problems involve CSA. VTAM uses large amounts of CSA for its control blocks and buffers that contain data that is being sent around the network. This problem can be because of various causes. For example, if one application is flooding another with data, and the second application is unable to receive the data and process it at an adequate rate, then the buffers build up in VTAM storage. This situation can be avoided by using session pacing.

If VTAM is unable to get enough storage to issue a message, then the normal storage shortage message is accompanied by an IST999E message.

RCPRI and RCSEC
The Primary return code (RCPRI) and secondary return code (RCSEC) are both given to DDF when an APPCCMD macro completes. On occasion, these codes might indicate that there is a storage shortage in VTAM. For example, if RCPRI is x'0084' and RCSEC is x'0000', then this indicates a storage shortage while VTAM was receiving data or sending a pacing response. An RCPRI of x'0098' with RCSEC of x'0000' indicates there is a temporary storage shortage while sending data. Usually this return code means that the send request temporarily depleted the buffer pool to such an extent that the pool must be expanded. The expansion did not occurr before the completion of the APPCCMD macro.
SNA sense code
Some SNA sense codes indicate there might be a storage shortage also. For example, a user might be accessing data from the remote database, and receive '084C0000'. Checking this sense code in the manual indicates that there is a permanent insufficient resource condition. This resource could be storage. Other sense codes, such as '800A000' indicate a storage type problem, but not necessarily a storage shortage.
Abends
There are a series of z/OS abends, which indicate storage shortage problems. For example, ABEND878 and ABEND80A. These are uncommon in VTAM.
Hangs
Depending on the storage shortage and the processing that is occurring, the storage shortage could manifest itself in a hang situation. For example, if a virtual route becomes blocked because of storage shortages, then all the sessions that were using that session hangs until the storage shortage is relieved and the virtual route becomes open again. When any of these indications of a storage problem is received, the following steps can be used to find out more information about the shortage:
Determine the area of storage shortage:

This task is important, as the VTAM display command has information about the VTAM CSA usage. This information does not help diagnosis if the storage shortage is in VTAM private. However, perhaps CSA usage should be checked anyway.

The area can be determined by checking the z/OS subpool number (for example, as given in a message). The following subpools are in the CSA area:
    SP227, SP228, SP231, SP239, and SP241.
Display buffer usage:

If the storage shortage occurred in one of the CSA subpools, then a D NET,BFRUSE command can be used to determine the amount of CSA storage that is being used by VTAM. Usually the buffer use display gives a good idea of which VTAM pool is causing the storage shortage.

Monitor buffer usage:

When you look at a buffer shortage, it is often helpful to know whether the onset of the problem was gradual or immediate. If regular buffer usage displays are done, then gradual increases in buffer use can be seen. These gradual increases might take days to manifest themselves into storage shortage problems. In fact, if VTAM is taken down regularly, the storage shortage symptom might not be seen.

Another benefit of regular monitoring of the buffers is that when a problem does occur "normal" buffer usage for that host is known, so some comparison of the buffer values can be done.

If the onset of the buffer shortage is very fast, then check the system console (or log), looking for some event that has triggered this problem. This could be virtually anything. For example, an NCP has a problem and large amounts of data that was heading out onto the network is now caught in VTAM while recovery of the NCP is attempted.

Create a dump file
To finally determine the cause of the problem, a dump is usually necessary. When you take the dump, be sure to include CSA in the dumping options. Without CSA the dump is almost useless. The VTAM diagnosis manual describes how to find each of the VTAM pools, and what control blocks are allocated in each pool.
Traces
The SMS option on the VTAM Internal Trace can be helpful when you look at storage-related problems. When used with other options (PSS, SSCP and PIU for example), it can give a much clearer insight into the processing that is causing the storage shortage.

The SMS storage trace can be used to monitor the buffer pool usage if regular displays are inconvenient.

Storage shortage problems fall into the following main categories:

No session pacing
In this case, the VTAM IO buffers (IOBUFs) are being flooded by one application. Finding the IOBUFs in the dump can often lead to the application at fault. Usually the problem starts after the introduction of (or changes to) an application. Look for any changes in the system that could have caused the problem.
Control blocks not freed
There have been problems in the past where control blocks were not freed when they were no longer needed. These control blocks gradually fill large amounts of CSA. This type of problem can normally be found with a search in INFOSYS, if this is available at the site, or RETAIN, if you call IBM Support or IBM Service. If a known problem is not found, then the IBM Support Center should be contacted for further problem diagnosis.
Buffer pool unable to expand
If the buffer pool is scheduled for expansion by VTAM, but for some reason is unable to do so, a storage shortage error can be received. This could be a transient condition while the buffer pool is being expanded.

If this is not a transient condition, check that the pool is eligible for expansion. That is, there is an expansion limit and expansion number that is assigned to the buffer pool. If dynamic expansion has not been allowed, then the pool might have used all of its base allocation. In this case, the base allocation should be increased.