|
| Problem | MustGather for heap leak problems with the WebSphere® Application Server Out of Memory component. Gathering this information before calling® IBM support will help familiarize you with the troubleshooting process and save you time. | | | | Solution | The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on an AIX® platform when you suspect a memory leak in the Java™ heap. This suspicion is based upon your analysis of the documentation collected from the technote: MustGather: Out of Memory errors on AIX, Part 1.
If you have already contacted support, continue to the component specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.
Out of Memory (heap leak) specific MustGather information:
- Enabling HeapDumps:
- WebSphere Application Server V6.0 release:
- To set the following Environment Entries from the administrative console,
- Select:
Servers > Application Servers >server_name> Java and Process Management > Process Definition > Environment Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP true
IBM_HEAP_DUMP true
IBM_HEAPDUMPDIR your_directory
IBM_HEAPDUMP_OUTOFMEMORY true
IBM_JAVADUMP_OUTOFMEMORY true
- Important:Make sure that you save changes to the master configuration.
- WebSphere Application Server V5.0 and V5.1 releases:
- To set the following Environment Entries from the administrative console
- Select:
Servers > Application Servers >server_name> Process Definition > Environment Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP true
IBM_HEAP_DUMP true
IBM_HEAPDUMPDIR your_directory
IBM_HEAPDUMP_OUTOFMEMORY true
IBM_JAVADUMP_OUTOFMEMORY true
- Important:Make sure that you save changes to the master configuration.
- WebSphere Application Server V4.0 and V3.5 releases:
To enable the heapdump, add the following lines to the top of the startupServer.sh file in the install_root/bin directory:
export IBM_HEAPDUMP=true
export IBM_HEAP_DUMP=true
export IBM_HEAPDUMP_OUTOFMEMORY=true
export IBM_JAVADUMP_OUTOFMEMORY=true
export IBM_HEAPDUMPDIR=directory_path
- Notes:
- The heapdump will be written to the current directory, install_root/bin, or the directory specified by the IBM_HEAPDUMPDIR environment variable.
- The file that is generated is called heapdumppid.time in internal format.txt. This file can be very large; allow for at least the maximum heap size. If the maximum heap is set to 512 MB, allow for 0.5 GB for the dump.
- This mechanism uses the same signal handler as javacore processing, so you must not have DISABLE_JAVADUMP set as an environment variable.
- Follow instructions to Enable verbosegc is WebSphere Application Server
- Restart the Application Server.
- After starting WebSphere Application Server, run the AIX_memory_leak.sh (select appropriate attached script depending on your version of AIX) shell script while trying to reproduce the problem:
AIX_memory_leak.sh -p pid-f base_filename-i interval
Where: pid is the process id to be profiled base_filename is a path and identifier for the output interval is the time in seconds between each iteration
For Example: AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120
- Wait for the OutOfMemoryError condition to occur. This causes the generation of a heapdump.
- If you want to analyze the heapdump yourself, download the HeapRoots tool. If you do not want to analyze this yourself, proceed to next step
- Collect the following:
- For WebSphere Application Server V6.0 release:
- All files in install_root/profiles/profile_name/logs/server_name directory. If you have configured the Application Server to write logs into a different location, then please send them accordingly.
- All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
- All javacorepid.time in internal format.txt files
- startServer.log
- Server.xml located in install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_namedirectory.
- List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.
- For WebSphere Application Server V5.0 and V5.1 releases:
- systemErr.log, systemOut.log, native_stderr.log, native_stdout.log and startServer.log
- All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
- All javacorepid.time in internal format.txt files
- Server.xml from the directory:
install_root/config/cells/cell_name/nodes/node_name/servers/server_name - List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.
- For WebSphere Application Server V4.0 and V3.5 releases:
- stderr, stdout, tracefile and nanny.trace
- All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
- All javacorepid.time in internal format.txt files
- Full XMLConfig export.
- List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.
- Follow instructions to send diagnostic information to IBM support.
For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site. | |
| | |   |
| |
|
Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory Operating system(s): AIX Software version: 3.5, 4.0, 5.0, 5.1, 6.0 Software edition: Advanced, Base, Express, Network Deployment, Single Server Reference #: 1138457 IBM Group: Software Group Modified date: 2004-12-07
(C) Copyright IBM Corporation 2000, 2004. All Rights Reserved.
|