|
Problem |
MustGather for heap leak problems with the WebSphere®
Application Server Out of Memory component. Gathering this information
before calling IBM support will help familiarize you with the
troubleshooting process and save you time. |
|
Solution |
The following steps outline how to troubleshoot
java.lang.OutOfMemoryError errors on an AIX® platform when you
suspect a memory leak in the Java™ heap. This suspicion is based upon your
analysis of the documentation collected from the technote: MustGather:
Out of Memory errors on AIX, Part 1.
If you have already contacted support, continue to the component specific
MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
Out of Memory (heap leak) specific MustGather information
- Enabling HeapDumps:
- WebSphere Application Server V6.0 release:
- To set the following Environment Entries from the administrative
console, select the following:
WebSphere Application Server V6.0 and V6.0.1
Servers > Application Servers > server_name> Java
and Process Management > Process Definition > Environment Entries
> New
WebSphere Application Server V6.0.2 and higher
Servers > Application servers > server_name > Java
and Process Management > Process Definition > Custom Properties >
New
- Add the following Name and Value pairs:
IBM_HEAPDUMP true
IBM_HEAP_DUMP true
IBM_HEAPDUMPDIR
your_directory
IBM_HEAPDUMP_OUTOFMEMORY true
IBM_JAVADUMP_OUTOFMEMORY true
IBM_JAVA_HEAPDUMP_TXT true
IBM_JAVACOREDIR
your_directory
Important: Make sure that you save changes to the master
configuration.
- WebSphere Application Server V5.1.1 releases:
- To set the following Environment Entries from the administrative
console, select the following:
Servers > Application Servers > server_name>
Process Definition > Environment Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP true
IBM_HEAP_DUMP true
IBM_HEAPDUMPDIR
your_directory
IBM_HEAPDUMP_OUTOFMEMORY true
IBM_JAVADUMP_OUTOFMEMORY true
IBM_JAVA_HEAPDUMP_TXT true
IBM_JAVACOREDIR
your_directory
Important: Make sure that you save changes to the master
configuration.
- WebSphere Application Server V5.0, V5.1, V4.0 and V3.5
releases:
To enable the heapdump, add the following lines to the top of the
startupServer.sh file in the
install_root/bin directory:
export IBM_HEAPDUMP=true
export IBM_HEAP_DUMP=true
export IBM_HEAPDUMP_OUTOFMEMORY=true
export IBM_JAVADUMP_OUTOFMEMORY=true
export IBM_HEAPDUMPDIR=directory_path
export IBM_JAVACOREDIR=directory_path
- Notes:
- The heapdump will be written to the current directory,
install_root/bin, or the directory specified by
the IBM_HEAPDUMPDIR environment variable.
- The file that is generated is called
heapdumppid.time in internal
format.txt. This file can be very large; allow for at least the
maximum heap size. If the maximum heap is set to 512 MB, allow
for 0.5 GB for the dump.
- The javacore file will be written to the current
directory, install_root/ bin, or the directory specified by the
IBM_JAVACOREDIR environment variable.
- This mechanism uses the same signal handler as javacore
processing, so you must not have DISABLE_JAVADUMP set as an
environment variable.
- Verify that you have sufficient file system space to
produce the Heapdumps. Heapdump files can range in size from 200MB to 2GB.
- Follow instructions to Enable
verbosegc is WebSphere Application Server
- Set the ulimit -f value to unlimited so it will not truncate the
Heapdump file.
- Restart the Application Server.
- After starting WebSphere Application Server, run the
AIX_memory_leak.sh (select appropriate attached script depending
on your version of AIX) shell script while trying to reproduce the
problem:
AIX_memory_leak.sh -p pid -f
base_filename -i interval
Where:
pid is the process id to be profiled
base_filename is a path and identifier for the output
interval is the time in seconds between each iteration
For Example:
AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120
- Wait for the OutOfMemoryError condition to occur. This causes
the generation of a heapdump.
- If you want to analyze the heapdump yourself, download the HeapRoots
tool. If you do not want to analyze this yourself, proceed to next
step
- Collect the following:
- For WebSphere Application Server V6.0 release:
- All files in
install_root/profiles/profile_name/logs/server_name
directory. If you have configured the Application Server to write logs
into a different location, then please send them accordingly.
- All heapdumppid.time
in internal format.txt files with each one in its own ZIP or TAR
file.
- All javacorepid.time
in internal format.txt files
- startServer.log
- Server.xml located in
install_root/profiles/profile_name/config/cells/
cell_name/nodes/node_name/servers/server_name
directory.
- List of all other software and version information on
the system. For example, any databases, WebSphere MQ®, and so forth.
- For WebSphere Application Server V5.0 and V5.1
releases:
- systemErr.log, systemOut.log,
native_stderr.log, native_stdout.log and
startServer.log
- All heapdumppid.time
in internal format.txt files with each one in its own ZIP or TAR
file.
- All javacorepid.time
in internal format.txt files
- Server.xml from the directory:
install_root/config/cells/cell_name/nodes/
node_name/servers/server_name
- List of all other software and version information on
the system. For example, any databases, WebSphere MQ®, and so forth.
- For WebSphere Application Server V4.0 and V3.5
releases:
- stderr, stdout, tracefile
and nanny.trace
- All heapdumppid.time
in internal format.txt files with each one in its own ZIP or TAR
file.
- All javacorepid.time
in internal format.txt files
- Full XMLConfig export.
- List of all other software and version information on
the system. For example, any databases, WebSphere MQ®, and so forth.
- Follow instructions to send
diagnostic information to IBM support.
For a listing of all technotes, downloads, and educational materials
specific to the Out of Memory component, search the WebSphere
Application Server support site. |
|
|