|
Problem |
MustGather for problems with the WebSphere® Application
Server (all versions) Out of Memory component on the Linux® platform.
Gathering this information before calling IBM® support helps familiarize
you with the troubleshooting process and saves you time. |
|
Solution |
This document describes how to troubleshoot
java.lang.OutOfMemoryError errors on a Linux platform. Your
suspicion of a memory leak in the Java™ heap is based on your analysis of
the documentation collected from MustGather:
Out of Memory errors on Linux, Part 1.
If you already contacted support, continue to the component-specific
MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
Out of Memory (heap leak) specific MustGather information
Debugging the Java virtual machine (JVM™) running out of Java heap:
- Enable HeapDumps:
- Enabling HeapDumps for all V6.0 release:
- To set the following Environment Entries from the administrative
console, select:
Servers > Application Servers >
server_name > Java and Process Management > Process
Definition > Environment Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP |
true |
IBM_HEAP_DUMP |
true |
IBM_HEAPDUMPDIR |
your_directory |
IBM_HEAPDUMP_OUTOFMEMORY |
true |
IBM_JAVADUMP_OUTOFMEMORY |
true |
IBM_JAVA_HEAPDUMP_TEXT |
true |
|
 |
- Make sure that you save changes to the master configuration.
- Enabling HeapDumps for all V5.1.1 releases:
- To set the following Environment Entries from the administrative
console, select:
Servers > Application Servers >
server_name > Process Definition > Environment
Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP |
true |
IBM_HEAP_DUMP |
true |
IBM_HEAPDUMPDIR |
your_directory |
IBM_HEAPDUMP_OUTOFMEMORY |
true |
IBM_JAVADUMP_OUTOFMEMORY |
true |
IBM_JAVA_HEAPDUMP_TEXT |
true |
|
 |
- Make sure that you save changes to the master configuration.
- Enabling HeapDumps for all V5.0 and V5.1.0
releases:
- To set the following Environment Entries from the administrative
console, select:
Servers > Application Servers >
server_name > Process Definition > Environment
Entries > New
- Add the following Name and Value pairs:
IBM_HEAPDUMP |
true |
IBM_HEAP_DUMP |
true |
IBM_HEAPDUMPDIR |
your_directory |
IBM_HEAPDUMP_OUTOFMEMORY |
true |
IBM_JAVADUMP_OUTOFMEMORY |
true |
|
 |
- Make sure that you save changes to the master configuration.
- Enabling HeapDumps for all V3.5 and 4.0 releases:
To enable your heap dump, add the following lines to the top of the
startupServer.sh file in the
install_root/bin directory:
export IBM_HEAPDUMP=true
export IBM_HEAP_DUMP=true
export IBM_HEAPDUMP_OUTOFMEMORY=true
export IBM_HEAPDUMPDIR=directory_path |
|
|
Notes regarding HeapDumps:
- The heap dump is written to your current directory,
install_root/bin, or the directory specified by the
IBM_HEAPDUMPDIR environment variable.
- The file that is generated is called
"heapdump<pid>.time in internal
format.txt". This file can be very large. Allow for at least the
maximum heap size. If the maximum heap is set to 512MB, allow for .5GB for
the dump.
- This mechanism uses the same signal handler as javacore
processing, so you must not have DISABLE_JAVADUMP set as an
environment variable.
|
|
- Follow instructions to enable
verbosegc in WebSphere Application Server.
- Delete all Application Server and FFDC log files, if possible. Back
them up first, if needed.
- Restart the Application Server.
- Run the Linux_memory_leak.sh (see attached script) shell
script while trying to reproduce your problem. Ensure that you update the
following lines of the script before running it:
log=file
pid=PID of java process that leaks memory |
|
- Change the amount of time the script sleeps ("sleep 600" in the
script) based upon how long it takes to reproduce your problem.
For example, if the problem can be reproduced in 1 hour, update the sleep
interval to 300 so that the document is collected every 5 minutes
instead of every 10 minutes. For a problem that takes 2 days to reproduce,
1200 is a good sleep interval.
- Wait for the OutOfMemoryError condition to occur. This causes the
generation of a heapdump.
- If you want to analyze the heapdump yourself, download the IBM HeapAnalyzer
tool. If you do not want to analyze this yourself, proceed to step 8.
- Send the following files to IBM support:
- For WebSphere Application Server V6.0:
- The server.xml file located in the following
directory:
install_root/profiles/profile_name/config/cells/
cell_name/nodes/node_name/servers/server_name |
|
- Everything in the following directory:
install_root/profiles/profile_name/logs/server_name |
|
Specifically, the following logs are needed:
- systemErr
- systemOut
- native_stderr
- native_stdout
|
|
- Everything in the following directory:
install_root/profiles/profile_name/logs/ffdc |
|
- All documents requested For all releases below.
- For WebSphere Application Server V5.0 and V5.1:
- The server.xml file located in the following
directory:
install_root/config/cells/nodes/node_name/
servers/server_name |
|
- Everything in the following directory:
install_root/logs/server_name |
|
Specifically, the following logs are needed:
- systemErr
- systemOut
- native_stderr
- native_stdout
|
|
- Everything in the following directory:
- All documents requested For all releases below.
- For WebSphere Application Server V3.5 and V4.0:
- A XMLConfig full export.
- Everything in the following directory from the WebSphere
Application Server system:
install_root/logs |
|
Specifically, the following logs are needed:
- stderr
- stdout
- tracefile
- nanny.trace
|
|
- All documents requested For all releases below.
- For all releases
- All javacore.txt files that were created when
OutOfMemory conditions occurred. You can find them in the following
directories:
- install_root
- install_root/bin
- The configured working directory.
- All
heapdumppid.time in
internal format.txt files. Submit each one in a
separate ZIP or TAR file.
- List of all other software and version information on
the system. For example, any databases, WebSphere MQ®, and so forth.
- Follow instructions to send
diagnostic information to IBM support.
Note: JVM heap dump generation on OutOfMemory are enabled by
default startng from IBM SDK 1.3.1 SR5 and in all of the IBM SDK 1.4.x
releases. However, IBM recommends that you enable JVM HeapDumps settings
for all IBM SDK releases so that you can control where JVM heap dump is
written.
For a listing of all technotes, downloads, and educational materials
specific to the Out of Memory component, search the WebSphere
Application Server support site. |
|
|