MustGather: Out of Memory errors on AIX, part 2b - heap leak

Technote (FAQ)
Problem
MustGather for heap leak problems with the WebSphere® Application Server Out of Memory component. Gathering this information before calling® IBM support will help familiarize you with the troubleshooting process and save you time.
Solution
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on an AIX® platform when you suspect a memory leak in the Java™ heap. This suspicion is based upon your analysis of the documentation collected from the technote: MustGather: Out of Memory errors on AIX, Part 1.

If you have already contacted support, continue to the component specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Out of Memory (heap leak) specific MustGather information:
  1. Enabling HeapDumps:
    • WebSphere Application Server V6.0 release:
      1. To set the following Environment Entries from the administrative console,
      2. Select:
        Servers > Application Servers >
        server_name> Java and Process Management > Process Definition > Environment Entries > New
      3. Add the following Name and Value pairs:
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR
        your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true
      4. Important:Make sure that you save changes to the master configuration.

    • WebSphere Application Server V5.0 and V5.1 releases:
      1. To set the following Environment Entries from the administrative console

      2. Select:
        Servers > Application Servers >server_name> Process Definition > Environment Entries > New

      3. Add the following Name and Value pairs:
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR
        your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true

      4. Important:Make sure that you save changes to the master configuration.

    • WebSphere Application Server V4.0 and V3.5 releases:

      To enable the heapdump, add the following lines to the top of the startupServer.sh file in the install_root/bin directory:

      export IBM_HEAPDUMP=true
      export IBM_HEAP_DUMP=true
      export IBM_HEAPDUMP_OUTOFMEMORY=true
      export IBM_JAVADUMP_OUTOFMEMORY=true
      export IBM_HEAPDUMPDIR=
      directory_path

    • Notes:
      • The heapdump will be written to the current directory, install_root/bin, or the directory specified by the IBM_HEAPDUMPDIR environment variable.

      • The file that is generated is called heapdumppid.time in internal format.txt. This file can be very large; allow for at least the maximum heap size. If the maximum heap is set to 512 MB, allow for 0.5 GB for the dump.

      • This mechanism uses the same signal handler as javacore processing, so you must not have DISABLE_JAVADUMP set as an environment variable.

  2. Follow instructions to Enable verbosegc is WebSphere Application Server

  3. Restart the Application Server.

  4. After starting WebSphere Application Server, run the AIX_memory_leak.sh (select appropriate attached script depending on your version of AIX) shell script while trying to reproduce the problem:

    AIX_memory_leak.sh -p pid-f base_filename-i interval

    Where:
    pid is the process id to be profiled
    base_filename is a path and identifier for the output
    interval is the time in seconds between each iteration

    For Example:
    AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120

  5. Wait for the OutOfMemoryError condition to occur. This causes the generation of a heapdump.

  6. If you want to analyze the heapdump yourself, download the HeapRoots tool. If you do not want to analyze this yourself, proceed to next step

  7. Collect the following:
    • For WebSphere Application Server V6.0 release:
      • All files in install_root/profiles/profile_name/logs/server_name directory. If you have configured the Application Server to write logs into a different location, then please send them accordingly.
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • startServer.log
      • Server.xml located in install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_namedirectory.
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • systemErr.log, systemOut.log, native_stderr.log, native_stdout.log and startServer.log
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • Server.xml from the directory:
        install_root/config/cells/cell_name/nodes/node_name/servers/server_name
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

    • For WebSphere Application Server V4.0 and V3.5 releases:
      • stderr, stdout, tracefile and nanny.trace
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • Full XMLConfig export.
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

  8. Follow instructions to send diagnostic information to IBM support.
For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
Related information
Mustgather: Out of Memory errors on AIX - Part 1

How to enable verbosegc in WebSphere

HeapRoots Tool for Analyzing JVM Memory Usage

Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server
Reference #: 1138457
IBM Group: Software Group
Modified date: 2004-12-07