MustGather: Out of Memory errors on AIX, part 2b - heap leak
 Technote (FAQ)
 
Problem
MustGather for heap leak problems with the WebSphere® Application Server Out of Memory component. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Solution
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on an AIX® platform when you suspect a memory leak in the Java™ heap. This suspicion is based upon your analysis of the documentation collected from the technote: MustGather: Out of Memory errors on AIX, Part 1.

If you have already contacted support, continue to the component specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Out of Memory (heap leak) specific MustGather information
  1. Enabling HeapDumps:
    • WebSphere Application Server V6.0 release:
      1. To set the following Environment Entries from the administrative console, select the following:

        WebSphere Application Server V6.0 and V6.0.1

        Servers > Application Servers >
        server_name> Java and Process Management > Process Definition > Environment Entries > New

        WebSphere Application Server V6.0.2 and higher

        Servers > Application servers >
        server_name > Java and Process Management > Process Definition > Custom Properties > New

      2. Add the following Name and Value pairs:

        IBM_HEAPDUMP             true
        IBM_HEAP_DUMP            true
        IBM_HEAPDUMPDIR
                 your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true
        IBM_JAVA_HEAPDUMP_TXT    true
        IBM_JAVACOREDIR          
        your_directory

        Important: Make sure that you save changes to the master configuration.

    • WebSphere Application Server V5.1.1 releases:
      1. To set the following Environment Entries from the administrative console, select the following:

        Servers > Application Servers >
        server_name> Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:

        IBM_HEAPDUMP             true
        IBM_HEAP_DUMP            true
        IBM_HEAPDUMPDIR
                 your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true
        IBM_JAVA_HEAPDUMP_TXT    true
        IBM_JAVACOREDIR          
        your_directory

        Important: Make sure that you save changes to the master configuration.

    • WebSphere Application Server V5.0, V5.1, V4.0 and V3.5 releases:

      To enable the heapdump, add the following lines to the top of the startupServer.sh file in the install_root/bin directory:

      export IBM_HEAPDUMP=true
      export IBM_HEAP_DUMP=true
      export IBM_HEAPDUMP_OUTOFMEMORY=true
      export IBM_JAVADUMP_OUTOFMEMORY=true
      export IBM_HEAPDUMPDIR=
      directory_path
      export IBM_JAVACOREDIR=directory_path

    • Notes:
      • The heapdump will be written to the current directory, install_root/bin, or the directory specified by the IBM_HEAPDUMPDIR environment variable.

      • The file that is generated is called heapdumppid.time in internal format.txt. This file can be very large; allow for at least the maximum heap size. If the maximum heap is set to 512 MB, allow for 0.5 GB for the dump.

      • The javacore file will be written to the current directory, install_root/ bin, or the directory specified by the IBM_JAVACOREDIR environment variable.

      • This mechanism uses the same signal handler as javacore processing, so you must not have DISABLE_JAVADUMP set as an environment variable.

      • Verify that you have sufficient file system space to produce the Heapdumps. Heapdump files can range in size from 200MB to 2GB.

  2. Follow instructions to Enable verbosegc is WebSphere Application Server

  3. Set the ulimit -f value to unlimited so it will not truncate the Heapdump file.

  4. Restart the Application Server.

  5. After starting WebSphere Application Server, run the AIX_memory_leak.sh (select appropriate attached script depending on your version of AIX) shell script while trying to reproduce the problem:

    AIX_memory_leak.sh -p pid -f base_filename -i interval

    Where:
    pid is the process id to be profiled
    base_filename is a path and identifier for the output
    interval is the time in seconds between each iteration

    For Example:
    AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120

  6. Wait for the OutOfMemoryError condition to occur. This causes the generation of a heapdump.

  7. If you want to analyze the heapdump yourself, download the HeapRoots tool. If you do not want to analyze this yourself, proceed to next step

  8. Collect the following:
    • For WebSphere Application Server V6.0 release:
      • All files in install_root/profiles/profile_name/logs/server_name directory. If you have configured the Application Server to write logs into a different location, then please send them accordingly.
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • startServer.log
      • Server.xml located in install_root/profiles/profile_name/config/cells/
        cell_name
        /nodes/node_name/servers/server_name directory.
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • systemErr.log, systemOut.log, native_stderr.log, native_stdout.log and startServer.log
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • Server.xml from the directory:
        install_root/config/cells/cell_name/nodes/
        node_name
        /servers/server_name
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

    • For WebSphere Application Server V4.0 and V3.5 releases:
      • stderr, stdout, tracefile and nanny.trace
      • All heapdumppid.time in internal format.txt files with each one in its own ZIP or TAR file.
      • All javacorepid.time in internal format.txt files
      • Full XMLConfig export.
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

  9. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
 
Related information
Mustgather: Out of Memory errors on AIX - Part 1
How to enable verbosegc in WebSphere
HeapRoots Tool for Analyzing JVM Memory Usage
Submitting information to IBM support
Steps to getting support
MustGather: Readme first
Troubleshooting guide
 
AIX_433_memory_leak.shAIX_51_memory_leak.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Out of Memory
Operating system(s): AIX
Software version: 4.0
Software edition:
Reference #: 1138457
IBM Group: Software Group
Modified date: Nov 30, 2005