MustGather: Out of memory errors on Linux, part 2b - heap leak
 Technote (FAQ)
 
Problem
MustGather for problems with the WebSphere® Application Server (all versions) Out of Memory component on the Linux® platform. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
 
Solution
This document describes how to troubleshoot java.lang.OutOfMemoryError errors on a Linux platform. Your suspicion of a memory leak in the Java™ heap is based on your analysis of the documentation collected from MustGather: Out of Memory errors on Linux, Part 1.

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Out of Memory (heap leak) specific MustGather information
Debugging the Java virtual machine (JVM™) running out of Java heap:
  1. Enable HeapDumps:
    • Enabling HeapDumps for all V6.0 release:
      1. To set the following Environment Entries from the administrative console, select:

        Servers > Application Servers > server_name > Java and Process Management > Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:

        Name Value
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true
        IBM_JAVA_HEAPDUMP_TEXT true

      3. Make sure that you save changes to the master configuration.

    • Enabling HeapDumps for all V5.1.1 releases:
      1. To set the following Environment Entries from the administrative console, select:

        Servers > Application Servers > server_name > Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:

        Name Value
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true
        IBM_JAVA_HEAPDUMP_TEXT true

      3. Make sure that you save changes to the master configuration.

    • Enabling HeapDumps for all V5.0 and V5.1.0 releases:
      1. To set the following Environment Entries from the administrative console, select:

        Servers > Application Servers > server_name > Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:

        Name Value
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR your_directory
        IBM_HEAPDUMP_OUTOFMEMORY true
        IBM_JAVADUMP_OUTOFMEMORY true

      3. Make sure that you save changes to the master configuration.

    • Enabling HeapDumps for all V3.5 and 4.0 releases:

      To enable your heap dump, add the following lines to the top of the startupServer.sh file in the install_root/bin directory:

      export IBM_HEAPDUMP=true
      export IBM_HEAP_DUMP=true
      export IBM_HEAPDUMP_OUTOFMEMORY=true
      export IBM_HEAPDUMPDIR=
      directory_path

    Notes regarding HeapDumps:
    • The heap dump is written to your current directory, install_root/bin, or the directory specified by the IBM_HEAPDUMPDIR environment variable.

    • The file that is generated is called "heapdump<pid>.time in internal format.txt". This file can be very large. Allow for at least the maximum heap size. If the maximum heap is set to 512MB, allow for .5GB for the dump.

    • This mechanism uses the same signal handler as javacore processing, so you must not have DISABLE_JAVADUMP set as an environment variable.

  2. Follow instructions to enable verbosegc in WebSphere Application Server.

  3. Delete all Application Server and FFDC log files, if possible. Back them up first, if needed.

  4. Restart the Application Server.

  5. Run the Linux_memory_leak.sh (see attached script) shell script while trying to reproduce your problem. Ensure that you update the following lines of the script before running it:

    log=file
    pid=PID of java process that leaks memory

  6. Change the amount of time the script sleeps ("sleep 600" in the script) based upon how long it takes to reproduce your problem.

    For example, if the problem can be reproduced in 1 hour, update the sleep interval to 300 so that the document is collected every 5 minutes instead of every 10 minutes. For a problem that takes 2 days to reproduce, 1200 is a good sleep interval.

  7. Wait for the OutOfMemoryError condition to occur. This causes the generation of a heapdump.

  8. If you want to analyze the heapdump yourself, download the IBM HeapAnalyzer tool. If you do not want to analyze this yourself, proceed to step 8.

  9. Send the following files to IBM support:
    • For WebSphere Application Server V6.0:
      • The server.xml file located in the following directory:

        install_root/profiles/profile_name/config/cells/
        cell_name
        /nodes/node_name/servers/server_name

      • Everything in the following directory:

        install_root/profiles/profile_name/logs/server_name

        Specifically, the following logs are needed:
        • systemErr
        • systemOut
        • native_stderr
        • native_stdout

      • Everything in the following directory:

        install_root/profiles/profile_name/logs/ffdc

      • All documents requested For all releases below.

    • For WebSphere Application Server V5.0 and V5.1:
      • The server.xml file located in the following directory:

        install_root/config/cells/nodes/node_name/
        servers/
        server_name

      • Everything in the following directory:

        install_root/logs/server_name

        Specifically, the following logs are needed:
        • systemErr
        • systemOut
        • native_stderr
        • native_stdout

      • Everything in the following directory:

        install_root/logs/ffdc

      • All documents requested For all releases below.

    • For WebSphere Application Server V3.5 and V4.0:
      • A XMLConfig full export.

      • Everything in the following directory from the WebSphere Application Server system:

        install_root/logs

        Specifically, the following logs are needed:
        • stderr
        • stdout
        • tracefile
        • nanny.trace

      • All documents requested For all releases below.

    • For all releases
      • All javacore.txt files that were created when OutOfMemory conditions occurred. You can find them in the following directories:
        • install_root
        • install_root/bin
        • The configured working directory.

      • All heapdumppid.time in internal format.txt files. Submit each one in a separate ZIP or TAR file.

      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

  10. Follow instructions to send diagnostic information to IBM support.


Note: JVM heap dump generation on OutOfMemory are enabled by default startng from IBM SDK 1.3.1 SR5 and in all of the IBM SDK 1.4.x releases. However, IBM recommends that you enable JVM HeapDumps settings for all IBM SDK releases so that you can control where JVM heap dump is written.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
 
Related information
Out of Memory Errors on Linux - Part 1
Submitting information to IBM support
Steps to getting support
MustGather: Readme first
Troubleshooting guide
 
Linux_memory_leak.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Out of Memory
Operating system(s): Linux
Software version: 6.0
Software edition:
Reference #: 1138586
IBM Group: Software Group
Modified date: Mar 31, 2006