MustGather: Out of memory errors on Linux, part 2b - heap leak

Technote (FAQ)
Problem
MustGather for problems with the WebSphere® Application Server (all versions) Out of Memory component on Linux®. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Solution
This document describes how to troubleshoot java.lang.OutOfMemoryError errors on a Linux platform. Your suspicion of a memory leak in the Java™ HEAP is based on your analysis of the documentation collected from MustGather: Out of Memory Errors on Linux, Part 1.

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Out of Memory (Linux) specific MustGather information
Debugging the Java Virtual Machine (JVM™) running out of Java heap:
  1. Enable HeapDumps
    • Enabling HeapDumps for all V6.0 release:
      1. Select Servers> Application Servers>server_name> Java and Process Management > Process Definition> Environment Entries > New to set the following Environment Entries from the administrative console

      2. Add the following Name/Value pairs:
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR
        your_directory
        IBM_HEAPDUMP_OUTOFMEMORYtrue
        IBM_JAVADUMP_OUTOFMEMORY true

      3. Make sure that you save changes to the master configuration.

    • Enabling HeapDumps for all V5.0 and V5.1 releases:
      1. Select Servers> Application Servers>server_name> Process Definition> Environment Entries > New to set the following Environment Entries from the administrative console

      2. Add the following Name/Value pairs:
        IBM_HEAPDUMP true
        IBM_HEAP_DUMP true
        IBM_HEAPDUMPDIR
        your_directory
        IBM_HEAPDUMP_OUTOFMEMORYtrue
        IBM_JAVADUMP_OUTOFMEMORY true

      3. Make sure that you save changes to the master configuration.

    • Enabling HeapDumps for all V3.5 and 4.0 releases:

      To enable your heap dump, add the following two lines to the top of the startupServer.sh file in the install_root/bin directory:

      export IBM_HEAPDUMP=true
      export IBM_HEAP_DUMP=true
      export IBM_HEAPDUMP_OUTOFMEMORY=true
      export IBM_HEAPDUMPDIR=
      directory_path

    • Notes regarding HeapDumps:
      The heap dump is written to your current directory, install_root/bin, or the directory specified by the IBM_HEAPDUMPDIR environment variable.

      The file that is generated is called "heapdump<pid>.timein internal format.txt". This file can be very large. Allow for at least the maximum heap size. If the maximum heap is set to 512MB, allow for .5GB for the dump.

      This mechanism uses the same signal handler as javacore processing, so you must not have DISABLE_JAVADUMP set as an environment variable.

  2. Follow instructions to Enable verbosegc is WebSphere Application Server

  3. Delete all Application Server and FFDC log files, if possible. Back up them first, if needed.

  4. Restart the Application Server.

  5. Run the Linux_memory_leak.sh (see attached script) shell script while trying to reproduce your problem. Ensure that you update the following lines of the script before running it:

    log=file
    pid=PID of java process that leaks memory

  6. Change the amount of time the script sleeps ("sleep 600" in the script) based upon how long it takes to reproduce your problem.

    For example, if the problem can be reproduced in 1 hour, update the sleep interval to 300 so that the document is collected every 5 minutes instead of every 10 minutes. For a problem that takes 2 days to reproduce, 1200 is a good sleep interval.

  7. Wait for the OutOfMemoryError condition to occur. This causes the generation of a heapdump.

  8. If you want to analyze the heapdump yourself, download the IBM HeapAnalyzer tool. If you do not want to analyze this yourself, proceed to step 8.

  9. Send the following files to IBM support:

    • For WebSphere Application Server V6.0 releases
      • The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
      • Everything in the install_root/profiles/profile_name/logs/server_name directory, specifically systemErr, systemOut, native_stderr and native_stdout logs are needed
      • Everything in the install_root/profiles/profile_name/logs/ffdc directory
      • All documents requested For all releases below
    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root/config/cells/nodes/node_name/servers/server_name directory, specifically systemErr, systemOut, native_stderr and native_stdout logs are needed
      • Everything in the install_root/logs/server_name directory
      • Everything in the install_root/logs/ffdc directory
      • All documents requested For all releases below
    • For WebSphere Application Server V3.5 and V4.0 release:
      • A XMLConfig full export.
      • Everything in the install_root/logs directory from the WebSphere Application Server system, specifically stderr and stdout, tracefile and nanny.trace
      • All documents requested For all releases below
    • For all releases
      • All javacore.txt files that were created when OutOfMemory conditions occurred. You can find them in the install_root, install_root/bin or in the configured working directory
      • All heapdumppid.time in internal format.txt files
        Submit each one in a separate ZIP or TAR file.
      • List of all other software and version information on the system. For example, any databases, WebSphere MQ®, and so forth.

10. Follow instructions to send diagnostic information to IBM support.


Notes:
  • JVM heap dump generation on OutOfMemory are enabled by default startng from IBM SDK 1.3.1 SR5 and in all of the IBM SDK 1.4.x releases. However, IBM recommends that you enable JVM HeapDumps settings for all IBM SDK releases so that you can control where JVM heap dump is written.
Related information
Out of Memory Errors on Linux - Part 1

Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): Linux
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server, Standard
Reference #: 1138586
IBM Group: Software Group
Modified date: 2004-12-15