MustGather: Out of Memory Errors on Linux, Part 1

Technote (FAQ)
Problem
Troubleshooting java.lang.OutOfMemoryError errors on Linux® platforms.

Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Solution
Two conditions can cause a java.lang.OutOfMemoryError:
  • The Java™ Virtual Machine (JVM™) might run out of Java heap space to allocate a Java object.
  • The JVM might not be able to allocate the native memory that it needs to execute it's threads.

If you have already contacted support, continue to the Out of Memory MustGather information. Otherwise, click the following link: MustGather: Read First for all WebSphere Application Server Products.


Out of memory specific MustGather information

Follow these instructions for initial environment set up before you re-create OutOfMemory conditions on the Application Server:
  1. Follow instructions on How to enable verbosegc on WebSphere Application Server before recreating the problem. You will have to restart the WebSphere Application Server.

  2. Clean all WebSphere Application Server and FFDC logs, if possible. It will require WebSphere Application Sever and node agent restart.


At the next occurrence of the OutOfMemory conditions, perform the following steps on the Application Server system:

  1. After starting the Application Server, run the Linux_memory_leak.sh (see attached script) shell script while trying to reproduce the problem. Please ensure that you update the following lines of the script before running it:
    log=file
    pid=PID_of_leaking_java_process

    Also change the amount of time the script sleeps (the line sleep 600 in the script) based upon how long it takes to reproduce the problem.

    For example, if you reproduce the problem in 1 hour, update the sleep interval to 300 so that the doc is collected every 5 minutes instead of 10. For a problem that takes 2 days to reproduce, 1200 would be a good sleep interval.

  2. Wait for the problem to re-occur.
  3. Stop the Application Server and the script.
  4. Collect the following information:
    • For WebSphere Application Server V6.0 release:
      • The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
      • Everything in the install_root/profiles/profile_name/logs/server_name directory, specifically systemErr, systemOut, native_stderr and native_stdout logs are needed
      • Everything in the install_root/profiles/profile_name>/logs/ffdc directory
      • All documents requested For all releases below
    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root/config/cells/nodes/node_name/servers/server_name directory, specifically systemErr, systemOut, native_stderr and native_stdout logs are needed
      • Everything in the install_root/logs/server_name directory
      • Everything in the install_root/logs/ffdc directory
      • All documents requested For all releases below
    • For WebSphere Application Server V4.0 release:
      • A XMLConfig full export.
      • Everything in the install_root/logs directory from the WebSphere Application Server system, specifically stderr and stdout, tracefile and nanny.trace
      • All documents requested For all releases below
    • For WebSphere Application Server V3.5:
      • A XMLConfig full export.
      • Everything in the install_root/logs directory from the WebSphere Application Server system, specifically stderr and stdout, tracefile and nanny.trace files are needed
      • All documents requested For all releases below

    • For all releases
      • All javacore.txt files that were created when OutOfMemory conditions occurred. If javacore.txt files were created, you can find them in the install_root, install_root/bin or in the configured working directory
      • The log file from running the Linux_memory_leak.sh

      • Include the Application Server systemErr, systemOut, native_stderr, and native_stdout logs, if they are located in a different directory



5. Follow instructions to send diagnostic information to IBM support.
    Related information
    JVM Diagnostic Guide

    Submitting information to IBM support

    Steps to getting support

    MustGather: Readme first

    Troubleshooting guide












    Document Information

    Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
    Operating system(s): Linux
    Software version: 3.5, 4.0, 5.0, 5.1, 6.0
    Software edition: Advanced, Base, Network Deployment, Single Server
    Reference #: 1138587
    IBM Group: Software Group
    Modified date: 2004-12-15