MustGather: Out of Memory errors on Linux, Part 1
 Technote (FAQ)
 
Problem
MustGather for Out of Memory errors with WebSphere® Application Server on Linux®. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Cause
Two conditions can cause a java.lang.OutOfMemoryError:
  • The Java™ Virtual Machine (JVM™) might run out of Java heap space to allocate a Java object.

  • The JVM might not be able to allocate the native memory that it needs to execute it's threads.
 
Solution
If you have already contacted support, continue to the Out of Memory MustGather information. Otherwise, refer to MustGather: Read first for all WebSphere Application Server products.


Out of Memory specific MustGather information
Follow these instructions for initial environment set up before you re-create OutOfMemory conditions on the Application Server:
  1. Follow the instructions in Enabling verbosegc on WebSphere Application Server before recreating the problem. You will have to restart the WebSphere Application Server.

  2. Clean all WebSphere Application Server and FFDC logs, if possible. It will require a WebSphere Application Sever and node agent restart.

At the next occurrence of the OutOfMemory conditions, perform the following steps on the Application Server system:
  1. After starting the Application Server, edit the attached shell script, Linux_memory_leak.sh, to update the following lines:

    log=file
    pid=PID_of_leaking_java_process

    Also change the amount of time the script sleeps (the line sleep 600 in the script) based upon how long it takes to reproduce the problem.

    For example, if you reproduce the problem in 1 hour, update the sleep interval to 300 so that the doc is collected every 5 minutes instead of 10. For a problem that takes 2 days to reproduce, 1200 would be a good sleep interval.

  2. Run the Linux_memory_leak.sh shell script.

  3. Wait for the problem to re-occur.

  4. Stop the Application Server and the script.

  5. Collect the following information:
    • For WebSphere V6.0:
      • The server.xml file located in install_root/profiles/profile_name/config/cells/
        cell_name
        /nodes/node_name/servers/server_name
      • Everything in install_root/profiles/profile_name/logs/server_name, specifically systemErr, systemOut, native_stderr and native_stdout.
      • Everything in install_root/profiles/profile_name/logs/ffdc.
      • All documents requested For all releases below.

    • For WebSphere V5.0 and V5.1:
      • The server.xml file located in the install_root/config/cells/nodes/node_name/servers/server_name directory, specifically systemErr, systemOut, native_stderr and native_stdout.
      • Everything in install_root/logs/server_name.
      • Everything in install_root/logs/ffdc.
      • All documents requested For all releases below.

    • For WebSphere V4.0:
      • A XMLConfig full export.
      • Everything in install_root/logs from WebSphere Application Server, specifically stderr and stdout, tracefile and nanny.trace
      • All documents requested For all releases below.

    • For WebSphere V3.5:
      • A XMLConfig full export.
      • Everything in install_root/logs from WebSphere Application Server, specifically stderr and stdout, tracefile and nanny.trace
      • All documents requested For all releases below.

    • For all WebSphere releases:
      • All javacore*.txt and heapdump*.txt files that were created when OutOfMemory conditions occurred. If javacore*.txt or heapdump*.txt files were created, you can find them in the install_root, install_root/bin or in the configured working directory
      • The log file from running the Linux_memory_leak.sh
      • Include the Application Server systemErr, systemOut, native_stderr, and native_stdout logs, if they are located in a different directory

  6. Follow instructions in Submitting Diagnostic Information to IBM Technical Support for Problem Determination.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
 
Related information
JVM Diagnostic Guide
Submitting information to IBM support
Steps to getting support
MustGather: Readme first
Troubleshooting guide
MustGather: Out of memory errors on Linux - Heap Leak
MustGather: Out of Memory errors on Linux - native leak
 
Linux_memory_leak.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Out of Memory
Operating system(s): Linux
Software version: 6.0
Software edition:
Reference #: 1138587
IBM Group: Software Group
Modified date: Apr 5, 2006