MustGather: Out of Memory errors on AIX, Part 1
 Technote (FAQ)
 
Problem
MustGather for Out of Memory errors with WebSphere® Application Server on an AIX® platform. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
 
Cause
The following describes how to troubleshoot java.lang.OutOfMemoryError errors on an AIX platform. Two conditions can cause a java.lang.OutOfMemoryError:
  • The Java™ virtual machine (JVM™) might run out of Java heap space to allocate a Java object.

  • The JVM might not be able to allocate the native memory that it needs to execute the threads.
 
Solution
If you have already contacted support, continue to the Out of Memory MustGather information. Otherwise, refer to MustGather: Read first for all WebSphere Application Server products.

Out of Memory errors specific MustGather information
Follow these instructions to set up the failing system for capturing required information:
  1. Follow instructions for Enabling verbosegc in WebSphere Application Server.

  2. Stop the Application Server. Clear all application server log files before starting the test. Restart the application server.

  3. After starting the application server, run the AIX_433_memory_leak.sh or AIX_51_memory_leak.sh shell script (see attached) while trying to reproduce the problem (select the appropriate script depending on your version of AIX):

    AIX_memory_leak.sh -p pid -f base_filename -i interval

    Where:
    pid is the process id that runs out of memory
    base_filename is a path and identifier for the output
    interval is the time in seconds between each iteration

    For example:
    AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120

  4. Wait for the problem to re-occur.

  5. Stop the Application Server and the script.

  6. Collect the following:
    • For WebSphere V6.0:
      1. All files in install_root/profiles/profile_name/logs/server_name.
      2. All files in install_root/profiles/profile_name/logs/ffdc.
      3. A copy of server.xml located in install_root/profiles/
        profile_name
        /config/cells/cell_name/nodes/node_name
        /servers/
        server_name.

    • For WebSphere V5.0 and V5.1:
      1. All files in install_root/logs/server_name directory.
      2. All files in install_root/logs/ffdc.
      3. A copy of server.xml located at install_root/config/cells/
        cell_name
        /nodes/node_name/servers/server_name.

    • For WebSphere V3.5 and V4.0:
      1. All files in install_root/logs.
      2. A copy of XMLExport for the server configuration.

    • For all WebSphere releases:
      • The log file output from memory leak script.
      • All heapdump*.* and javacore*.* files, if any were generated by the Out Of Memory condition.
      • Note: For all versions, If you have configured the application server to write logs into a different location, send them accordingly.

  7. Determine if the source for the leak is the Java heap or native memory:
    1. If available memory does NOT drop, follow the instructions in
      MustGather: Out of Memory errors on AIX, Part 2a - Native Leak

    2. If available memory listed in the heap drops over time, follow the instructions in
      MustGather: Out of Memory errors on AIX, Part 2b - Heap Leak

  8. Follow the instructions in the following Technote: Submitting Diagnostic Information to IBM Technical Support for Problem Determination.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
 
Related information
MustGather: Out of Memory errors on AIX; part 2a
MustGather: Out of Memory errors on AIX; part 2b
Enabling verbosegc in WebSphere Application Server
JVM Diagnostics Guide
Submitting Diagnostic Information to Technical Support
Steps to get support for WebSphere Application Server
MustGather: Read first for all WebSphere
Troubleshooting guide
 
AIX_433_memory_leak.shAIX_51_memory_leak.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Out of Memory
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1084067
IBM Group: Software Group
Modified date: Apr 5, 2006