MustGather: Out of Memory errors on AIX, part 1

Technote (FAQ)
Problem
MustGather for Out of Memory errors with WebSphere® Application Server on AIX®. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
Solution
The following describes how to troubleshoot java.lang.OutOfMemoryError errors on an AIX platform. Two conditions can cause a java.lang.OutOfMemoryError:
  • The Java™ virtual machine (JVM™) might run out of Java heap space to allocate a Java object.
  • The JVM might not be able to allocate the native memory that it needs to execute the threads.

If you already contacted support, continue to the Out of Memory MustGather information. Otherwise, click: MustGather: Read First for all WebSphere Application Server Products.

Follow the below instructions to setup the failing system for capturing required information:
  1. Follow instructions for Enabling verbosegc in WebSphere Application Server
  2. Stop the Application Server. Clear all application server log files before starting test. Restart the application server.
  3. After starting the application server, run the AIX_433_memory_leak.sh or AIX_51_memory_leak.sh shell script while trying to reproduce the problem (select the appropriate attached script depending on your version of AIX).
    AIX_memory_leak.sh -p pid-f base_filename-i interval
    Where:
    pid is the process id that runs out of memory
    base_filename is a path and identifier for the output
    interval is the time in seconds between each iteration

    For example:
    AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120
  4. Wait for the problem to re-occur.
  5. Stop the Application Server and the script.
  6. Collect the following:
    • For release V6.0:
      1. All files in install_root/profiles/profile_name/logs/server_name directory.
      2. All files in install_root/profiles/profile_name/logs/ffdc directory.
      3. A copy of server.xml located in install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_namedirectory
    • For release V5.0 and V5.1:
      1. Include all of the files from the install_root/logs/server_namedirectory.
      2. All files in install_root/logs/ffdc directory.
      3. A copy of server.xml located at install_root/config/cells/cell_name/nodes/node_name/servers/server_name directory
    • For releases V3.5 and V4.0:
      1. Include all of the files from the install_root/logs directory.
      2. A copy of XMLExport for the server configuration
    • For all releases:
      • The log file output from memory leak script
      • All heapdump*.* and javacore*.* files if any were generated by the Out Of Memory condition.
      • Note: For all versions, If you have configured the application server to write logs into a different location, send them accordingly.

  7. Determine if source for leak is the Java heap or native memory:
    1. If available memory does NOT drop, follow instructions in
      MustGather: Out of Memory errors on AIX, part 2a - native leak

    2. If available memory listed in heap drops over time, follow instructions in
      MustGather: Out of Memory errors on AIX, part 2b - heap leak

  8. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.

Related information
MustGather: Out of Memory errors on AIX; part 2a

MustGather: Out of Memory errors on AIX; part 2b

How to enable verbosegc

JVM Diagnostics Guide

Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server, Standard
Reference #: 1084067
IBM Group: Software Group
Modified date: 2004-12-15