MustGather: Out of Memory errors on Solaris - Native Leak
 Technote (FAQ)
 
Problem
MustGather for problems java.lang.OutOfMemoryError on Solaris platform. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
 
Cause
Two conditions can cause a java.lang.OutOfMemoryError error:
  • The Java™ Virtual Machine (JVM) runs out of Java heap space to allocate a Java object.
  • The JVM is not be able to allocate the native memory that it needs to create or execute threads.
 
Solution
Native memory leak can be a result of a local process address space exhaustion or a system-wide virtual-memory exhaustion of physical memory
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on an Solaris platform when you suspect that there is a native memory leak
  1. Enable verbosegc. For details, see How to enable verbosegc on WebSphere Application Server.

  2. Stop the application server and clear the logs.

  3. Start the application server.

  4. Run the Solaris_ps.sh and Solaris_pmap.sh scripts (see attached scripts).

    These scripts will collect data at specified intervals. The frequency of these intervals are determined by the number you pass to the scripts. This number is the number of seconds for each interval.

    If the problem can be reproduced in 1 hour, pass the script 300 so that the documentation is collected every 5 minutes. For a problem that takes 2 days to reproduce, 1200 is a good sample period.

  5. Wait for problem to re-occur.

  6. Execute the following script (see attached) where PID is the process ID of the Java process.

    ./wsasprocess.sh  PID

  7. Stop the Application Server and the script.

  8. Obtain the kernel message logs, normally located as /var/adm/messages. (Older message logs often have '.0', '.1', '.2', ... etc. appended to the name of the log file should additional log files be needed.)

  9. Collect the following:
    • For WebSphere Application Server Version 6.0 and 6.1:
      1. All files in install_root/profiles/profile_name/logs/server_name directory.
      2. A copy of server.xml located in install_root/profiles/profile_name/config/cells/cell_name/nodes
        /
        node_name/servers/server_name directory
      3. All jtc*.log files created from running the scripts in Step 4
      4. All of the *.out files created by the wascpu.sh script in Step 6
    • For WebSphere Application Server Version 5.0 and 5.1:
      1. Include all of the files from the install_root/logs/server_name directory.
      2. A copy of server.xml located at install_root/config/cells/cell_name
        /nodes/
        node_name/servers/server_name directory
      3. All jtc*.log files created from running the scripts in Step 4
      4. All of the *.out files created by the wascpu.sh script in Step 6
  10. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
 
Related information
Out of Memory Error on Solaris - Heap Leak
How to enable verbosegc in WebSphere
Tuning Garbage Collection
Submitting information to IBM support
Steps to getting support
MustGather: Readme first
Troubleshooting guide
 
Solaris_ps.shSolaris_pmap.shwsasprocess.sh
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Out of Memory
Operating system(s): Solaris
Software version: 6.1
Software edition:
Reference #: 1104470
IBM Group: Software Group
Modified date: Feb 14, 2007