MustGather: 100% CPU usage on Solaris
 Technote (troubleshooting)
 
Problem(Abstract)
Collecting data for problems with the WebSphere® Application Server or other Java™ Virtual Machine (JVM™) experiencing 100%, or unusually high, CPU usage on Solaris™ operating systems. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
 
Resolving the problem
The following document outlines what is needed to troubleshoot a JVM that causes the CPU utilization to reach 100%, or an unusually high percentage, on a Solaris system.

If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

100% CPU usage specific MustGather information
Follow these instructions to set up the failing System for capturing required information:
  1. Enable verbosegc for the failing application server. This is necessary for debugging High CPU issues and does not have significant impact on performance.

  2. Clear all application server log files before starting test. You will have to stop the Application Server to delete all of the Application Server log files and then you can start the Application Server.

  3. After starting the Application Server, run the following netstat and vmstat commands to capture typical system CPU and connection activity:

    % netstat -an > netstat_normal.out
    % vmstat 5 12 > vmstat_normal.out

Collect the following information when the application server process is experiencing the problem:
  1. Execute the following script (see attached) where PID is the process ID of the Java process causing high CPU.

    ./wascpu.sh  PID

  2. Collect the following files:
    • For WebSphere Application Server V6.0:
      • All files in the following directory:

        profile_root/logs/server_name

      • A copy of the server.xml file located in the following directory:

        profile_root/config/cells/cell_name/nodes/
        node_name
        /servers/server_name

    • For WebSphere Application Server V5.0 and V5.1:
      • All files in the following directory:

        install_root\logs\server_name

      • A copy of the server.xml file located in the following directory:

        install_root\config\cells\cell_name\nodes\
        node_name
        \servers\server_name

    • If the Web server is remote, send the http_plugin.log file from the Web server system.

    • All of the *.out files created by the wascpu.sh script.

    • netstat_normal.out and vmstat_normal.out (if possible).

  3. Follow instructions to submit information to IBM Technical Support for problem determination.

If the thread dumps are not getting generated for kill -3, you might want to try using the alternate threading library, see Changing to the alternate threading library on Solaris 8.

For a listing of all technotes, downloads, and educational materials specific to 100% CPU usage issues, search the WebSphere Application Server support site.
 
Related information
Enabling verbosegc in WebSphere Application Server
Determining which Java thread is consuming CPU
lsof download link
Steps to Getting Support
MustGather: Read first for all WebSphere
Troubleshooting Guide
Changing to the alternate threading library
 
wascpu.sh
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation Solaris 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > 100% CPU Usage
Operating system(s): Solaris
Software version: 6.0
Software edition:
Reference #: 1115625
IBM Group: Software Group
Modified date: Sep 19, 2008