MustGather: 100% CPU usage on Linux
 Technote (troubleshooting)
 
Problem(Abstract)
MustGather for problems where IBM® WebSphere® Application Server Java™ Virtual Machine (JVM™) processes experience 100% CPU Usage on Linux®. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
 
Resolving the problem
If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

100% CPU usage specific MustGather information

Perform the following setup instructions:
  1. Follow instructions to enable verbosegc in WebSphere Application Server

  2. Run the following command:

    top -d delaytime -c -b > top.log

    Where delaytime is the number of seconds to delay. This must be 60 seconds or greater, depending on how soon the failure is expected.

  3. Run the following scripts:

    ./ps.sh ps_eLf.log
    ./vmstat.sh vmstat.log


    Notes:
    . The scripts ps.sh and vmstat.sh, as provided, roll over every 24 hours.
    . You might need to modify the scripts to meet your needs.
    . The preceding scripts will run forever. After the error condition is reached, you will have to kill them manually.

  4. When the problem occurs, collect the following logs:
    Run the following:

    netstat -an > netstat1.out

  5. If the Web server is remote, run the following on the Web server system:

    netstat -an > netstatwebserver1.out

  6. Run the following:

    kill -3 [PID_of_problem_JVM]

    The kill -3 commands create javacore*.txt files or javacore data written to the stderr file of the Application Server.

    Note: If you are not able to determine which JVM process is experiencing the high CPU usage then you should issue the kill -3 PID for each of the JVM processes.

  7. Wait two minutes.

  8. Run the following:

    kill -3 [PID_of_problem_JVM]

  9. Wait two minutes.

  10. Run the following:

    kill -3 [PID_of_problem_JVM]

  11. Wait two minutes.

  12. Run the following:

    netstat -an > netstat2.out  

  13. If the Web server is remote, run the following on the Web server system:

    netstat -an > netstatwebserver2.out

  14. If you are unable to generate javacore files, then perform the following:
    • Run the following:

      kill -11 [PID_of_problem_JVM]

      The kill -11 will terminate the JVM process, produce a core file, and possibly a javacore.

    • Also, follow steps in MustGather: Crash on Linux. Steps to format a core file are included.

  15. Collect the following documentation:
    • All Application Server JVM log files for the Application Server experiencing the problem.
    • All administrative server log files from the machine experiencing the problem.
    • WebSphere Application Server plug-in log
    • Web server error and access log
    • top.log, ps_eLf.log and vmstat.log
    • javacore*.*
    • All netstat*.out files
    • /var/log/messages
    • Indicate which JVM, such as the Application Server or administrative server, is experiencing the problem.

  16. Follow instructions to send diagnostic information to IBM support.

Problem determination documentation to help you resolve your problem:
For a listing of all technotes, downloads, and educational materials specific to the 100% CPU Usage component, search the WebSphere Application Server support site.
 
Related information
How to enable verbosegc
Steps for getting support
MustGather: Readme first
Troubleshooting Guide
Submitting information to IBM support
 
vmstat.sh ps.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > 100% CPU Usage
Operating system(s): Linux
Software version: 6.1
Software edition:
Reference #: 1132699
IBM Group: Software Group
Modified date: May 14, 2008