MustGather: 100% CPU usage on Linux

Technote (FAQ)
Problem
MustGather for problems where WebSphere® Application Server Java™ Virtual Machine (JVM™) processes experience 100% CPU Usage on Linux®. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Solution
If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

100% CPU usage specific MustGather information

Set up:

  1. Follow instructions to Enable verbosegc is WebSphere Application Server
  2. top -d delaytime-C -c -b > top.log
    where delaytime = number of seconds to delay. This must be 60 seconds or greater, depending on how soon the failure is expected.
  3. Run the following scripts:
    ./ps.sh ps.log
    ./vmstat.sh vmstat.log

    Notes:
    • The scripts ps.sh and vmstat.sh roll over every 24 hours.
    • You might need to modify the scripts to meet your needs.

When the problem occurs, collect the following logs:
  1. netstat -an > netstat1.out

  2. If the Web server is remote then do the following on Web server system:
    netstat -an > netstatwebserver1.out

  3. ps -efH > ps_efh.txt

  4. kill -3 [PID_of_problem_JVM]
    These kill -3' commands createjavacore*.txt files or javacore data written to the stderr file of the Application Server.

    Note: If you are not able to determine which JVM process is experiencing the high CPU usage then you should issue the kill -3 PID for each of the JVM processes.

  5. Wait two minutes

  6. kill -3 [PID_of_problem_JVM]

  7. Wait two minutes

  8. kill -3 [PID_of_problem_JVM]

  9. Wait two minutes

  10. netstat -an > netstat2.out

  11. If the Web server is remote then do the following on Web server system:
    netstat -an > netstatwebserver2.out

  12. If you cannot get javacore files then perform the following:
    • kill -11 [PID_of_problem_JVM]

      The kill -11 will terminate the JVM process, produce a core file and possibly a javacore.
    • Also follow steps in MustGather: Crash on Linux. Steps to format a core file are included.

  13. Collect the following documentation:
    • All Application Server JVM log files for the Application Server experiencing the problem.
    • All administrative server log files from the machine experiencing the problem
    • WebSphere Application Server plug-in log.
    • Web server error and access log
    • top.log, ps.log and vmstat.log
    • ps_efh.txt
    • javacore*.*
    • All netstat*.out files
    • /var/log/messages
    • Indicate which JVM, such as the Application Server oradministrative server, is experiencing the problem.

  14. Follow instructions to send diagnostic information to IBM support.

Problem determination documentation to help you resolve your problem:

Related information
How to enable verbosegc

Steps for getting support

MustGather: Readme first

Troubleshooting Guide

Submitting information to IBM support












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, 100% CPU Usage
Operating system(s): Linux
Software version: 3.5, 4.0, 5.0, 5.1
Software edition: Advanced, Base, Network Deployment, Single Server
Reference #: 1132699
IBM Group: Software Group
Modified date: 2004-12-07