MustGather: 100% CPU Usage on AIX Platforms

Technote (FAQ)
Problem
MustGather for problems with the WebSphere® Application Server 100% CPU Usage component. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.

Solution
This list documents what is needed to troubleshoot a Java™ Virtual Machine (JVM™) that reaches 100% CPU utilization, or an unusually high percentage on an AIX® system.


If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Follow the below instructions to setup the failing System for capturing required information:

  1. After starting the Application Server JVM execute the following dbx script for each AppServer/clone having the problem:

    ./dbxtrace_aix.sh -a PID_appserver>dbx_startup_clone_name.out
  2. Follow instructions to Enable verbosegc for the failing application server.
  3. Edit vmstat_script.sh and ps_script.sh to modify the sleep interval. The default value of sleep interval is 5 minutes, which is goof if problem can be recreated in few hours. If it takes a week for the problem to occur, then increase the interval and how often the script will overwrite the output file.
  4. Execute the 2 scripts:
    ./vmstat_script.sh vmstat.out
    ./ps_script.sh ps.out
  5. Clear all application server log files before starting test. You may have to stop the application server to delete the files and then start them.

Collect the following information when the application server process is experiencing the problem:

  1. netstat -an > netstat1.out
  2. If the Web server is remote then do the following on Web server system:
    netstat -an > netstatwebserver1.out
  3. Execute script:
    ./tprof_ps.sh tprof_ps
  4. kill -3 [PID_of_problem_JVM]
    The kill -3 commands should create javacore.txt files in the working directory of the application server. By default, for WebSphere Application Server 3.5.x and 4.0.x, it is install_root/bin directory. For WebSphere Application Server 5.x and 6.x, it is install_root/ directory.
  5. Wait two minutes
  6. kill -3 [PID_of_problem_JVM]
  7. Wait two minutes
  8. kill -3 [PID_of_problem_JVM]
  9. Wait two minutes
  10. netstat -an > netstat2.out
  11. If the Web server is remote then do the following on Web server system:
    netstat -an > netstatwebserver2.out
  12. ./dbxtrace_aix.sh -a [PID_of_hung_application_server]> dbx-hung.out

  13. Collect the following files
    • For all releases of V6.0:
      • All files in <install_root>/profiles/<ProfileName>/logs/<ServerName> directory.
      • A copy of server.xml located in <install_root>/profiles/<ProfileName>/config/cells/<CellName>/nodes/<NodeName>/servers/<ServerName>director
    • For all releases of V5.x:
      • Include all of the files from the <install_root>\logs\<ServerName>directory.
      • A copy of server.xml located at <install_root>\config\cells\<CellName>\nodes\<nodeName>\servers\<ServerName> directory
    • For all releases of V3.0.2, V3.5, and V4.0:
      • Include all of the files from the <install_root>\logs directory.
      • A copy of XMLExport for the server configuration
    • If the Web server is remote, send the trace.log.Web_server_name.date (V3.5), native.log (V4.0), or http_plugin.log (V5.0 and V6.0) file from the Web server system.
    • dbxtrace.out
    • All javacore.txt files created
    • Output from vmstat_script.sh and ps_script.sh.
    • All files generated bytprof_ps.sh script. These files will be in the directory specified as a script parameter.
    • All netstat*.out files

  14. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the componentName component, search the WebSphere Application Server support site.
Related information
How to enable verbosegc

Steps to getting support

MustGather: Readme first

Troubleshooting guide













Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, 100% CPU Usage
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Edition Independent
Reference #: 1116458
IBM Group: Software Group
Modified date: 2004-12-09