MustGather: 100% CPU Usage on Windows Platforms

Technote (FAQ)
Problem
MustGather for problems with the WebSphere® Application Server 100% CPU Usage component on Windows® Platforms. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
Solution
If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

100% CPU Usage specific MustGather information

This document lists what is needed to begin troubleshooting a Java™ Virtual Machine (JVM™) that pegs your CPU to 100% (or an unusually high percentage) on a Windows® system.

Windows NT/2000 includes a performance viewer (part of the Microsoft management console) that can be used to determine which threads in a Java process are consuming the CPU.


Follow the below instructions to setup the failing System for capturing required information:

  1. Follow instructions to Enable verbosegc for the failing application server.
  2. Clear all application server log files before starting test. You may have to stop the application server to delete the files and then start them.

Collect the following information when the application server process is experiencing the problem:

  1. From the Start menu, select Programs >Administrative Tools>Performance Monitor. (Alternatively you might find Start >Settings> Control Panel, double-click Administrative Tools, and double-click Performance to open the Performance Monitor tool.)
  2. Performance data can be captured to a log file. The log file can be forwarded to a location for analysis and rerun through the performance monitor.
    From the performance monitor, expand Performance logs and alerts,
    right-click Counter Logs, New Log Settings.
  3. Type a log setting name, then click OK.
  4. Click Add to add counters.
  5. In the Performance Object drop down window, select Thread as Object.
  6. Click Select counter from List (default), then select the following counters:
    Counter = % Processor Time
    Counter = ID Process
    Counter = ID Thread
  7. Click Select instances from list and select all Java threads as shown in this example:


  8. Click Add to add new properties, then click Close.
  9. Logging starts immediately.The log file icon turns green.
  10. While the performance data is being logged, follow the relevant Performance Degradation link at the end of this document.This is a very important step. Support requires this information to determine the cause of the CPU consumption.
  11. Once Performance Degradation documentation is collected, end the performance logging, From the Performance Monitor tool. right-click Log Name in the counter window, then click Stop. The log file to send to IBM is referenced in the Log File Name column.
  12. Generate a user.dmp of the Java process. Issue the following command:
    drwtsn32 -p <PID of Java process>
  13. Collect the following files
    • The perfmon log
    • The javacore*.txt files generated
    • The user.dmp file generated from step 11. This is most likely located in the "C:\Documents and Settings\All Users\Documents\DrWatson" directory.
    • Remaining information referenced in the Performance Degradation documentation.
    • For V6.0 release:
      • All files in <install_root>/profiles/<ProfileName>/logs/<ServerName> directory.
      • A copy of server.xml located in <install_root>/profiles/<ProfileName>/config/cells/<CellName>/nodes/<NodeName>/servers/<ServerName>director
    • For V5.0 and V5.1 releases:
      • Include all of the files from the <install_root>\logs\<ServerName>directory.
      • A copy of server.xml located at <install_root>\config\cells\<CellName>\nodes\<nodeName>\servers\<ServerName> directory
    • For V3.5 and V4.0 releases:
      • Include all of the files from the <install_root>\logs directory.
      • A copy of XMLExport for the server configuration
  14. Follow instructions tosend diagnostic information to IBM support.

Analyzing the data

  1. To review the log, click the View Log File Data icon on the tool bar in the System Monitor View, and open the performance log file.
  2. Click the plus ( + ) button. In the Add To Chart window, select the following items:
    Object = Thread
    Counter = % Processor Time
    Counter = ID Process
    Counter = ID Thread
    And select all instances.
  3. Determine the thread ID that consumes the CPU

  4. Convert the thread ID (3480) to hexadecimal using the Windows®calculator

    3480 in decimal = D98 in hex

  5. In the javacore.txt file, locate the thread that has a native ID of 0xD98
    For example:
    "SoapConnectorThreadPool : 0" (TID:0x10A25A98, sys_thread_t:0x24A31B30, state:R, native ID:0xD98) prio=5
  6. Review the stack of that thread in each javacore to determine the cause of the CPU usage.

For a listing of all technotes, downloads, and educational materials specific to the100% CPU Usage component, search the WebSphere Application Server support site.

Related information
Ten Steps to Getting Support

Performance Degradation on WebSphere v3.5 and v4.0

Performance Degradation on WebSphere v5

How to Enable verbosegc for WebSphere Application Serve












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, 100% CPU Usage
Operating system(s): Windows
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Enterprise, Express, Network Deployment, Single Server
Reference #: 1137447
IBM Group: Software Group
Modified date: 2004-12-09