MustGather: 100% CPU usage on Windows
 Technote (troubleshooting)
 
Problem(Abstract)
Collecting data for problems with the IBM® WebSphere® Application Server 100% CPU usage component on Microsoft® Windows® Platforms. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.
 
Resolving the problem
If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


100% CPU Usage specific MustGather information.

This document lists the MustGather data that is needed to begin troubleshooting a Java™ Virtual Machine (JVM™) that causes 100% CPU utilization (or an unusually high percentage) on a Windows system.

Windows XP/NT®/2000 includes a performance viewer (part of the Microsoft management console) that can be used to determine which threads in a Java process are consuming the CPU.


Follow these instructions to setup the failing System for capturing required information:

  1. Follow the instructions to enable verbosegc for the failing Application Server. 
  2. Clear all application server log files before starting test. You may have to stop the Application Server to delete the files.
  3. Ensure userdump utility is installed. If it is not installed follow the steps below:
    1. Download the userdump tool. If the link does not work, then please search Microsoft website for "Oem3sr2.zip" and download it.
    2. Unzip Oem3sr2.zip file to any temp directory. For example: C:\Debuggers

Collect the following information when the problem is with the application server process:

  1. From the Start menu, select Programs > Administrative Tools > Performance Monitor. (Alternatively, you might find Start > Settings > Control Panel, double-click Administrative Tools, and double-click Performance to open the Performance Monitor tool). A new performance monitor window titled Performance will be opened.
  2. Performance data can be captured to a log file. The log file can be forwarded to a location for analysis and rerun through the performance monitor. From the performance monitor window, expand Performance logs and alerts, right-click Counter Logs, New Log Settings.
  3. Type a log setting name, then click OK. A new window titled with the name you provided will now be opened.
  4. Click Add to add counters on the window opened in the above step. A new window titled Add Counters will now be opened.
  5. In the Performance Object drop down, select Thread as Object.
  6. Click Select counter from List (default), then select the following counters:
    Counter = % Processor Time
    Counter = ID Process
    Counter = ID Thread
  7. Click Select instances from list and select all Java threads as shown in this example:



  8. Click Add to add new properties, then click Close.
  9. Logging starts immediately.The log file icon turns green.
  10. While the performance data is being logged, gather relevant Performance Degradation documentation. You can get instructions for gathering performance degradation by selecting the appropriate link for your Application Server Version:

    V5
    V6

    The above is a very important step. Support requires this information to determine the cause of the CPU consumption.
  11. Once Performance Degradation documentation is collected, end the Performance Monitor logging. From the Performance Monitor tool, right-click Log Name in the counter window, then click Stop. The log file to send to IBM is referenced in the Log File Name column.
  12. Generate 3 dumps of the Java process with an interval of 2 to 3 minutes. userdump.exe is part of the userdump utility that must be installed on your system. Look at the above section titled: Follow these instructions to setup the failing System for capturing required information.
    • userdump.exe <PID of Java process> <complete path and file name of the target dump>
    • Example: c:\userdump\userdump.exe 7144 c:\dumpdata\firstDump.dmp
    • Note: If the target path is not specified, then the output will go into the directory from where the command was issued.
  13. Collect the following files
    • The perfmon log
    • The javacore*.txt files generated (Look at step 10)
    • The three dump files generated from step 12.
    • For V6.0 release:
      • All files in <install_root>\profiles\<ProfileName>\logs\
        <ServerName>
        directory.
      • A copy of server.xml located in <install_root>\profiles\<ProfileName>
        \config\cells\
        <CellName>\nodes\<NodeName>\servers\<ServerName> directory.
    • For V5.0 and V5.1 releases:
      • Include all of the files from the <install_root>\logs\<ServerName> directory.
      • A copy of server.xml located at <install_root>\config\cells\<CellName>
        \nodes\
        <nodeName>\servers\<ServerName> directory
  14. Follow instructions to send diagnostic information to IBM support.

Analyzing the data

To review the log, open the performance log file using perfmon. "Perfmon" is a tool provided by Microsoft and shipped as part of the operating system. It is usually located in the system32 folder.

On the main screen titled "Performance", click the "View Log File Data" icon (cylinder shaped). This will open System Monitor View, where you should click on the "Log Files" radio button. Then click on "Add" to select the log file.

Now back on the main "perfmon" screen titled "Performance", click the plus ( + ) button. In the "Add Counters" window, select the following items:


Performance Object = Thread
Click on Radio button: "Select counters from list" and select the three listed counters:
Counter = % Processor Time
Counter = ID Process
Counter = ID Thread
And select all instances.

Then click on "Add" and close the "Add Counters" window.

On the main screen titled "Performance", click the "View Reports" icon to determine the thread ID that consumes the CPU

Convert the thread ID (3480) to hexadecimal using the Windows calculator

3480 in decimal = D98 in hex

In the javacore.txt file, locate the thread that has a native ID of 0xD98
For example:
"SoapConnectorThreadPool : 0" (TID:0x10A25A98, sys_thread_t:0x24A31B30, state:R, native ID:0xD98) prio=5

Review the stack of that thread in each javacore to determine the cause of the CPU usage.

For a listing of all technotes, downloads, and educational materials specific to the 100% CPU Usage issue, search the WebSphere Application Server support site.

 
Related information
Ten Steps to Get Support
Performance Degradation on WebSphere v5 and V6
How to Enable verbosegc for WebSphere Application Serve
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > 100% CPU Usage
Operating system(s): Windows
Software version: 6.0
Software edition:
Reference #: 1137447
IBM Group: Software Group
Modified date: Jun 9, 2008