If you have already contacted support, continue on to the
component-specific MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
100% CPU Usage specific MustGather information.
This document lists the MustGather data that is needed to begin
troubleshooting a Java™ Virtual Machine (JVM™) that causes 100% CPU
utilization (or an unusually high percentage) on a Windows system.
Windows XP/NT®/2000 includes a performance viewer (part of the
Microsoft management console) that can be used to determine which threads
in a Java process are consuming the CPU.
Follow these instructions to setup the failing System for capturing
required information:
- Follow the instructions to enable
verbosegc for the failing Application Server.
- Clear all application server log files before starting test. You may
have to stop the Application Server to delete the files.
- Ensure userdump utility is installed. If it is not installed follow
the steps below:
- Download
the userdump tool. If the link does not work, then please search
Microsoft website for "Oem3sr2.zip" and download it.
- Unzip Oem3sr2.zip file to any temp directory. For example:
C:\Debuggers
Collect the following information when the problem is with the
application server process:
- From the Start menu, select Programs > Administrative
Tools > Performance Monitor. (Alternatively, you might find
Start > Settings > Control Panel, double-click
Administrative Tools, and double-click Performance to open
the Performance Monitor tool). A new performance monitor window titled
Performance will be opened.
- Performance data can be captured to a log file. The log file can be
forwarded to a location for analysis and rerun through the performance
monitor. From the performance monitor window, expand Performance logs and
alerts, right-click Counter Logs, New Log Settings.
- Type a log setting name, then click OK. A new window titled with the
name you provided will now be opened.
- Click Add to add counters on the window opened in the above step. A
new window titled Add Counters will now be opened.
- In the Performance Object drop down, select Thread as
Object.
- Click Select counter from List (default), then select the
following counters:
Counter = % Processor Time
Counter = ID Process
Counter = ID Thread
- Click Select instances from list and select all Java threads as
shown in this example:
- Click Add to add new properties, then click Close.
- Logging starts immediately.The log file icon turns green.
- While the performance data is being logged, gather relevant
Performance Degradation documentation. You can get instructions for
gathering performance degradation by selecting the appropriate link for
your Application Server Version:
V5
V6
The above is a very important step. Support requires this information to
determine the cause of the CPU consumption.
- Once Performance Degradation documentation is collected, end the
Performance Monitor logging. From the Performance Monitor tool,
right-click Log Name in the counter window, then click Stop.
The log file to send to IBM is referenced in the Log File Name
column.
- Generate 3 dumps of the Java process with an interval of 2 to 3
minutes. userdump.exe is part of the userdump utility that must be
installed on your system. Look at the above section titled: Follow these
instructions to setup the failing System for capturing required
information.
- userdump.exe <PID of Java process> <complete
path and file name of the target dump>
- Example: c:\userdump\userdump.exe 7144
c:\dumpdata\firstDump.dmp
- Note: If the target path is not specified, then the output
will go into the directory from where the command was issued.
- Collect the following files
- The perfmon log
- The javacore*.txt files generated (Look at step
10)
- The three dump files generated from step 12.
- For V6.0 release:
- All files in
<install_root>\profiles\<ProfileName>\logs\
<ServerName>directory.
- A copy of server.xml located in
<install_root>\profiles\<ProfileName>
\config\cells\<CellName>\nodes\<NodeName>\servers\<ServerName>
directory.
- For V5.0 and V5.1 releases:
- Include all of the files from the
<install_root>\logs\<ServerName> directory.
- A copy of server.xml located at
<install_root>\config\cells\<CellName>
\nodes\<nodeName>\servers\<ServerName>
directory
- Follow instructions to send
diagnostic information to IBM support.
Analyzing the data
To review the log, open the performance log file using perfmon.
"Perfmon" is a tool provided by Microsoft and shipped as part of the
operating system. It is usually located in the system32 folder.
On the main screen titled "Performance", click the "View Log File
Data" icon (cylinder shaped). This will open System Monitor View,
where you should click on the "Log Files" radio button. Then click on
"Add" to select the log file.
Now back on the main "perfmon" screen titled "Performance", click the
plus ( + ) button. In the "Add Counters" window, select the
following items:
Performance Object = Thread
Click on Radio button: "Select counters from list" and select the three
listed counters:
Counter = % Processor Time
Counter = ID Process
Counter = ID Thread
And select all instances.
Then click on "Add" and close the "Add Counters" window.
On the main screen titled "Performance", click the "View
Reports" icon to determine the thread ID that consumes the CPU

Convert the thread ID (3480) to hexadecimal using the Windows
calculator
3480 in decimal = D98 in hex
In the javacore.txt file, locate the thread that has a native ID of
0xD98
For example:
"SoapConnectorThreadPool : 0" (TID:0x10A25A98, sys_thread_t:0x24A31B30,
state:R, native ID:0xD98) prio=5
Review the stack of that thread in each javacore to determine the cause
of the CPU usage.
For a listing of all technotes, downloads, and educational materials
specific to the 100% CPU Usage issue, search the WebSphere
Application Server support site.
|