This list documents what is needed to troubleshoot a Java™
Virtual Machine (JVM™) that reaches 100% CPU utilization, or an unusually
high percentage on an AIX® system.
If you have already contacted support, continue on to the 100% CPU Usage
MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
Follow the below instructions to setup the failing System for
capturing required information:
- After starting the Application Server JVM execute the following dbx
script for each AppServer/clone having the problem:
./dbxtrace_aix.sh -a
PID_appserver >
dbx_startup_clone_name.out
- Follow instructions to Enable
verbosegc for the failing application server.
- Edit vmstat_script.sh and ps_script.sh to modify the sleep interval.
The default value of sleep interval is 5 minutes, which is good if problem
can be recreated in few hours. If it takes a week for the problem to
occur, then increase the interval and how often the script will overwrite
the output file.
- Execute the 2 scripts:
./vmstat_script.sh vmstat.out
./ps_script.sh ps.out
- Clear all application server log files before starting test. You may
have to stop the application server to delete the files and then start
them.
Collect the following information when the application server
process is experiencing the problem:
- netstat -an > netstat1.out
- If the Web server is remote then do the following on Web server
system:
netstat -an > netstatwebserver1.out
- Execute script:
./tprof_ps.sh tprof_ps
- kill -3 [PID_of_problem_JVM]
The kill -3 commands should create javacore.txt files in the
working directory of the application server. By default, for WebSphere
Application Server 3.5.x and 4.0.x, it is
install_root/bin directory. For WebSphere
Application Server 5.x and 6.x, it is
install_root/ directory.
- Wait two minutes
- kill -3 [PID_of_problem_JVM]
- Wait two minutes
- kill -3 [PID_of_problem_JVM]
- Wait two minutes
- netstat -an > netstat2.out
- If the Web server is remote then do the following on Web server
system:
netstat -an > netstatwebserver2.out
- ./dbxtrace_aix.sh -a
[PID_of_hung_application_server] >
dbx-hung.out
- Collect the following files
- For all releases of V6.0:
- All files in
install_root/profiles/profile_name/logs/server_name
directory.
- A copy of server.xml located in
install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
director
- For all releases of V5.x:
- Include all of the files from the
install_root/logs/server_name directory.
- A copy of server.xml located at
install_root/config/cells/cell_name/nodes/node_name/servers/server_name
directory
- For all releases of V3.0.2, V3.5, and V4.0:
- Include all of the files from the
install_root/logs directory.
- A copy of XMLExport for the server configuration
- If the Web server is remote, send the
trace.log.Web_server_name.date
(V3.5), native.log (V4.0), or http_plugin.log (V5.0 and
V6.0) file from the Web server system.
- dbxtrace.out
- All javacore.txt files created
- Output from vmstat_script.sh and
ps_script.sh.
- All files generated by tprof_ps.sh script.
These files will be in the directory specified as a script parameter.
- All netstat*.out files
- Follow instructions to send
diagnostic information to IBM support.
For a listing of all technotes, downloads, and educational materials
specific to the 100% CPU Usage component, search the WebSphere
Application Server support site. |