To troubleshoot a hung Java™ virtual machine (JVM™),
collect the following information in the time during which the hang
occurs.
If you already contacted support, continue to the component-specific
MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
No response (hang) or performance degradation specific MustGather
information
Follow these instructions for initial environment set up before you
re-create the hang problem:
- If possible, follow instructions on How to enable verbosegc on WebSphere Application
Server before recreating the problem.
- Synchronize clocks on all Web server and WebSphere Application Server
systems. If clocks on the Web server and WebSphere Application Systems
were not synchronized, state which clock was faster and the difference
between the two times.
- If possible, clean all Web server, plug-in, WebSphere Application
Servers and FFDC logs.
- When you first start the application server JVM, run the dbx script
that is attached at the bottom of the page. Run the following script for
each clone using clone_name to distinguish between clones:
./dbxtrace_aix.sh -a [PID_JVM] >
dbx-startup-[clone_name].out
When you experience the hang, enter the following commands from a
command-line prompt:
- netstat -an > netstat1.out
If the Web server is remote, gather netstat output on the Application
Server system and the Web server system.
- vmstat 5 12 > vmstat1.out
- Run the appropriate svmon command based on the version of AIX:
- For AIX 4.3.3:
svmon -P -m pid -i 60 5 > svmon.out &
- For AIX 5.1:
svmon -P -m -r pid -i 60 5 >
svmon.out &
- For AIX 5.2 and AIX 5.3:
svmon -P pid -m -r -i 60 5 >
svmon.out &
- Can you serve a static document from the Web server?
- Can you serve dynamic documents such as servlets or JSPs when
bypassing the Web server and WebSphere Application Server plug-in?
- tprof -k -s -e -x sleep 60
- kill -3 [PID_of_hung_JVM]
- Wait two minutes.
- kill -3 [PID_of_hung_JVM]
- Wait two minutes.
- kill -3 [PID_of_hung_JVM]
- Wait two minutes.
- ./dbxtrace_aix.sh -a [PID_of_hung_JVM]
> dbx-hung.out
- netstat -an > netstat2.out
If the Web server is remote, gather netstat output on the Application
Server system and the Web server systems.
- vmstat 5 12 > vmstat2.out
- In some cases, the kill -3 commands do not produce
javacores. In other cases, the javacores do not contain full Java stack
traces. Therefore, you might need to issue the following commands against
the PID of the hung JVM:
netstat -an > netstat2.out
kill -11 [PID_of_hung_JVM]
- The preceding step creates a core file. Follow the directions in the
following technote and send in the requested documentation: MustGather: Crash on AIX
The kill -3 command creates javacore.txt files in the
install_root, install_root/bin or in the configured working directory. The
kill -11 creates a core in the
install_root/bin directory or the
operating_system_root/tmp directory.
- Collect the following information:
- For WebSphere Application Server 6.0 releases:
- The server.xml file located in the
install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
- plugin-cfg.xml and http_plugin.log
- Everything in the
install_root/profiles/profile_name/logs/server_name
directory
- Everything in the
install_root/profiles/profile_name/logs/ffdc
directory
- All documents requested for all releases below
- For WebSphere Application Server 5.0 and 5.1 releases:
- The server.xml file located in the
install_root/config/cells/cell_name/nodes/node_name/servers/server_name
directory
- plugin-cfg.xml and http_plugin.log
- Everything in the
install_root/logs/server_name
directory
- Everything in the
install_root/logs/ffdc
directory
- All documents requested for all releases below
- For WebSphere Application Server 4.0 release:
- A XMLConfig full export.
- plugin-cfg.xml and native.log
- Everything in the
install_root/logs directory
- All documents requested for all releases below
- For all releases:
- dbx*.out
- All javacore.txt files created
- All netstat*.out files
- All vmstat*.out files
- All smvon*.out files
- Send all files generated by tprof,
particularly _prof.all. These files are in the directory from which
the tprof command was run and they begin with an underscore "_". Some
versions of AIX might produce only a sleep.prof file.
- All files generated from following MustGather: Crash on AIX (above)
- If the Web server is remote, send the appropriate file
from the Web server system including Web server configuration files and
Web server logs. For example for IBM HTTP Server and Apache Web server
provide httpd.conf, access and error logs, for Sun ONE™
6 Web server provide magnus.conf, obj.conf, access
and error logs.
- Follow instructions to send
diagnostic information to IBM support.
If you want to analyze the Java thread dumps yourself, download the ThreadAnalyzer tool.
ThreadAnalyzer is a technology preview that can analyze thread
dumps from WebSphere Application Server. It is useful for identifying
deadlocks, contention, and bottlenecks, as well as for summarizing the
state of threads within WebSphere Application Server.
For a listing of all technotes, downloads, and educational materials
specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.
|