This list documents what is needed to troubleshoot a
stalled Java™ virtual machine (JVM™) on Linux® platforms.
If you have already contacted support, continue on to the
Hangs/Performance Degradation MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
Hangs/performance degradation specific MustGather
information
- Follow these instructions for initial environment set up before you
re-create the hang problem:
- Enable verbosegc on WebSphere Application Server before
recreating the problem. This is necessary for debugging performance issues
and does not have significant impact on performance.
- Synchronize clocks on all Web server and WebSphere Application Server
systems. If clocks on the Web server and WebSphere Application Server
systems were not synchronized for some reason, provide information which
clock was faster and what was the difference.
- If possible, clean all Web server, plug-in and WebSphere Application
Server and FFDC logs.
- At the next occurrence of the hang, do the following on the
Application Server system:
- netstat -an > netstat.out
If the Web server is remote, gather netstat output on the
Application Server system and the Web server systems.
- vmstat 5 12 >> vmstat.log
- ps -efH > ps_efh.txt
- Check to see if you can serve a static .html document from the Web
server.
- Check to see if you can serve dynamic documents such as servlets or
JSPs when bypassing the Web server and WebSphere Application Server
plug-in.
- Find the PID of the failing JVM. You can find the PID using multiple
ways, here are couple of ways:
- Starting V5, a file named <servername>.pid will get updated with
the PID of the application server.
- For all versions, the process id will be printed on the SystemOut.log
at each restart. You wil have to pick the latest one from log.
- kill -3 [PID of hung JVM]
- Wait two minutes
- kill -3 [PID of hungJVM ]
- Wait two minutes
- kill -3 [PID of hungJVM]
- Wait two minutes
- netstat -an > netstat2.out
- In some cases, the kill -3 commands do not produce
javacores. In other cases, the javacores do not contain full Java stack
traces. Because of this, you might need to issue the kill -11 command
against the PID of the hung JVM. If javacore files are generated
successfully, steps m through o are optional, since kill -11 will
stop the process. If javacore files were not generated for some reason or
if they are truncated, perform the following steps
kill -11 [PID of hungJVM ]
- netstat -an > netstat3.out
- The preceding step creates a core file. You will need to follow the
directions in the following technote and send in the requested
documentation: MustGather: Crash on Linux
The kill -3 command creates javacore.txt files in the
install_root, install_root/bin or
in the configured working directory. The kill
-11 creates a core in
the install_root/bin or the
operating_system_root/tmp directory
- Gather the following files:
- For WebSphere Application Server V6.0 release:
- The server.xml file located in the
install_root/profiles/profile_name/config/cells/cell_name
/nodes/node_name/servers/server_name
- plugin-cfg.xml and http_plugin.log
- Everything in the
install_root/profiles/profile_name/logs/server_name
directory
- Everything in the
install_root/profiles/profile_name/logs/ffdc
directory
- All documents requested for all releases below
- For WebSphere Application Server V5.0 and V5.1 releases:
- The server.xml file located in the
install_root/config/cells/nodes/node_name/servers/server_name
directory
- plugin-cfg.xml and http_plugin.log
- Everything in the
install_root/logs/server_name
directory
- Everything in the
install_root/logs/ffdc
directory
- All documents requested for all releases below
- For WebSphere Application Server V4.0 release:
- A XMLConfig full export.
- plugin-cfg.xml and native.log
- Everything in the
install_root/logs directory from the WebSphere
Application Server system.
- All documents requested for all releases below
- For all releases
- ps_efh.txt
- All javacore.txt files created
- All netstat*.out files
- All vmstat*.out files
- Web server access and error logs
- core.sdff
- If the Web server is remote, send the appropriate file
from the Web server system including Web server configuration files and
Web server logs. For example for IBM HTTP Server and Apache Web server
provide httpd.conf, access and error logs, for
SunOne 6 Web server provide magnus.conf, obj.conf, access
and error logs.
- Follow instructions to send
diagnostic information to IBM support.
If you want to analyze the Java thread dumps yourself, download the ThreadAnalyzer tool. ThreadAnalyzer is a
technology preview that can analyze thread dumps from WebSphere
Application Server. It is useful for identifying deadlocks, contention,
and bottlenecks, as well as summarizing the state of threads within
WebSphere Application Server.
For a listing of all technotes, downloads, and educational materials
specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.
|