|
Problem(Abstract) |
Collecting data for when your IBM® WebSphere® Application
Server is not responding (hangs) on the Linux® operating system. Gathering
this information before calling IBM support will help familiarize you with
the troubleshooting process and save you time. |
|
|
|
Resolving the
problem |
If you have already contacted support, continue on to the
component-specific MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
Collecting MustGather information for an Application Server hang or
performance degradation
The following steps should be taken prior to reproducing the hang
or performance degradation issue:
- Enable
verbosegc on WebSphere Application Server. This is necessary for
debugging performance issues and does not have significant impact on
performance.
- Synchronize clocks on all Web server and WebSphere Application Server
systems. If clocks on the Web server and WebSphere Application Server
systems were not synchronized for some reason, provide information on
which clock was faster and what was the difference.
- If possible, clean all Web server, plug-in and WebSphere Application
Server and FFDC logs.
When you are able to reproduce the problem or when the problem recurs,
perform the following steps to collect the MustGather information needed
to debug the problem:
- Run the following command:
netstat -an > netstat.out |
|
Note: If the Web server is remote, gather
netstat output on the Application Server system and the Web
server systems. |
|
- Run the following command:
vmstat 5 12 >>
vmstat.log |
|
- Run the following command:
- Check to see if you can serve a static HTML document from the Web
server.
- Check to see if you can serve dynamic documents such as servlets or
JSPs when bypassing the Web server and WebSphere Application Server
plug-in.
- Find the PID of the failing Java™ virtual machine (JVM). You can find
the PID using multiple ways, here are couple of ways:
- A file named server_name.pid will
get updated with the PID of the application server.
- For all versions, the process ID will be printed in the
SystemOut.log file at each restart. You will have to pick the
latest one from the log file.
- Run the following command:
kill -3 [PID of hung
JVM] |
|
Note: The preceding and following kill -3
commands will create javacore.txt files in the
install_root, install_root/bin, or in the
configured working directory. |
|
- Wait two minutes.
- Run the following command:
kill -3 [PID of
hung JVM] |
|
- Wait two minutes.
- Run the following command:
kill -3 [PID of hung
JVM] |
|
- Wait two minutes.
- Collect CPU usage statistics by running the following command:
top -d 60 -n 2 -c -b >
top.log |
|
- In some cases, the kill -3 commands do not produce javacores.
In other cases, the javacores do not contain full Java stack traces. A
Linux operating system core will be needed to ascertain the Java thread
stack in this case. Perform the following steps to generate the core file:
- Attach the GDB debugger to the Java process of the hung JVM:
gdb
JAVA_HOME/jre/bin/java [PID of hung
JVM] |
|
- Run a gcore command from the gdb command line:
- Detach the gdb debugger from the process:
- The preceding steps should create a system core file without
terminating the JVM process. If for some reason the core is not generated,
run the following command to terminate the process and generate a system
core:
kill -ABRT [PID of
hung JVM] |
|
Note: The preceding gdb/kill -ABRT command
creates a core in the install_root/bin or
the operating_system_root/tmp directory. |
|
- The preceding step creates a core file. You will need to follow the
directions in MustGather: Crash on Linux and then send in the
requested diagnostic information.
- Gather the following files:
- For WebSphere Application Server V6.0 and
V6.1:
- The server.xml file located in the following
directory:
profile_root/config/cells/cell_name/nodes/
node_name/servers/server_name |
|
- The plugin-cfg.xml and http_plugin.log
files
- Everything in the following directory:
profile_root/logs/server_name |
|
- Everything in the following directory:
- All documents requested for all releases below.
- For WebSphere Application Server V5.0 and 5.1:
- The server.xml file located in the following
directory:
install_root/config/cells/nodes/node_name/
servers/server_name |
|
- The plugin-cfg.xml and http_plugin.log
files
- Everything in the following directory:
install_root/logs/server_name |
|
- Everything in the following directory:
- All documents requested for all releases below.
- For all WebSphere Application Server releases
- ps_eLf.txt
- All javacore.txt files created
- All netstat.out file
- All vmstat*.out files
- top.log
- Web server access and error logs
- All files created from processing the system core
(core.sdff, output from libsgrabber.sh, and so on)
- Follow instructions to send
diagnostic information to IBM support.
If you want to analyze the Java thread dumps yourself, download the IBM Thread and Monitor Dump
Analyzer for Java Technology. ThreadAnalyzer is a technology
preview that can analyze thread dumps from WebSphere Application Server.
It is useful for identifying deadlocks, contention, and bottlenecks, as
well as summarizing the state of threads within WebSphere Application
Server.
For a listing of all technotes, downloads, and educational materials
specific to a hang or performance degradation, search the WebSphere Application Server support site. |
|
|
|
|
|
Cross Reference information |
Segment |
Product |
Component |
Platform |
Version |
Edition |
Application Servers |
WebSphere Application Server - Express |
|
Linux, Linux Red Hat - pSeries |
|
Edition Independent |
Application Servers |
WebSphere Application Server Enterprise |
|
Linux, Linux Red Hat - pSeries |
|
Edition Independent |
Application Servers |
Runtimes for Java Technology |
Java SDK |
|
|
|
|
|
|