|
Problem |
Collecting data for when your IBM® WebSphere® Application
Server is not responding (hangs) for V5.0, V5.1, and V6.0 releases on the
Windows® operating system. Gathering this information before calling IBM
support will help familiarize you with the troubleshooting process and
save you time. |
|
Solution |
If you already contacted support, continue to the Hangs or Performance
Degradation MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
No response (hang) or performance degradation specific MustGather
information
This technote describes what information you need to begin
troubleshooting an Application server that is not responding or is
hung. The method used to create a thread dump differs from earlier
releases of the product. Use the following directions to gather the
necessary documentation to debug the problem:
Follow these instructions for initial environment set up before
you re-create the hang problem:
- If possible, following instructions on Enabling
verbosegc on WebSphere Application Server before recreating the
problem.
- Synchronize clocks on all Web server and WebSphere Application Server
systems. If clocks on the Web server and WebSphere Application Server
systems were not synchronized for some reason, provide information which
clock was faster and what was the difference.
- If possible, clean all Web server, plug-in, WebSphere Application
Server and FFDC logs.
- Complete steps 1 - 9 in MustGather:
Getting user.dmp when hangs/performance degradation prevents generating a
javacore to configure drwtsn32 tool to produce a
user.dump file from the Java process. Also follow the
instructions in the same technote to install and configure the Microsoft
userdump tool. Installing this tool might require you to restart
the server.
When you experience the hang, enter the following commands from a
command-line prompt on the WebSphere Application Server machine and any
remote Web server systems:
- Take the initial output of netstat command to get information
about TCP/IP sockets:
netstat -an >
netstat_before.out |
|
- Next, from the command prompt, enter the command wsadmin.bat to
get a wsadmin command prompt.
Note: If security is enabled or the default SOAP ports have been
changed, you will need to pass additional parameters to the batch file in
order to get a wsadmin prompt. For example:
wsadmin.bat [-host
host_name] [-port
port_number] [-user
userid[-password
password] |
|
Note: You can connect wsadmin to any of the server
JVM in the cell. After running the wsadmin command it will display
the server process for which it has attached to. Depending on the process
that it has attached to, you can get thread dumps for various JVMs. If
wsadmin is connected to deployment manager, then you can get thread dumps
for any JVM in that cell. If it is attached to a node agent, then you can
get thread dumps for any JVM in that Node. If it is attached to a server,
then you can get thread dumps only for the server to which has connected
to. |
|
- Get a handle to the problem application server.
Note: The contents in brackets "[.....]", along with the brackets,
is not optional. It must be entered to set the jvm object. Also,
note that there is a space between the words "completeObjectName" and
"type":
wsadmin> set jvm [$AdminControl
completeObjectName
type=JVM,process=server1,*] |
|
Where server1 is the name of the
application server that does not respond (or is hung). If wsadmin
is connected to a Deployment Manager and if the server names in the cell
are not unique, then you can qualify the JVM with node attribute in
addition to process. |
|
- Generate the thread dump:
wsadmin>$AdminControl invoke $jvm
dumpThreads |
|
- Wait 2 minutes.
- Generate the thread dump:
wsadmin>$AdminControl invoke $jvm
dumpThreads |
|
- Wait 2 minutes.
- Generate the thread dump:
wsadmin>$AdminControl invoke $jvm
dumpThreads |
|
- The preceding steps should have created 3 javacores which
should be created in the installation root directory (for example,
WebSphere\AppServer). In some cases, javacores cannot be
created when the JVM is hung or the Java thread stacks in the
javacores are not complete. To ensure that you have the data
necessary to debug the hang, follow instructions in MustGather:
Getting user.dmp when hangs/performance degradation prevents generating a
javacore to produce one user.dmp file from the hung Java
process.
Note: If you are not getting javacores from the preceding
wsadmin commands, collect a series of three user.dmp files
at 2 minute intervals.
- Take the final output of netstat command to get information
about TCP/IP sockets:
netstat -an >
netstat_after.out |
|
- Collect the following diagnostic information:
- For WebSphere Application Server V6.0:
- The server.xml file located in the following
directory:
profile_root\config\cells\cell_name\nodes\
node_name\servers\server_name |
|
- plugin-cfg.xml and http_plugin.log
- Everything in the following directory:
profile_root\logs\server_name |
|
- Everything in the following directory:
- For WebSphere Application Server V5.0 and V5.1:
- The server.xml file located in the following
directory:
install_root\config\cells\nodes\node_name\
servers\server_name |
|
- plugin-cfg.xml and http_plugin.log
- Everything in the following directory:
install_root\logs\server_name |
|
- Everything in the following directory:
- For all releases:
- All javacore.date.time.id.txt files generated.
These files should be in the installation root directory (for example:
WebSphere\AppServer).
- netstat*.out
- Include the following Application Server log files, if
they are located in a different directory:
- systemErr.log
- systemOut.log
- native_stderr.log
- native_stdout.log
- If the Web server is remote, send the appropriate file
from the Web server system including Web server configuration files and
Web server logs. For example for IBM HTTP Server and Apache Web server
provide httpd.conf, access and error logs. For
SunOne 6 Web server provide magnus.conf, obj.conf,
access and error logs.
- Follow instructions to send
diagnostic information to IBM Support.
If you want to analyze the Java thread dumps yourself, download the
. ThreadAnalyzer is a technology preview tool that can analyze thread
dumps from WebSphere Application Server. It is useful for identifying
deadlocks, contention, bottlenecks, and to summarize the state of threads
within WebSphere Application Server.
For a listing of all technotes, downloads, and educational materials
specific to a hang or performance degradation, search the WebSphere
Application Server support site. |
|
|
|
|
Cross Reference information |
Segment |
Product |
Component |
Platform |
Version |
Edition |
Application Servers |
WebSphere Application Server - Express |
Hangs/performance degradation |
Windows |
6.0, 5.1, 5.0 |
|
Application Servers |
Runtimes for Java Technology |
Java SDK |
|
|
|
|
|
|
|