This section introduces the concept of thread dumps in WebSphere Application Server.
A thread represents a work item or task, such as a servlet request. Java processes are
usually multi-threaded. This means there can be many tasks occurring simultaneously(i.e. multi-threading)
within one JVM (Java Virtual Machine) process. Therefore, understanding what is occurring within a JVM
process is means obtaining information about all the different threads that are defined within the process.
There are two types of thread dumps that could appear when running Java programs:
System thread dumps provide a system view of a failing JVM (Java Virtual Machine) process. On Unix
systems, they usually appear as core files. On Window's systems they appear as drwtsn32.log files.
System dumps do not understand Java classes. Everything in a system dump is C library oriented. The
system dump information provided for JVM processes refers to Java's C libraries and not the reference
class files.
System dumps should only be interrogated when a Java thread dump is unavailable. Pertinent information
can be obtained from system dumps. However, mapping this information back into Java source code is very
difficult. The following sections explain how to interrogate the core and drwtsn32.log files. When they
are generated by the system, they need to be interrogated.
The following is an example of using the df command to verify that there is
enough room in the file system for the core.
The core file is placed in the ./bin directory. On AIX this is in the /usr filesystem.
A core file can be 200MB.
The ObjectFile parameter is an object (executable) file produced by a compiler. Use the -g (generate symbol table) flag when compiling your program to produce the information the dbx command needs.
If the -c flag is not specified, the dbx command checks for a .dbxinit file in
the user's $HOME directory. It then checks for a .dbxinit file in the user's
current directory. If a .dbxinit file exists in the current directory, that file overrides the .dbxinit
file in the user's $HOME directory. If a .dbxinit file exists in the user's $HOME directory or current
directory, that file's subcommands run at the beginning of the debug session. Use an editor to create
a .dbxinit file.
If ObjectFile is not specified, then dbx asks for the name of the object file to be examined. The
default is a.out. If the core file exists in the current directory or a core file parameter is specified,
then dbx reports the location where the program failed. Variables, registers and memory held in the
core image may be examined until execution of ObjectFile begins. At that point the dbx debug program
prompts for commands.
The common procedure of interrogating a core file is to change the directory to where the core file
resides. You can then issue the command with the binary executable file as the parameter. It is
important that the binary executable is used. Usually the java command is a shell script that calls
the executable. If you enter the shell script, java, as the parameter a "cannot find" error
message is returned.
The following commands show you how to find the binary executable and invoke the
dbx command. It also shows an illegal instruction was executed (i.e. Invalid opcode):
---------------------------------------------------------------------------------------
Type 'help' for help.
Warning: The core file is truncated. You may need to increase the ulimit
for file and core dump, or free some space on the file system.
Reading symbolic information ...Warning: no source compiled with -g [using memory image in core]
Illegal instruction (reserved addressing fault) in . at 0x0 ($t29)
0x00000000 00000001 Invalid opcode.
-----------------------------------------------------------------------------------------
If you don't know where the java binary is located, the following command will display the true java
executable name of the core:
After you enter dbx, the where command provides a stack trace of where the error occurred. The
following example shows a:
The next step is to identify the port number for either the administratiave server or a managed server.
The port number is different for the administratiave server and each of the managed server(s).
The port number values are contained in the standard out files for each of these processes. Information
on how to find these files and the port number are described below.
After starting the administratiave server, you should obtain the DrAdmin port number within the
.\logs\tracefile file inside the message:
After starting the managed server (for example, default server), you should obtain the DrAdmin port number
provided in the standard output file for the managed server. The location of the file can be found with
the administrative console interface in the managed server configuration via the General tab to standard output field.
The message within the file containing the port number is:
The DrAdmin.bat file can now be executed providing the port number obtained above. The format of the
command to use is:
where xxxx is the port number from the above message (without the comma).
The administratiave server thread information is placed in .\logs\adminserver_stderr.log file.
Because this file is not closed, it's length of 0 will not change. The application server thread information
is placed in the standard error file which can be found with the administrative console interface in the
application server configuration via the General tab to standard error field. The thread information is
dumped immediately into this file.
In order to view the thread information, copy the above files into a new file. Edit the new file with an
HTML editor, which will display the thread information. Some editors (i.e. emacs and vi) will allow you to
view the thread information directly from the .\logs\adminserver_stderr.log file or the standard error file.
A thread dump can be forced or can occur when a Java process error occurs. When a thread dump
is not forced, it usually means that an error within a Java process has occurred and it should to be
investigated. A thread dump of a Java process needs to be forced when the process has a thread deadlock
condition. A thread deadlock condition is defined as:
Thread A currently owns Lock X.
Thread B owns Lock Y.
Thread A is waiting for the release of Lock Y in order to continue processing.
Thread B is waiting for the release of Lock X in order to continue processing.
Because of this stalemate condition, neither thread is able to complete its processing.
The referenced Java thread dump information is taken from a sample AIX dump. Java thread dumps on other
platforms have similar information, but they may be formatted different.
In order to have a thread safe application, the application may have to ensure that two threads don't
execute the same code simultaneously. This can be accomplished with the use of a synchronized()statement
or a synchronized modifier of a class method.
Each of the above will create a monitor/lock that will prevent other threads from executing the same code.
It is important to understand that threads can be holding multiple monitors/locks while processing a request.
Therefore two threads could find themselves in a deadlock condition defined by the following situation:
Thread A owns Lock X.
Thread B owns Lock Y.
Thread A is waiting for the release of Lock Y.
Thread B is waiting for the release of Lock X.
Because of this stalemate condition, neither thread is able to complete its processing.
The above diagnosis requires an understanding of the involved source code (i.e. which methods are
synchronized). However, the Java thread dump does provide the pointers to do this additional investigation.
A summary of the object monitors will provide additional information that identifies Thread A as blocked
Thread B with:
com.ibm.CORBA.iiop.IIOPConnection@4fe89740: owner: "Thread B"
"Thread A" (0x36951ba8) blocked
Unfortunately, this information does not appear in the summary for the monitor being held by Thread A.
Stack traces represent the current call path of a thread. Call path information explains what
functional calls were made to get to the thread's current location.
During startup of the different WebSphere Application Server processes, the processes are initialized and
placed in a state to accept additional network activity. One of the first steps in initializing a
process is to create an ORB instance. This step will create threads that will be used to complete the
initialization step and later accept network activity to be processed.
These activities are described within each of the two diagrams of the next two sections. The diagrams
describe:
- Administratiave server startup and immediate administrative server takedown
- Managed server startup with servlet traffic
The administratiave server has two ORBs defined within it. For each ORB there is at least one ORB
server listener thread that continually waits for input on a port. When input is received, it is dispatched
to a ORB server reader thread so the ORB server listener thread can again wait for input on the port. The
ORB server reader thread again dispatches the request to a third thread that completes the work activity.
The reply to the work activity is sent from the third thread to a ORB client reader thread that receives
replies from the ORB reader thread. An ORB request has four steps/threads involved:
- ORB Server Listener thread receives input on port X.
- ORB Server Reader thread is given a request.
- Pooled/instantiated thread handles the request and sends a reply.
- ORB Client Reader thread handles the reply.
The managed server has one ORB defined within it. There are two ORB server listener threads and
multiple ORB client threads. The servlet traffic does not use the ORB for communications. It is done with
the plug-in interface. This interface supports a pool of worker threads (Worker#_) that complete the
HTTP requests.
The following port numbers are preset:
- 9000 is used for obtaining naming services (i.e. data source names, EJB names)
- 900 is used by the administratiave server to listen for administrative client requests
Other port numbers are randomly chosen for ORB communications.
Thread names
ORB threads
The ORB instance creates reader and listener threads. The names of these threads get changed after
they begin processing (i.e. during run() method processing). The name is constructed with the following
parameters separated by a colon (:):
- ORB information
- P = unique for this process and algorithmically constructed from a time stamp
- O = number of ORB's within this process
- Thread type
- StandardRT = identifies which reader thread is within the ORB
- CT = client thread
- LT = listener thread
- Connection values
- LocalPort = Local port that thread is dealing with
- RemoteHost = Hostname for ORB server reader thread, or IP address for ORB client reader thread
- RemotePort = Port number on the remote host for the connection
Worker#__ (SERVLET ENGINE THREADS)
These thread names begin with Worker# and process HTTP requests.
Thread-x
These thread names are the default thread name for Windows 2000 and AIX. Because no thread name is
provided this name is used. X is incremented as each new thread is created.
Pooled ORB request dispatch WorkerThread
These threads are created by the main thread (i.e. P=479481:O=0:CT)and handle the request/replies
that are sent across IIOP connections.
Web server plug-in configuration thread
Thread used for setting the Web server configuration.
Alarm manager
This thread manages the creation of alarm thread x's.
Alarm thread 1
The alarm thread 1 reclaims unused connections.
BackgroundLruEvictionStrategy
This thread sweeps a cache, reclaiming the least recently used objects.
Refresh
This thread insures that any changes to a model get propagated to clones.
Thread stack traces
When a thread is created, the start() method is used to invoke the run() method. The start() method is
executed on one thread and the run() method is executed on the newly created thread. Depending
on when the stack trace is obtained, an activity (i.e. piece of work) could have different stack traces.
Therefore, thread names have two base method calls. The following text describes these base method calls
for the common thread names used for both the administratiave server and the managed server. Two stack
trace examples of base method calls are also provided:
Base method calls
- Main or P=xx:O=0:CT
- run ---> com.ibm.ejs.sm.server.AdminServer.main()
- ORB server listener thread (JavaIDL Listener or P=xx:O=0:LT=0:port=9000)
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.CORBA.iiop.ListenerThread.run()
- ORB server reader thread (JavaIDL Reader for hostname:port# or P=xx:O=0:StandardRT=0:LocalPort=port#:RemoteHost=hostname:RemotePort=port#:)
- start ---> com.ibm.CORBA.iiop.ListenerThread.run()
- run ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
- ORB client reader thread (JavaIDL Reader for ipaddr:port# or P=xx:O=1:StandardRT=1:LocalPort=port#:RemoteHost=ipaddr:RemotePort=port#:)
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
- Pooled ORB request dispatch WorkerThread
- start ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
- run ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
- Worker#__
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run() java.lang.Thread.run()
- Web server plug-in configuration thread
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run() java.lang.Thread.run()
- Alarm manager
- start ---> com.ibm.ejs.sm.server.AdminServer.main() <--AdminServer
- com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run() <--AppServer
- run ---> com.ibm.ejs.util.am.AlarmManagerThread.run() java.lang.Thread.run()
- Alarm thread 1
- start ---> com.ibm.ejs.util.am.AlarmManagerThread.run() java.lang.Thread.run() <--AdminServer
- com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
<--AppServer
- run ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run() <--AdminServer
- com.ibm.ejs.util.am.AlarmThread.run() <--AppServer
- BackgroundLruEvictionStrategy
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.ejs.util.cache.BackgroundLruEvictionStrategy.run()
- RefreshThread
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.ejs.wlm.server.config.ServerGroupRefresh$RefreshThread.run()
Examples
Thread dump of a standard reader thread:
"P=863240:O=1:StandardRT=16:LocalPort=10502:RemoteHost=gofast:RemotePort=2619:"
(TID:0x11ccef0, sys_thread_t:0xcdd81d0, state:R, native ID:0x128) prio=5
>
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
at com.ibm.rmi.iiop.Message.readFully(Message.java(Compiled Code))
at com.ibm.rmi.iiop.Message.createFromStream(Message.java:173)
at com.ibm.CORBA.iiop.IIOPConnection.createInputStream(Unknown Source)
at com.ibm.CORBA.iiop.StandardReaderThread.run(Unknown Source)
The base method, com.ibm.CORBA.iiop.StandardReaderThread.run(), is identified as the run base method for
JavaIDL Reader for hostname:port# threads. Also, the thread is waiting for input because it is
in the java.net.SocketInputStream.socketRead() method.
Thread dump of a worker thread:
"Worker#49" (TID:0x10793660, sys_thread_t:0xab25b0, state:R, native ID:0x19a) prio=5
at com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.nativeGetSeviceMessageId()
at com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.getSeviceMessageId()
at com.ibm.servlet.engine.oselistener.serverqueue.SQWrapperEventSource$SelectRunnable.getNewConnectionFromQueue()
at com.ibm.servlet.engine.oselistener.serverqueue.SQWrapperEventSource$SelectRunnable.run()
at com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run()
at java.lang.Thread.run()
The base method, java.lang.Thread.run(), is identified as the run base method for Worker#__ threads.
Also, the thread is waiting for input from the Web server plug-in (native code) because it is in the
com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.nativeGetSeviceMessageId() method.
The following diagram has highlighted request flows that start with a SendReqXXX where XXX is the port
number of the send request. The steps in the flow changes between different threads. The sequence of the
steps are identified with, for example, 1A,1B, 1C and 1D. It also shows how the port that the request is
sent to determines which thread the proccessing has completed.
Diagram Legend
In each diagram, every continuous line (-------) is a thread. The name of the thread always appears
between (...). The letters in the diagram have the following meanings:
C = Thread name changed to (.....)
S = Start method called on this thread
R = Run method called on this thread
W = Thread is in wait state waiting for notify
WM = Thread is waiting for message from plugin (Worker# threads only)
SendReq____ = Request sent to port number (____)
SendReply___ = Reply sent to port number (____)
For example:
C(P=479481:O=0:CT) = thread name is changed to P=479481:O=0:CT
R = thread is placed in a running state
Diagram
ORB 0 Threads(i.e. O=0)
main
|
|
C(P=479481:O=0:CT)
|
|S(JavaIDL Listener) R C(P=479481:O=0:LT=0:port=9000)
|-----------------------------------------------------------------------------------------> | |
| S(JavaIDL Reader for rbostick:1294)
| |
| |
| R
| |
| |
| C(P=479481:O=0:StandardRT=0:LocalPort=9000:
| RemoteHost=rbostick:RemotePort=1294:)
| |
| 1B
| |S(Thread-1) R
SendReq9000(1A) |-1C--------------------->
| |
| 2B
| |S(Thread-2) R
SednReq9000(2A) |-2C--------------------->
| |
| 5B
| |
| |
| |
| V
|
|S(JavaIDL Reader for 9.27.63.245:9000) R C(P=479481:O=1:StandardRT=1:LocalPort=1294:
| RemoteHost=9.27.63.245:RemotePort=9000:)
|------1D--2D--5D-------------------------------------------------------------------------> |
|
|
|
|
|
|
|
|
|
|
|
V
ORB 1 Threads (i.e. O=1)
|
|
|
|S(JavaIDL Listener) R C(P=479481:O=1:LT=1:port=1295)
|--------------------------------------------------------------------------------------------> |
|
|
|
|
|S(JavaIDL Listener) R C(P=479481:O=1:LT=2:port=1296)
|-------------------------------------------------------------------------------------------->
| |
| S(JavaIDL Reader for rbostick:1299)
| |
| R
| |
| C(P=479481:O=1:StandardRT=5:LocalPort=1296:
| RemoteHost=rbostick:RemotePort=1299:)
| |
| 6B
| |
| V
|
|
|
|S(JavaIDL Reader for 9.27.63.245:1299) R C(P=479481:O=1:StandardRT=4:LocalPort=1299:
| RemoteHost=9.27.63.245:RemotePort=1296:)
|------6D--------------------------------------------------------------------------------->
|
|
|
|S(JavaIDL Listener) R C(P=479481:O=1:LT=3:port=900)
|--------------------------------------------------------------------------------------------> | |
| S(JavaIDL Reader for rbostick:1297)
| |
| R
| |
| C(P=479481:O=1:StandardRT=3:LocalPort=900
| :RemoteHost=rbostick:RemotePort=1297:)
| |
| 3B
| |
| 4B
| |
| V
|
|
|S(JavaIDL Reader for 9.27.63.245:900) R C(P=479481:O=1:StandardRT=2:LocalPort=1297:
| RemoteHost=9.27.63.245:RemotePort=900:)
|-----3D--4D--------------------------------------------------------------------------------->
|
|
|
SendReq900(3A)
|
SendReq900(4A)
|
SendReq9000(5A)
|
SendReq1296(6A)
|
SednReq1296(7A)
|
Other Threads
|
|
|
|
|
|S(Pooled ORB request dispatch WorkerThread) W R
|------3C--5C----------------------------------------------------------------------------->
|
|
|
|S(Pooled ORB request dispatch WorkerThread) W R |------4C--6C----------------------------------------------------------------------------->
|
|
|
|
|S(Alarm Manager) R
|----------------------------------------------------------------------------------------->
| S(Alarm Thread 1)
| |
| R
| |
| V
|
|S(Thread-3) R
|---------------------------------------------------------------------------------------->
|
|
|S(Thread-4) R
|---------------------------------------------------------------------------------------->
| | | | | |
| S(Thread-8) S(Thread-9) S(Thread-10) S(Thread-11) S(Thread-12) | | | | | |
| R R R R R
| | | | | |
| V V V V V
|
|
|S(Worker#0) R S(Worker#0) R
|-------------------------------------------------------------------------------------------->
|
|
|S(WebServer-Plugin-Cfg-Thread) R
|-------------------------------------------------------------------------------------------->
|
|S(BackgroundLruEvictionStrategy) R
|-------------------------------------------------------------------------------------------->
|
|S(RefreshThread) R
|--------------------------------------------------------------------------------------->
V
Thread-1(2) (worker threads)
- start ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
run ---> com.ibm.CORBA.iiop.WorkerThread.run()
Thread-3 (transaction timeout)
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.ejs.jts.tran.JavaClock.run()
Thread-4 (used for AdminServer takedown)
- start ---> com.ibm.ejs.sm.server.AdminServer.main()
- run ---> com.ibm.ejs.sm.server.ManagedServer$DiagonisticThread.run()
Thread-8,9,10,11,12 (threads for takedown process)
>
- start ---> com.ibm.ejs.sm.server.ManagedServer$DiagonisticThread.run()
- run ---> com.ibm.ejs.sm.util.task.AsyncTaskEngine$WorkerThread.run()
Note: Thread-x are default names of threads. The above numbers may be different depending on
the system that the administratiave server runs on.
The Worker#_threads are the threads on which servlet requests are processed. The threads
start during the managed server startup and wait on input from the Web server plug-in interface.
main
|
|
C(P=905990:O=0:CT)
|
|
|S(Thread-0) R
|--------------------------------------------------------------------------------------->
|
|
|S(Pooled ORB request dispatch WorkerThread) W R
|-------------------------------------------------------------------------------------------->
| | | | | | | |
| S | | S(Worker#0)S(Worker#1).......S(Worker#24) S(Thread-6) (BackgroundLruEvictionStrategy) | | | |
| | | R WM WM WM R
| R | | | | | |
| | S(AlarmManager) | S(Worker#0) | Servlet |
| | | | | | Request |
| | | S(pluginRegenScheduler) | | | |
| | R | WM | WM |
| | | V V V V V
| | |
| | |
| V |S(AlarmThread1) R
| |---------------------------->
| V
|
|
|S(Pooled ORB request dispatch WorkerThread) W R W |------------------------------------------------------------------------------------>
|
|S(Thread-1) R
|------------------------------------------------------------------------------------>
|
|
|
|S(Thread-3) R
|------------------------------------------------------------------------------------>
|
|
|
|
ORB 0 Threads (i.e. O=0)
|
|S(JavaIDL Reader for 9.27.63.129:9000) R C(P=905990:O=0:StandardRT=0:LocalPort=1480:
| RemoteHost=9.27.63.129:RemotePort=9000:
|------------------------------------------------------------------------------------>
|
|
|
|S(JavaIDL Listener) R C(P=905990:O=0:LT=0:port=1481)
|------------------------------------------------------------------------------------->
|
|
|S(JavaIDL Reader for 9.27.63.129:1434) R C(P=905990:O=0:StandardRT=1:LocalPort=1482:
| RemoteHost=9.27.63.129:RemotePort=1434:)
|------------------------------------------------------------------------------------->
|
|
|
|
|S(JavaIDL Reader for 9.27.63.129:900) R C(P=905990:O=0:StandardRT=2:LocalPort=1483:
| RemoteHost=9.27.63.129:RemotePort=900:)
|------------------------------------------------------------------------------------->
|
|
|
|S(JavaIDL Reader for 9.27.63.129:1433) R C(P=905990:O=0:StandardRT=3:LocalPort=1484:
| RemoteHost=9.27.63.129:RemotePort=1433:)
|------------------------------------------------------------------------------------->
|
|
|
|
|S(JavaIDL Listener) R C(P=905990:O=0:LT=1:port=1485)
|------------------------------------------------------------------------------------------>
| |
| S(JavaIDL Reader for rbostick:1487)
| |
| R
| |
| C(P=905990:O=0:StandardRT=4:LocalPort=1485:
| RemoteHost=rbostick:RemotePort=1487:)
| |
| |
| V
V
Thread-0 (transaction timeout)
- start ---> com.ibm.ejs.sm.server.ManagedServer.main()
- run ---> com.ibm.ejs.jts.tran.JavaClock.run()
Thread-1 (Used for logging messages)
- start ---> com.ibm.ejs.sm.server.ManagedServer.main()
- run ---> com.ibm.ejs.sm.server.SeriousEventListener$DeliveryThread.run()
Thread-3
>
- start ---> com.ibm.ejs.sm.server.ManagedServer.main()
- run ---> com.ibm.ejs.sm.server.ManagedServer.main()
Thread-6 (AdminServer ping)
- start ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
- run ---> com.ibm.ejs.sm.server.ManagedServer$PingThread.run()
Note: Thread-x are default names of threads. The above numbers may be different. depending on the system the managed server runs on.
Summary
In multi-processing and multi-thread environments, problem determination can require analysis of
actively running threads. This thread information can be obtained with system thread dumps and Java thread
dumps. When doing problem determination in a WebSphere Application Server environment, Java thread dumps
provide much more information and are recommended. However, sometimes system thread dumps are the only
information obtained and should be interrogated.
When dealing with thread deadlock problems, Java thread dumps can be forced using kill -3 on Unix
platforms and DrAdmin on all platforms.
The output of these commands provides thread information necessary to diagnose the problem.