If an application server process spontaneously closes,
or web modules stop responding to new requests, it is important that
you quickly determine why this stoppage is occurring. You can use
some of the following techniques to determine whether the problem
is a web module problem or an application server environment problem.
If an application server process spontaneously closes,
or web modules
running on the application server stop responding to new requests:
- Try to Isolate the problem by installing the web
modules on different
servers, if possible.
- Check the product directory
structure for a file
with a name like javacore[number].txt. This file
is a Java thread dump file that the JVM creates if
an application server process spontaneously closes.
- Use the Tivoli® performance
viewer to determine if
any of the application server resources, such as the Java heap, or
database connections, have reached their maximum capacity. If there
is a resource problem, review the application code for a possible
cause:
- If database connections are being assigned
to a request but are
not being released when the requests finish processing, ensure that
the application code performs a close() on any opened Connection object
within a finally{} block.
- If there is a steady increase
in servlet engine threads in use,
review application synchronized code blocks for possible deadlock
conditions.
- If there is a steady increase in a JVM heap size,
review application
code for memory leak opportunities, such as static (class-level) collections,
that can cause objects to never get garbage-collected.
- Enable verbose garbage collection on the application server to
help you determine if you have a memory leak problems. This feature
adds detailed statements about the amount of available and in-use
memory to the JVM error log file.
To enable up verbose garbage
collection:
- In the administrative
console, click Servers >
Server Types > Application servers > server_name.
.
Then, under Server Infrastructure, click Java and process management >
Process definition > Java virtual machine, and select Verbose
garbage collection.
- Stop and restart
the application server.
- Periodically, browse the log file
for garbage collection statements.
Look for statements beginning with "allocation failure". This string
indicates that a need for memory allocation has triggered a JVM garbage
collection, to release unused memory. Allocation failures are normal
and do not necessarily indicate a problem. However, the statements
that follow the allocation failure statement show how many bytes are
needed and how many are allocated. If these bytes needed statements
indicate that the JVM keeps allocating more memory for its own use,
or that the JVM is unable to allocate as much memory as it needs,
there might be a memory leak.
You
can also use the Tivoli performance
viewer to detect memory leak problems.
- Determine
if the application server is running
out of memory. If you determine that the application server is running
out of memory, one of the following situations might be occurring:
- There is a memory leak in application code that
you must address.
To pinpoint the cause of a memory leak, enable the RunHProf property
on the Java Virtual Machine page of the administrative console. server_name is
the name of the problem application server. After you enable the RunHProf property,
you must:
- Set the HProf Arguments field
to a value similar to depth=20,file=heapdmp.txt.
This value shows exception stacks to a maximum of 20 levels, and
saves the heapdump output to the app_server_root/bin/heapdmp.txt file.
- Save the settings.
- Stop and restart the application server.
- If possible, reenact the scenario or access the resource that
caused the application server's process to spontaneously close, or
its web modules to stop responding to new requests. Then stop the
application server. If you cannot reenact the scenario or access the
resource, wait until the problem reoccurs, and then stop the application
server.
- Examine the file into which the heap dump was saved.
For example,
examine the app_server_root/bin/heapdmp.txt file:
- Search for the string, "SITES BEGIN". This finds
the location
of a list of Java objects in memory, which shows the amount of memory
allocated to the objects.
- The list of Java objects occurs
each time there was a memory allocation
in the JVM. There is a record of what type of object the memory instantiated
and an identifier of a trace stack, listed elsewhere in the dump,
that shows the Java method that made the allocation.
- The list
of Java object is in descending order by number of bytes
allocated. Depending on the nature of the leak, the problem class
should show up near the top of the list, but this is not always the
case. Look throughout the list for large amounts of memory or frequent
instances of the same class being instantiated. In the latter case,
use the ID in the trace stack column to identify
allocations occurring repeatedly in the same class and method.
- Examine
the source code indicated in the related trace stacks
for the possibility of memory leaks.
- The
JVM is using the maximum heap size that it is allowed to use.
In this situation, you should increase the maximum heap size setting
for application server if you have enough storage available to do
so.
- The server runtime is experiencing a problem. If you determine
that there is a problem with the server runtime, make sure that you
have applied all of the service updates for the product. If, after
you apply all of the service updates, the problem still exists, contact IBM® Support.
- Browse the thread dump for clues:
The
JVM creates a thread dump whenever an application server process spontaneously
closes. You can also force an application to create a thread dump.
After a dump is created, you can check the dump for clues as to why
new requests are not being processed.
To force a thread dump:
- Using the wsadmin command prompt, get a handle to
the problem
application server:
wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server_name,*]
where server_name is
the name of your server.
- Generate the thread dump:
wsadmin>$AdminControl invoke $jvm dumpThreads
- Look for an output file, in the installation root
directory for the product, with a name like javacore.date.time.id.txt.
After the application creates the dump, you can check for
the following clues:
- "Error"
or "exception information" strings at
the beginning of the file. These strings indicate the thread that
caused the application server process to spontaneously close. These
strings are not be present if you forced the dump.
- Look
at the snapshot of each thread in the process.
The thread dump contains a snapshot of each thread in the process,
starting in the section labeled "Full thread dump."
- Look for threads with a description that contains "state:R". Such
threads are active and running when the dump is forced, or the process
exited.
- Look for multiple threads in the same Java application
code source
location. Multiple threads from the same location might indicate
a deadlock condition (multiple threads waiting on a monitor) or an
infinite loop, and help identify the application code with the problem.
IBM Support has documents and tools that can
save you time gathering information needed to resolve problems as
described in
Troubleshooting help from IBM. Before opening
a problem report, see the Support page: