If an application server process spontaneously closes, or Web modules
stop responding to new requests, it is important that you quickly determine
why this stoppage is occurring. You can use some of the following techniques
to determine whether the problem is a Web module problem or an application
server environment problem.
If an application server process spontaneously closes, or Web modules running
on the application server stop responding to new requests:
- Try to Isolate the problem by installing the Web modules on different
servers, if possible.
- Check the product directory structure for a file with
a name like javacore[number].txt. This file is a Java
thread dump file that the JVM creates if an application server process spontaneously
closes.
- Use the Tivoli performance viewer to determine if any of the application
server resources, such as the Java heap, or database connections, have reached
their maximum capacity. If there is a resource problem, review the application
code for a possible cause:
- If database connections are being assigned to a request but are not being
released when the requests finish processing, ensure that the application
code performs a close() on any opened Connection object within
a finally{} block.
- If there is a steady increase in servlet engine threads in use, review
application synchronized code blocks for possible deadlock conditions.
- If there is a steady increase in a JVM heap size, review application code
for memory leak opportunities, such as static (class-level) collections, that
can cause objects to never get garbage-collected.
- Enable verbose garbage collection on the application server to help you
determine if you have a memory leak problems. This feature adds detailed statements
about the amount of available and in-use memory to the JVM error log file.
To
enable up verbose garbage collection:
- In the administrative console, click Servers
> Application Servers > server_name. . Then, under Server Infrastructure,
click Java and Process Management > Process Definition > Java Virtual Machine,
and enable Verbose Garbage Collection.
- Stop and restart the application server.
- Periodically, browse the log file for garbage collection statements. Look
for statements beginning with "allocation failure". This string indicates
that a need for memory allocation has triggered a JVM garbage collection,
to release unused memory. Allocation failures are normal and do not necessarily
indicate a problem. However, the statements that follow the allocation failure
statement show how many bytes are needed and how many are allocated. If these
bytes needed statements indicate that the JVM keeps allocating more memory
for its own use, or that the JVM is unable to allocate as much memory as it
needs, there might be a memory leak.
You can also use the Tivoli performance
viewer to detect memory leak problems.
- Determine if the application server is running out of
memory. If you determine that the application server is running out of memory,
one of the following situations might be occurring:
- There is a memory leak in application code that you must address. To pinpoint
the cause of a memory leak, enable the RunHProf property on the Java
Virtual Machine page of the administrative console. server_name is
the name of the problem application server. After you enable the RunHProf property,
you must:
- Set the HProf Arguments field to a value similar to depth=20,file=heapdmp.txt.
This value shows exception stacks to a maximum of 20 levels, and saves the
heapdump output to the app_server_root/bin/heapdmp.txt file.
- Save the settings.
- Stop and restart the application server.
- If possible, reenact the scenario or access the resource that caused the
application server's process to spontaneously close, or its Web modules to
stop responding to new requests. Then stop the application server. If you
cannot reenact the scenario or access the resource, wait until the problem
reoccurs, and then stop the application server.
- Examine the file into which the heap dump was saved. For example, examine
the app_server_root/bin/heapdmp.txt file:
- Search for the string, "SITES BEGIN". This finds the location of a list
of Java objects in memory, which shows the amount of memory allocated to the
objects.
- The list of Java objects occurs each time there was a memory allocation
in the JVM. There is a record of what type of object the memory instantiated
and an identifier of a trace stack, listed elsewhere in the dump, that shows
the Java method that made the allocation.
- The list of Java object is in descending order by number of bytes allocated.
Depending on the nature of the leak, the problem class should show up near
the top of the list, but this is not always the case. Look throughout the
list for large amounts of memory or frequent instances of the same class being
instantiated. In the latter case, use the ID in the trace stack column
to identify allocations occurring repeatedly in the same class and method.
- Examine the source code indicated in the related trace stacks for the
possibility of memory leaks.
- The JVM is using the maximum heap size that it is allowed to use. In this
situation, you should increase the maximum heap size setting for application
server if you have enough storage available to do so.
- The server runtime is experiencing a problem. If you determine that there
is a problem with the server runtime, make sure that you have applied all
of the service updates for the product. If, after you apply all of the service
updates, the problem still exists, contact IBM Support.
- Browse the thread dump for clues:
The JVM creates
a thread dump whenever an application server process spontaneously closes.
You can also force an application to create a thread dump. After a dump is
created, you can check the dump for clues as to why new requests are not being
processed.
To force a thread dump:
- Using the wsadmin command prompt, get a handle to the problem application
server:
wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
- Generate the thread dump:
wsadmin>$AdminControl invoke $jvm dumpThreads
- Look for an output file in the WebSphere Application Server
installation root directory with a name like javacore.date.time.id.txt.
After the application creates the dump, you can check for the following
clues:
- "Error" or "exception information" strings at the beginning
of the file. These strings indicate the thread that caused the application
server process to spontaneously close. These strings are not be present if
you forced the dump.
- Look at the snapshot of each thread in the process. The
thread dump contains a snapshot of each thread in the process, starting in
the section labeled "Full thread dump."
- Look for threads with a description that contains "state:R". Such threads
are active and running when the dump is forced, or the process exited.
- Look for multiple threads in the same Java application code source location.
Multiple threads from the same location might indicate a deadlock condition
(multiple threads waiting on a monitor) or an infinite loop, and help identify
the application code with the problem.
IBM Support has documents and tools that can save you time
gathering information needed to resolve problems as described in
Troubleshooting help from IBM
. Before opening a problem report, see the Support
page: