MustGather: No response (hang) or performance degradation on Windows
 Technote (FAQ)
 
Problem
Collecting data for when your IBM® WebSphere® Application Server is not responding (hangs) for V5.0, V5.1, and V6.0 releases on the Windows® operating system. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Solution

Do you want to automate the collection of MustGather data?
Collecting the following MustGather information has now been automated in the IBM Support Assistant. For more information about automating data collection, see Using IBM Support Assistant to collect MustGather data.

If you already contacted support, continue to the Hangs or Performance Degradation MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

No response (hang) or performance degradation specific MustGather information
This technote describes what information you need to begin troubleshooting an Application server that is not responding or is hung. The method used to create a thread dump differs from earlier releases of the product. Use the following directions to gather the necessary documentation to debug the problem:

Follow these instructions for initial environment set up before you re-create the hang problem:
  1. If possible, following instructions on Enabling verbosegc on WebSphere Application Server before recreating the problem.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems. If clocks on the Web server and WebSphere Application Server systems were not synchronized for some reason, provide information which clock was faster and what was the difference.

  3. If possible, clean all Web server, plug-in, WebSphere Application Server and FFDC logs.

  4. Complete steps 1 - 9 in MustGather: Getting user.dmp when hangs/performance degradation prevents generating a javacore to configure drwtsn32 tool to produce a user.dump file from the Java process. Also follow the instructions in the same technote to install and configure the Microsoft userdump tool. Installing this tool might require you to restart the server.

When you experience the hang, enter the following commands from a command-line prompt on the WebSphere Application Server machine and any remote Web server systems:
  1. Take the initial output of netstat command to get information about TCP/IP sockets:

    netstat -an > netstat_before.out

  2. Next, from the command prompt, enter the command wsadmin.bat to get a wsadmin command prompt.

    Note: If security is enabled or the default SOAP ports have been changed, you will need to pass additional parameters to the batch file in order to get a wsadmin prompt. For example:

    wsadmin.bat [-host host_name] [-port port_number] [-user userid[-password password]

    Note: You can connect wsadmin to any of the server JVM in the cell. After running the wsadmin command it will display the server process for which it has attached to. Depending on the process that it has attached to, you can get thread dumps for various JVMs. If wsadmin is connected to deployment manager, then you can get thread dumps for any JVM in that cell. If it is attached to a node agent, then you can get thread dumps for any JVM in that Node. If it is attached to a server, then you can get thread dumps only for the server to which has connected to.

  3. Get a handle to the problem application server.

    Note: The contents in brackets "[.....]", along with the brackets, is not optional. It must be entered to set the jvm object. Also, note that there is a space between the words "completeObjectName" and "type":

    wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

    Where server1 is the name of the application server that does not respond (or is hung). If wsadmin is connected to a Deployment Manager and if the server names in the cell are not unique, then you can qualify the JVM with node attribute in addition to process.

  4. Generate the thread dump:

    wsadmin>$AdminControl invoke $jvm dumpThreads

  5. Wait 2 minutes.

  6. Generate the thread dump:

    wsadmin>$AdminControl invoke $jvm dumpThreads

  7. Wait 2 minutes.

  8. Generate the thread dump:

    wsadmin>$AdminControl invoke $jvm dumpThreads

  9. The preceding steps should have created 3 javacores which should be created in the installation root directory (for example, WebSphere\AppServer). In some cases, javacores cannot be created when the JVM is hung or the Java thread stacks in the javacores are not complete. To ensure that you have the data necessary to debug the hang, follow instructions in MustGather: Getting user.dmp when hangs/performance degradation prevents generating a javacore to produce one user.dmp file from the hung Java process.

    Note: If you are not getting javacores from the preceding wsadmin commands, collect a series of three user.dmp files at 2 minute intervals.

  10. Take the final output of netstat command to get information about TCP/IP sockets:

    netstat -an > netstat_after.out

  11. Collect the following diagnostic information:
    • For WebSphere Application Server V6.0:
      • The server.xml file located in the following directory:

        profile_root\config\cells\cell_name\nodes\
        node_name
        \servers\server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        profile_root\logs\server_name

      • Everything in the following directory:

        profile_root\logs\ffdc

    • For WebSphere Application Server V5.0 and V5.1:
      • The server.xml file located in the following directory:

        install_root\config\cells\nodes\node_name\
        servers\
        server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        install_root\logs\server_name

      • Everything in the following directory:

        install_root\logs\ffdc

    • For all releases:
      • All javacore.date.time.id.txt files generated. These files should be in the installation root directory (for example: WebSphere\AppServer).

      • netstat*.out

      • Include the following Application Server log files, if they are located in a different directory:
        • systemErr.log
        • systemOut.log
        • native_stderr.log
        • native_stdout.log

      • If the Web server is remote, send the appropriate file from the Web server system including Web server configuration files and Web server logs. For example for IBM HTTP Server and Apache Web server provide httpd.conf, access and error logs. For SunOne 6 Web server provide magnus.conf, obj.conf, access and error logs.

  12. Follow instructions to send diagnostic information to IBM Support.

If you want to analyze the Java thread dumps yourself, download the
IBM Thread and Monitor Dump Analyzer for Java Technology.
. ThreadAnalyzer is a technology preview tool that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, bottlenecks, and to summarize the state of threads within WebSphere Application Server.

For a listing of all technotes, downloads, and educational materials specific to a hang or performance degradation, search the WebSphere Application Server support site.
 
Related information
How to enable verbosegc for WebSphere
IBM Thread and Monitor Dump Analyzer
Steps to getting support
Submitting information to IBM support
MustGather: Read first
Troubleshooting guide
Not geting javacores? Instructions to get user.dmp.
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation Windows 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): Windows
Software version: 6.0
Software edition:
Reference #: 1111364
IBM Group: Software Group
Modified date: Feb 19, 2007