MustGather: No response (hang) or performance degradation on Linux
 Technote (troubleshooting)
 
Problem(Abstract)
Collecting data for when your IBM® WebSphere® Application Server is not responding (hangs) on the Linux® operating system. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Resolving the problem

Do you want to automate the collection of MustGather data?
Collecting the following MustGather information has now been automated in the IBM Support Assistant. For more information about automating data collection, see Using IBM Support Assistant to collect MustGather data.

If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Collecting MustGather information for an Application Server hang or performance degradation

The following steps should be taken prior to reproducing the hang or performance degradation issue:
  1. Enable verbosegc on WebSphere Application Server. This is necessary for debugging performance issues and does not have significant impact on performance.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems. If clocks on the Web server and WebSphere Application Server systems were not synchronized for some reason, provide information on which clock was faster and what was the difference.

  3. If possible, clean all Web server, plug-in and WebSphere Application Server and FFDC logs.

When you are able to reproduce the problem or when the problem recurs, perform the following steps to collect the MustGather information needed to debug the problem:
  1. Run the following command:

    netstat -an > netstat.out

    Note: If the Web server is remote, gather netstat output on the Application Server system and the Web server systems.

  2. Run the following command:

    vmstat 5 12 >> vmstat.log

  3. Run the following command:

    ps -eLf > ps_eLf.txt

  4. Check to see if you can serve a static HTML document from the Web server.

  5. Check to see if you can serve dynamic documents such as servlets or JSPs when bypassing the Web server and WebSphere Application Server plug-in.

  6. Find the PID of the failing Java™ virtual machine (JVM). You can find the PID using multiple ways, here are couple of ways:
    • A file named server_name.pid will get updated with the PID of the application server.

    • For all versions, the process ID will be printed in the SystemOut.log file at each restart. You will have to pick the latest one from the log file.

  7. Run the following command:

    kill -3 [PID of hung JVM]

    Note: The preceding and following kill -3 commands will create javacore.txt files in the install_root, install_root/bin, or in the configured working directory.

  8. Wait two minutes.

  9. Run the following command:

    kill -3 [PID of hung JVM]

  10. Wait two minutes.

  11. Run the following command:

    kill -3 [PID of hung JVM]

  12. Wait two minutes.

  13. Collect CPU usage statistics by running the following command:

    top -d 60 -n 2 -c -b > top.log

  14. In some cases, the kill -3 commands do not produce javacores. In other cases, the javacores do not contain full Java stack traces. A Linux operating system core will be needed to ascertain the Java thread stack in this case. Perform the following steps to generate the core file:
    1. Attach the GDB debugger to the Java process of the hung JVM:

      gdb JAVA_HOME/jre/bin/java [PID of hung JVM]

    2. Run a gcore command from the gdb command line:

      (gdb) gcore

    3. Detach the gdb debugger from the process:

      (gdb) detach

    4. The preceding steps should create a system core file without terminating the JVM process. If for some reason the core is not generated, run the following command to terminate the process and generate a system core:

      kill -ABRT  [PID of hung JVM]

      Note: The preceding gdb/kill -ABRT command creates a core in the install_root/bin or the operating_system_root/tmp directory.

  15. The preceding step creates a core file. You will need to follow the directions in MustGather: Crash on Linux and then send in the requested diagnostic information.

  16. Gather the following files:
    • For WebSphere Application Server V6.0 and V6.1:
      • The server.xml file located in the following directory:

        profile_root/config/cells/cell_name/nodes/
        node_name/servers/server_name

      • The plugin-cfg.xml and http_plugin.log files
      • Everything in the following directory:

        profile_root/logs/server_name

      • Everything in the following directory:

        profile_root/logs/ffdc

      • All documents requested for all releases below.

    • For WebSphere Application Server V5.0 and 5.1:
      • The server.xml file located in the following directory:

        install_root/config/cells/nodes/node_name/
        servers/
        server_name

      • The plugin-cfg.xml and http_plugin.log files
      • Everything in the following directory:

        install_root/logs/server_name

      • Everything in the following directory:

        install_root/logs/ffdc

      • All documents requested for all releases below.

    • For all WebSphere Application Server releases
      • ps_eLf.txt
      • All javacore.txt files created
      • All netstat.out file
      • All vmstat*.out files
      • top.log
      • Web server access and error logs
      • All files created from processing the system core (core.sdff, output from libsgrabber.sh, and so on)

  17. Follow instructions to send diagnostic information to IBM support.

If you want to analyze the Java thread dumps yourself, download the IBM Thread and Monitor Dump Analyzer for Java Technology. ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as summarizing the state of threads within WebSphere Application Server.

For a listing of all technotes, downloads, and educational materials specific to a hang or performance degradation, search the WebSphere Application Server support site.
 
Related information
How to enable verbosegc
Steps to Getting Support
MustGather: Readme First
Troubleshooting Guide
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Linux, Linux Red Hat - pSeries Edition Independent
Application Servers WebSphere Application Server Enterprise Linux, Linux Red Hat - pSeries Edition Independent
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): Linux
Software version: 6.0
Software edition:
Reference #: 1115785
IBM Group: Software Group
Modified date: Jul 31, 2007