MustGather: No response (hang) or performance degradation on AIX
 Technote (troubleshooting)
 
Problem(Abstract)
MustGather for when your IBM® WebSphere® Application Server is not responding (hangs) on the AIX® platform. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
 
Resolving the problem

Do you want to automate the collection of MustGather data?
Collecting the following MustGather information has now been automated in the IBM Support Assistant. For more information about automating data collection, see Using IBM Support Assistant to collect MustGather data.

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

No response (hang) or performance degradation specific MustGather information
To troubleshoot a hung Java™ virtual machine (JVM™), collect the following information when a hang occurs.

Follow these instructions for initial environment set up before you re-create the hang problem:
  1. If possible, follow instructions on enabling verbosegc on WebSphere Application Server before recreating the problem.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems. If clocks on the Web server and WebSphere Application Systems were not synchronized, state which clock was faster and the difference between the two times.

  3. If possible, clear all Web server, plug-in, WebSphere Application Servers and FFDC logs.


When you experience the hang, enter the following commands from a command-line prompt:
  1. netstat -an > netstat.out

    If the Web server is remote, gather netstat output on the Application Server system and the Web server system.

  2. vmstat 5 12 > vmstat.out

  3. Collect CPU usage statistics by entering the following command:

    tprof -k -s -e -x sleep 60

  4. Generate a set of three Javacores for the hung JVM by issuing the following commands:

    A. kill -3 [PID_of_hung_JVM]

    B. Wait two minutes.

    C. kill -3 [PID_of_hung_JVM]

    D. Wait two minutes

    E. kill -3 [PID_of_hung_JVM]

  5. ./dbxtrace_aix.sh -a [PID_of_hung_JVM] > dbx-hung.out

  6. If you are serving static content from a WebServer, please determine whether your WebServer is responding to requests for that information. Determine whether you can serve dynamic documents such as servlets or JSPs when bypassing the WebServer and WebSphere Application Server plug-in.

  7. Collect the following information:
    • For WebSphere Application Server V6.0 and v6.1 releases:
      • The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/
        node_name/
        servers/server_name
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root/profiles/profile_name/logs/server_name directory
      • Everything in the install_root/profiles/profile_name/logs/ffdc directory
      • All documents requested for all releases below

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root/config/cells/cell_name/nodes/node_name/servers/server_name directory
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root/logs/server_name directory
      • Everything in the install_root/logs/ffdc directory
      • All documents requested for all releases below

    • For all releases:
      • dbx*.out
      • All javacore.txt files created
      • All netstat*.out files
      • All vmstat*.out files
      • Send all files generated by tprof, particularly _prof.all. These files are in the directory from which the tprof command was run and they begin with an underscore "_". Some versions of AIX might produce only a sleep.prof file.
      • Send the configuration and log files from the Web server system. For example, for IBM HTTP Server and Apache Web, server provide the httpd.conf, access and error logs. If you are running the Sun ONE™ 6 Web server, provide magnus.conf, obj.conf, access and error logs.

  8. Follow instructions to send diagnostic information to IBM support.

If you want to analyze the Java thread dumps yourself, download the IBM Thread and Monitor Dump Analyzer for Java Technology.
ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as for summarizing the state of threads within WebSphere Application Server.

For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.

 
Related information
How to enable verbosegc in WebSphere
IBM Thread and Monitor Dump Analyzer
Submitting information to IBM support
Steps for getting support
MustGather: Readme first
Troubleshooting guide
 
dbxtrace_aix.sh
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation AIX 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): AIX
Software version: 6.1
Software edition:
Reference #: 1052641
IBM Group: Software Group
Modified date: Jan 5, 2007