MustGather: No response (hang) or performance degradation on AIX

Technote (FAQ)
Problem
MustGather for when your WebSphere® Application Server is not responding (hangs) on the AIX® platform. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
Solution
To troubleshooting a hung Java™ virtual machine (JVM™), collect the following information during the time that the hang is occurring.

If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

No response (hang) or performance degradation specific MustGather information

Follow these instructions for initial environment set up before you re-create the hang problem:
  1. If possible, following instructions on How to enable verbosegc on WebSphere Application Server before recreating the problem.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems.If clocks on the Web server and WebSphere Application Systems were not synchronized for some reason, provide information which clock was faster and what was the difference.

  3. If possible, clean all Web server, plug-in and WebSphere Application Server and FFDC logs.

  4. When you first start the application server JVM, run the attached dbx script at the bottom of the page. Run the following script for each clone using clone_name to distinguish between clones:

    ./dbxtrace_aix.sh -a [PID_JVM]> dbx-startup-[clone_name].out


When you experience the hang, enter the following commands from a command-line prompt:
  1. netstat -an > netstat1.out

    If the Web server is remote, gather netstat output on the Application Server system and the Web server system.
  2. vmstat 5 12 > vmstat1.out

  3. This step differs based upon your version of AIX:
    • For AIX V4.3.3:
      svmon -P -m pid -i 600 > svmon.out
    • For AIX V5.1:
      svmon -P -m -r pid-i 600 > svmon.out
    • For AIX V5.2:
      svmon -P pid-m -r -i 600 > svmon.out

  4. Can you serve a static document from the Web server?

  5. Can you serve dynamic documents such as servlets or JSPs when bypassing the Web server and WebSphere Application Server plug-in?

  6. tprof -k -s -e -x sleep 60

  7. kill -3 [PID_of_hung_JVM]

  8. Wait two minutes

  9. kill -3 [PID_of_hung_JVM]

  10. Wait two minutes

  11. kill -3 [PID_of_hung_JVM]

  12. Wait two minutes

  13. ./dbxtrace_aix.sh -a [PID_of_hung_JVM]> dbx-hung.out

  14. netstat -an > netstat2.out

    If the Web server is remote, gather netstat output on the Application Server system AND the Web server systems.

  15. vmstat 5 12 > vmstat2.out

  16. This step differs based upon the version of AIX:
    • For AIX V4.3.3:
      svmon -P -m pid-i 600 > svmon2.out
    • For AIX V5.1:
      svmon -P -m -r pid-i 600 > svmon2.out
    • For AIX V5.2:
      svmon -P pid-m -r -i 600 > svmon2.out

  17. In some cases, the kill -3 commands do not produce javacores. In other cases, the javacores do not contain full Java stack traces. Because of this, you might need to issue the following commands against the PID of the hung JVM:

    netstat -an > netstat2.out
    kill -11 [PID_of_hung_JVM]

  18. The preceding step creates a core file. You will need to follow the directions in the following technote and send in the requested documentation: MustGather: Crash on AIX

    The kill -3 command creates javacore.txt files in the install_root, install_root/bin or in the configured working directory. The kill -11creates a core in the install_root/bin directory or the operating_system_root/tmp directory

  19. Collect the following information:

    • For WebSphere Application Server V6.0 releases:
      • The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
      • plugin-cfg.xml and http_plugin.log

      • Everything in the install_root/profiles/profile_name/logs/server_name directory

      • Everything in the install_root/profiles/profile_name/logs/ffdc directory
      • All documents requested for all releases below

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root/config/cells/nodes/node_name/servers/server_name directory
      • plugin-cfg.xml and http_plugin.log

      • Everything in the install_root/logs/server_name directory

      • Everything in the install_root/logs/ffdc directory
      • All documents requested for all releases below

    • For WebSphere Application Server V4.0 release:
      • A XMLConfig full export.
      • plugin-cfg.xml and native.log

      • Everything in the install_root/logs directory
      • All documents requested for all releases below


    • For WebSphere Application Server V3.5:
      • A XMLConfig full export.
      • queues, rules, vhosts property files and trace.log.webserver_name.date

      • Everything in the install_root/logs directory
      • All documents requested for all releases below

    • For all releases:
      • dbx*.out
      • All javacore.txt files created
      • All netstat*.out files
      • All vmstat*.out files
      • All smvon*.out files
      • Send all files generated bytprof, particularly _prof.all. These files are in the directory from which the tprof command was run and they begin with an underscore "_". Some versions of AIX might produce only a sleep.prof file.
      • All files generated from following MustGather: Crash on AIX (above)
      • If the Web server is remote, send the appropriate file from the Web server system including Web server configuration files and Web server logs. For example for IBM HTTP Server and Apache Web server provide httpd.conf, access and error logs, for Sun ONE™ 6 Web server provide magnus.conf, obj.conf, access anderror logs.
  20. Follow instructions to send diagnostic information to IBM support.

If you want to analyze the Java thread dumps yourself, download the ThreadAnalyzer tool. ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as summarizing the state of threads within WebSphere Application Server.

For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.

Related information
How to enable verbosegc in WebSphere

ThreadAnalyzer Technology Preview

Submitting information to IBM support

Steps for getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Hangs/Performance Degradation
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server
Reference #: 1052641
IBM Group: Software Group
Modified date: 2004-12-08