MustGather: No response (hang) or performance degradation on Solaris

Technote (FAQ)
Problem
MustGather for when your WebSphere® Application Server is not responding (hangs) on the Solaris™ platform. Gathering this information before calling IBM® Support will help familiarize you with the troubleshooting process and save you time.
Solution
If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


No response (hang) or performance degradation specific MustGather information

Follow these instructions for initial environment set up before you re-create the hang problem:
  1. If possible, following instructions on How to enable verbosegc on WebSphere Application Server before recreating the problem.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems.If clocks on the Web server and WebSphere Application Server systems were not synchronized for some reason, provide information which clock was faster and what was the difference.

  3. If possible, clean all Web server, plug-in and WebSphere Application Server and FFDC logs.


At the next occurrence of the hang, do the following on the the Application Server system:

To begin troubleshooting your problem, execute the following commands during the time when the WebSphere Application Server system Java™ Virtual Machine (JVM™) does not respond:
  1. netstat -an > netstat.out

  2. lsof -p [PID_of_hung_JVM]> lsof.out

  3. vmstat 5 12 > vmstat1.out

  4. /usr/proc/bin/pfiles [PID_of_hung_JVM]> pfiles.out

  5. /usr/proc/bin/pmap [PID_of_hung_JVM]> pmap.out

  6. /usr/proc/bin/pstack [PID_of_hung_JVM]> pstack1.out

  7. kill -3 [PID_of_hung_JVM]

  8. Wait two minutes.

  9. /usr/proc/bin/pstack [PID_of_hung_JVM]> pstack2.out

  10. kill -3 [PID_of_hung_JVM]

  11. Wait two minutes.

  12. /usr/proc/bin/pstack [PID_of_hung_JVM]> pstack3.out

  13. kill -3 [PID_of_hung_JVM]

  14. vmstat 5 12 > vmstat2.out

  15. Capture the following machine environment information:
    env > env.out
    ulimit -a > ulimit.out
    uname -a > uname.out
    showrev -p showrev.out
    pkginfo -l pkginfo.out

  16. Gather the following files:
    • For WebSphere Application Server V6.0 release:
      • The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root/profiles/profile_name/logs/server_name directory
      • Everything in the install_root/profiles/profile_name/logs/ffdc directory
      • All documents requested For all releases below

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root/config/cells/nodes/node_name/servers/server_name directory
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root/logs/server_name directory
      • Everything in the install_root/logs/ffdc directory
      • All documents requested for all releases below

    • For WebSphere Application Server V4.0 release:
      • A XMLConfig full export.
      • plugin-cfg.xml and native.log
      • Everything in the install_root/logs directory from the WebSphere Application Server system.
      • All documents requested for all releases below

    • For WebSphere Application Server V3.5 release:
      • A XMLConfig full export.
      • queues, rules, vhosts property files and trace.log.webserver_name.date
      • Everything in the install_root/logs directory from the WebSphere Application Server system.
      • All documents requested for all releases below

    • For all releases:
      • All of the *.out files collected above
      • Include the Application Server native_stderr and native_stdout, if they are located in a different directory
      • Web server's access and error logs
      • If the Web server is remote, send the appropriate file from the Web server system including Web server configuration files and Web server logs. For example for IBM HTTP Server and Apache Web server provide httpd.conf, access anderror logs, for Sun ONE™ 6 Web server provide magnus.conf, obj.conf, access anderror logs.


  17. Follow instructions to send diagnostic information to IBM support.


If you want to analyze the Java thread dumps yourself, download the ThreadAnalyzer tool. ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as summarizing the state of threads within the Application Server.

For all technotes, downloads, and educational materials specific to the Hangs or Performance Degradation component, search the WebSphere Application Server Support site.

Related information
How to Enable Verbosegc on WebSphere

ThreadAnalyzer Technology Preview

Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Hangs/Performance Degradation
Operating system(s): Solaris
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server
Reference #: 1052644
IBM Group: Software Group
Modified date: 2004-12-08