MustGather: No response (hang) or performance degradation on Solaris
 Technote (troubleshooting)
 
Problem(Abstract)
Collecting data for when your IBM® WebSphere® Application Server is not responding (hangs) on a Solaris operating system. Gathering this information before calling IBM Support will help familiarize you with the troubleshooting process and save you time.
 
Resolving the problem

Do you want to automate the collection of MustGather data?
Collecting the following MustGather information has now been automated in the IBM Support Assistant. For more information about automating data collection, see Using IBM Support Assistant to collect MustGather data.

If you have already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

No response (hang) or performance degradation specific MustGather information
Follow these instructions for initial environment set up before you re-create the hang problem:
  1. If possible, following instructions on Enabling verbosegc on WebSphere Application Server before recreating the problem.

  2. Synchronize clocks on all Web server and WebSphere Application Server systems. If clocks on the Web server and WebSphere Application Server systems were not synchronized for some reason, provide information which clock was faster and what was the difference.

  3. If possible, clean all Web server, plug-in and WebSphere Application Server and FFDC logs.

To begin troubleshooting your problem, execute the following commands during the time when the WebSphere Application Server system JVM does not respond:
  1. Run the following command

    netstat -an > netstat.out

  2. Run the following command:

    lsof -p [PID_of_hung_JVM] > lsof.out

  3. Run the following command:

    vmstat 5 12 > vmstat1.out

  4. Run the following command:

    /usr/proc/bin/pfiles [PID_of_hung_JVM] > pfiles.out

  5. Run the following command:

    /usr/proc/bin/pmap [PID_of_hung_JVM] > pmap.out

  6. Run the following command:

    /usr/proc/bin/pstack [PID_of_hung_JVM] > pstack1.out

  7. Run the following command:

    kill -3 [PID_of_hung_JVM]

  8. Wait two minutes.

  9. Run the following command:

    /usr/proc/bin/pstack [PID_of_hung_JVM] > pstack2.out

  10. Run the following command:

    kill -3 [PID_of_hung_JVM]

  11. Wait two minutes.

  12. Run the following command:

    /usr/proc/bin/pstack [PID_of_hung_JVM] > pstack3.out

  13. Run the following command:

    kill -3 [PID_of_hung_JVM]

  14. Run the following command:

    vmstat 5 12 > vmstat2.out

  15. Capture the following machine environment information:

    env > env.out
    ulimit -a > ulimit.out
    uname -a > uname.out
    showrev -p > showrev.out
    pkginfo -l > pkginfo.out

  16. Gather the following files:
    • For WebSphere Application Server V6.0:
      • The server.xml file located in the following directory:

        profile_root/config/cells/cell_name/nodes/
        node_name
        /servers/server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        profile_root/logs/server_name

      • Everything in the following directory:

        profile_root/logs/ffdc

      • All documents requested for all releases below

    • For WebSphere Application Server V5.0 and 5.1:
      • The server.xml file located in the following directory:

        install_root/config/cells/nodes/node_name/
        servers/
        server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        install_root/logs/server_name

      • Everything in the following directory:

        install_root/logs/ffdc

      • All documents requested for all releases below

    • For all releases:
      • All of the preceding *.out files collected

      • Include the Application Server native_stderr and native_stdout, if they are located in a different directory

      • The Web server access and error logs

  17. Follow instructions to send diagnostic information to IBM support.   

If you want to analyze the Java™ thread dumps yourself, download the ThreadAnalyzer tool. ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as summarizing the state of threads within the Application Server.

For all technotes, downloads, and educational materials specific to a hang or performance degradation, search the WebSphere Application Server Support site.
 
Related information
How to Enable Verbosegc on WebSphere
ThreadAnalyzer Technology Preview
Submitting information to IBM support
Steps to get support for WebSphere Application Server
MustGather: Readme first
Troubleshooting guide
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation Solaris 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): Solaris
Software version: 6.0
Software edition:
Reference #: 1052644
IBM Group: Software Group
Modified date: Feb 19, 2007