MustGather: Administrative Server, Application Server, Node Agent, Deployment Manager hang problems
 Technote (FAQ)
 
Problem
MustGather for problems with the WebSphere® Application Server Hangs/Performance Degradation component. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
 
Solution
If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Hangs/Performance Degradation specific MustGather information for the following:
  • Administrative Server
  • Application Server
  • Node Agent
  • Deployment Manager

For all the Java™ Virtual Machine (JVM) hang issues, collect the following information. There is a list of links provided at the bottom of this document that routes you to the how-to documents for steps that require additional instructions.
  • For releases of V3.5 and V4.0:

    1. Thread Dump from the hung JVM. "Appendix A: How to collect JVM thread dump for the server in hang"

    2. Trace dump for the hung JVM. "Appendix B: How to collect trace dump for the server in hang"

    3. The admin.config file from the $WAS_HOME/bin/ directory

    4. The tracefile from the $WAS_HOME/logs/ directory

    5. UNIX®: The nanny.trace from the $WAS_HOME/logs/ directory

    6. Corresponding application server's stderr and stdout files

    7. The product.xml from the $WAS_HOME\properties\com\ibm\WebSphere directory

    8. A full XML export for the domain

    Note: For multi-node domains, the admin.config, tracefile, nanny.trace, and product.xml files from each node are required.

  • For releases of V5.0:
    1. Thread Dump from the hung JVM. "Appendix A: How to collect JVM thread dump for the server in hang"

    2. Trace dump from the hung JVM. "Appendix B: How to collect trace dump for the server in hang"

    3. Collector tool output
      • Running the collector tool V5.0
      • Running the collector tool V5.1

  • Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the component, search the WebSphere Application Server support site.


Appendix A: How to collect JVM thread dump for the server in hang

For WebSphere Application Server V3.5 and V4.0 (all releases)
Administrative Server/Application Server Hang
UNIX:

Identify the PID and follow instructions in the appropriate technotes to collect required thread dump and other details.
Windows®:
Follow the instructions in the technote to collect required thread dump.
For WebSphere Application Server V5.0 (all releases)
AppServer, Node Agent, Deployment Manager Hang Issues
UNIX:
Option 1:
Identify the PID and follow the instructions in the appropriate technotes to collect required thread dump
Option 2:
Note: The title is referring to Windows, but it can be used for UNIX environment Windows:
Appendix B: How to collect trace dump for the server in hang

WebSphere V3.5.x and V4.0.x
Admin Server/Application Server Hang

Collect the server run time traces using DrAdmin.
Link for DrAdmin
http://www-1.ibm.com/support/docview.wss?uid=swg21141474

DrAdmin also can be used to collect a thread dump in all operating systems.
In WebSphere V5.0.x the DrAdmin feature has been deprecated. Use wsadmin scripting to collect the information:
1. Save the below runTimeTrace.jacl script to your WAS_BASE or ND_HOME/bin directory.

2. Start wsadmin from command line with -f option.
Example:
wsadmin.bat/sh -f runTimeTrace.jacl

Note: You can place the runTimeTrace.jacl to some other directory; you must provide the fully-qualified name in starting wsadmin.
Example:
File downloaded to C:\myDir\runTimeTrace.jacl

wsadmin -f C:\myDir\runTimeTrace.jacl
3. There are several prompts that will ask for environment specific information.

At this prompt:
Enter New Trace String as per IBM Support recommendations
>
Type the following response:

com.ibm.ejs.sm.*=all=enabled
Enter New Trace String as per IBM Support recommendations
>com.ibm.ejs.sm.*=all=enabled
Notes:
  1. Federated Cell Environment
    This script uses $AdminConfig and $AdminControl Objects. If you are trying to connect wsadmin to a NodeAgent or an ApplicationServer, port the script fails.

    $AdminConfig service is not available for the NodeAgent or Application Servers in a federated cell environment. If you want to connect wsadmin to a particular Server/NodeAgent SOAP port (Federated Cell Env), comment the $AdminConfig function from this script

    #set kenAvailableServer [$AdminConfig list Server]
    #puts "$kenAvailableServer"

    We suggest that in a federated cell environment, always start wsadmin using dmgr port (ND SOAP Port). This enables access to all servers in the cell.
  2. Stand alone Base Environment
    If you are connecting wsadmin to a Application Server's SOAP port, no editing of the file is required.
  3. The Web Console also can be used to set runtime traces.
Additional Information:
 
runTimeTrace.jacl
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): HP-UX
Software version: 4.0
Software edition:
Reference #: 1141514
IBM Group: Software Group
Modified date: Apr 30, 2005