MustGather: Administrative Server, Application Server, Node Agent, Deployment Manager hang issues

Technote (FAQ)
Problem
MustGather for problems with the WebSphere® Application Server Hangs/Performance Degradation component. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
Solution
Administrative Server, Application Server, Node Agent, Deployment Manager hang issues

If you already contacted support, continue to the component's specific MustGather information. Otherwise, click: MustGather: Read First for all WebSphere Application Server Products.

Hangs/Performance Degradation specific MustGather information for the following:
  • Administrative Server
  • Application Server
  • Node Agent
  • Deployment Manager

For all the Java™ Virtual Machine (JVM) hang issues, collect the following information. There is a list of links provided at the bottom of this document that routes you to the how-to documents for steps that require additional instructions.

For WebSphere Application Server V3.5 and V4.0 (all releases):
  1. Thread Dump from the hung JVM."Appendix A: How to collect JVM thread dump for the server in hang"
  2. Trace dump for the hung JVM."Appendix B: How to collect trace dump for the server in hang"
  3. The admin.config file from the $WAS_HOME/bin/ directory
  4. The tracefile from the $WAS_HOME/logs/ directory
  5. UNIX®: The nanny.trace from the $WAS_HOME/logs/ directory
  6. Corresponding application server's stderr and stdout files
  7. The product.xml from the $WAS_HOME\properties\com\ibm\WebSphere directory
  8. A full XML export for the domain
    Note:For multi-node domains, the admin.config, tracefile, nanny.trace, and product.xml files from each node are required.

    For WebSphere Application Server V5.0 (all releases):
    1. Thread Dump from the hung JVM."Appendix A: How to collect JVM thread dump for the server in hang"
    2. Trace dump from the hung JVM."Appendix B: How to collect trace dump for the server in hang"
    3. All the logs from corresponding server (Application Server, Cluster member, nodeagent, and/or Deployment Manager)

      Example:
      For AppServer hang:
      $WAS_BASE_HOME/logs/server1

      For nodeagent hang:
      $WAS_BASE_HOME/logs/nodeagent

      For Deployment manager hang:
      $WAS_ND_HOME/logs/dmgr
    4. Obtain the versionInfo for the corresponding node; issue the following command from the bin directory of the specific installation:
        WAS_BASE or ND_HOME/bin/versionInfo.bat/sh >Base_version_node1.txt

        Note: If you have multiple BASE nodes in a Cell, collect the version information for all BASE nodes and for ND.
    Obtain the genVersionReport; issue the following command from the bin directory of the specific installation and send the collected details:
        WAS_BASEor ND_HOME/bin/genVersionReport.bat/sh -format html -file myBaseVersionReport.html -efixes -ptfs -componentDetail

          Syntax:
          -format html Provides html output

          -file filename.html The name of the output file

          -efixes Includes all intern fixes and monthly cumulatives installed

          -ptfs Includes all fix packs installed

          -componentDetail Provides component information

        8. Calling IBM support at 800-IBM-SERV, then send the documentation:
          a. email: weblev2@us.ibm.com (<10 MB)
          b. FTP (>10 MB)
            testcase.software.ibm.com
            userid: anonymous
            password: your e-mail address
            Then place the file /ps/toibm/internet/pmr number

      Appendix A: How to collect JVM thread dump for the server in hang

      For WebSphere Application Server V3.5 and V4.0 (all releases)
      Administrative Server/Application Server Hang
      UNIX:

      Identify the PID and follow instructions in the appropriate technotes to collect required thread dump and other details.
      Windows®:
      Follow the instructions in the technote to collect required thread dump.
      For WebSphere Application Server V5.0 (all releases)
      AppServer, Node Agent, Deployment Manager Hang Issues
      UNIX:
      Option 1:
      Identify the PID and follow the instructions in the appropriate technotes to collect required thread dump
      Option 2:
      Note: The title is referring to Windows, but it can be used for UNIX environmentWindows:
      Appendix B: How to collect trace dump for the server in hang

      WebSphere V3.5.x and V4.0.x
      Admin Server/Application Server Hang

      Collect the server run time traces using DrAdmin.
      In WebSphere V5.0.x the DrAdmin feature has been deprecated. Use wsadmin scripting to collect the information:
        1. Save the below runTimeTrace.jacl script to your WAS_BASE or ND_HOME/bin directory.

        2. Start wsadmin from command line with -f option.
          Example:
          wsadmin.bat/sh -f runTimeTrace.jacl

        Note: You can place the runTimeTrace.jacl to some other directory; you must provide the fully-qualified name in starting wsadmin.
          Example:
          File downloaded to C:\myDir\runTimeTrace.jacl

          wsadmin -f C:\myDir\runTimeTrace.jacl
        3. There are several prompts that will ask for environment specific information.

        At this prompt:
          Enter New Trace String as per IBM Support recommendations
          >
        Type the following response:

          com.ibm.ejs.sm.*=all=enabled
          Enter New Trace String as per IBM Support recommendations
          >com.ibm.ejs.sm.*=all=enabled
        Notes:
        1. Federated Cell Environment
          This script uses $AdminConfig and $AdminControl Objects. If you are trying to connect wsadmin to a NodeAgent or an ApplicationServer, port the script fails.

          $AdminConfig service is not available for the NodeAgent or Application Servers in a federated cell environment. If you want to connect wsadmin to a particular Server/NodeAgent SOAP port (Federated Cell Env), comment the $AdminConfig function from this script

          #set kenAvailableServer [$AdminConfig list Server]
          #puts "$kenAvailableServer"

          We suggest that in a federated cell environment, always start wsadmin using dmgr port (ND SOAP Port). This enables access to all servers in the cell.
        2. Stand alone Base Environment
          If you are connecting wsadmin to a Application Server's SOAP port, no editing of the file is required.
        3. The Web Console also can be used to set runtime traces.
      Additional Information:

      Follow instructions to send diagnostic information to IBM support.

      For a listing of all technotes, downloads, and educational materials specific to the component, search the WebSphere Application Server support site.












      Document Information

      Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Hangs/Performance Degradation
      Operating system(s): Multi-Platform
      Software version: 3.5, 4.0, 5.0
      Software edition: Advanced, Base, Network Deployment
      Reference #: 1141514
      IBM Group: Software Group
      Modified date: 2004-12-06