MustGather: Getting user.dmp when no response (hang) or performance degradation prevents generating a javacore.
 Technote (troubleshooting)
 
Problem(Abstract)
This MustGather explains how to force a user.dmp file. IBM® WebSphere® Application Server Administrators should follow these directions when experiencing a hang or degradation in performance and javacores are not being produced through DrAdmin or wsadmin on the Microsoft® Windows® operating system. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Resolving the problem

Do you want to automate the collection of MustGather data?
Collecting the following MustGather information has now been automated in the IBM Support Assistant. For more information about automating data collection, see Using IBM Support Assistant to collect MustGather data.

If you have already contacted support, continue on to the hang or performance degradation MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Steps to setup a Microsoft Windows environment to generate Dr. Watson logs and Userdump files
  1. On the Start menu, click Run.

  2. Enter drwtsn32, and then click OK. This will display the DrWatson window.

  3. Type the location to where you want the Dr Watson log to be stored in the field provided for Log File Path.

  4. Type the location and the file name where you want the user.dmp file to be stored in the field provided for Crash Dump.

  5. Set Crash Dump Type radio button to Full.

  6. Make sure that the following options are checked in DrWatson screen:
    • Dump Symbol Table
    • Dump All Thread Contexts
    • Visual Notification
    • Create Crash Dump file
    • For Windows 2003, Crash Dump Type option needs to be set to Windows NT® 4 compatible Full

  7. Click OK.

  8. Click Run on the Start menu.

  9. Enter drwtsn32 -i and click OK. This will enable DrWatson to become the default debugger, and it will generate the DrWatson log and Userdump when an application crashes.

Generating the user.dmp files during a hang situation
The following steps can be used to generate a series of user.dmp files, without interrupting the execution of the java process.

The Userdump.exe tool can be used to generate a user.dmp file of a process that stops responding (hangs). In most cases, this tool will not kill the Java™ Virtual Machine (JVM) and will allow for multiple user.dmp files to be taken during the time of the problem.
  1. Download the Userdump Tool. Extract the files to a safe directory of your choosing. There is no need to install or run setup, the Userdump.exe tool can be used directly from the archive. 

  2. Gather netstat data. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

    netstat -an > netstat_before.out

  3. Create the first of three Userdump files for the hanging process:
    1. Run the Setup.exe program that is included with the Userdump tool from the userdump folder.

      When the Application Server process hangs, from a command prompt, enter the following command:

      userdump pid

      Where pid is the process ID of the hanging java.exe. There are multiple ways to get the PID for the failing application server. The following are a couple of ways:
      • You can look in to the SystemOut.log file for the failing application server and get the PID from the latest server startup log entries.

      • You can find a file named server_name.pid in the logs folder for the failing server, that has PID for the server.

    2. This produces java.dmp file in the directory from where the userdump command was executed.

    3. Rename the java.dmp file to java1.dmp.

    4. Wait 2 minutes.

    5. Create the second of three userdump files for the hanging processes, renaming java.dmp to java2.dmp.

    6. Create the third of three userdump files for the hanging processes, renaming java.dmp to java3.dmp.

    Note: If the preceding steps do not appear to work, see the section later in this document named Alternative method for using drwtsn32 to force a user.dmp However, the alternative method kills the java process.

  4. Run the following command against each of the java.dmp files:

    For WebSphere Application Server V6.1:

    jextract [DUMP_PATH] javadump.sdff

    For WebSphere Application Server Versions V5.0, V5.1, and V6.0:

    jextract -o javadump.sdff [DUMP_PATH]

    Where:

    jextract - is found in the install_root/java/jre/bin directory.
    javadump.sdff - is the output file.
    DUMP_PATH - is the path and file name for the java.dmp.

  5. Gather netstat data. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

    netstat -an > netstat_after.out

  6. Collect the following data:
    • For WebSphere Application Server V6.0:
      • The server.xml file located in the following directory:

        profile_root\config\cells\cell_name\nodes\
        node_name
        \servers\server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        profile_root\logs\server_name

      • Everything in the following directory:

        profile_root\logs\ffdc

    • For WebSphere Application Server V5.1 and 5.0:
      • The server.xml file located in the following directory:

        install_root\config\cells\nodes\node_name\
        servers\
        server_name

      • plugin-cfg.xml and http_plugin.log

      • Everything in the following directory:

        install_root\logs\server_name

      • Everything in the following directory:

        install_root\logs\ffdc

    • For all releases:
      • All java.dmp files

      • All javadump.sdff files

      • netstat*.out files

      • Include the Application Server systemErr.log, systemOut.log, native_stderr.log, and native_stdout.log, if they are located in a different directory

      • Web server access and error logs

      • If the Web server is remote, send the appropriate file from the Web server system including Web server configuration files and Web server logs. For example, for IBM HTTP Server and Apache Web server, provide the httpd.conf, access and error logs. For the SunOne 6 Web server, provide magnus.conf, obj.conf, access and error logs.

  7. Follow instructions to send diagnostic information to IBM support.

Alternative method for using drwtsn32 to force a user.dmp
This is not the preferred method, since the Java process will be killed, but can be used in situations where no other option works.

Drwtsn32 can be used to force a user.dmp file. This should already be loaded on the system. This will kill the JVM and it will need to be restarted.
  1. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

    netstat -an > netstat_before.out

  2. Determine the pid of the WebSphere Application Server process (review step 3a. from the preceding section, Generating the user.dmp files during a hang situation).

  3. Issue the following command from a Windows command prompt (where pid is the pid of the JVM that will not produce a javacore):

    drwtsn32 -p pid

    A user.dmp file should be created in the following directory, or whatever location you previously set (See step 4 from the section, Steps to setup a Microsoft Windows environment to generate Dr. Watson logs and Userdump files):

    operating_system_root\Documents and Settings\
    All Users\Application Data\Microsoft\Dr Watson

  4. Send in the generated user.dmp file and all applicable files and logs from step 5 in the preceding section.

  5. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to a hang or performance degradation, search the WebSphere Application Server support site.
 
Related information
Submitting information to IBM support
Steps to get support
MustGather: Readme first
Troubleshooting guide
MustGather: Crash on Windows
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation Windows 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): Windows
Software version: 6.0
Software edition:
Reference #: 1138203
IBM Group: Software Group
Modified date: Feb 18, 2007