MustGather: Getting user.dmp when hangs/performance degradation prevents generating a javacore

Technote (FAQ)
Problem
MustGather explains how to force a user.dmp for problems with the WebSphere® Application Server administrators should follow these directions when experiencing a hang or degradation in performance and javacores are not being produced through DrAdmin or wsadmin on the Microsoft® Windows® operating system. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
Solution
If you have already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Steps to setup Microsoft Windows environment to generate DrWatson log and Userdump files:
  1. On the Start menu, click Run.

  2. Enter drwtsn32, and then click OK.

    This will display the DrWatson screen.

  3. Type the location to where you want the Dr Watson log to be stored in the field provided for Log File Path.

  4. Type the location and the file name where you want the user.dmp file to be stored in the field provided for Crash Dump.

  5. Make sure that the following options are checked in DrWatson screen:
    • Dump Symbol Table
    • Dump All Thread Contexts
    • Visual Notification
    • Create Crash Dump file
    • For Windows 2003, "Crash Dump Type" option needs to be set to Windows NT 4 compatible Full
  6. Click OK.

  7. Click Run on the Start menu.

  8. Enter drwtsn32 -iand click OK. This will enable DrWatson to become the default debugger, and it will generate the DrWatson log and Userdump when an application crashes.

  9. The following steps to generate a series of user.dmp files, without interrupting execution of the java process.

    Note: An alternative method is provided below for use on systems where these steps do not appear to work. However, the alternative method kills the java process.

    Userdump.exe tool can be used to generate a user.dmp of a process that stops responding (hangs). In most cases, this tool will not kill the Java™ Virtual Machine (JVM™) and will allow for multiple user.dmp to be taken during the time of the problem.
    1. Download the Userdump Tool.

    2. Unzip the Oem3sr2.zip file to some directory, for example: C:\Debuggers.

    3. Gather netstat data. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

      netstat -an > netstat_before.out
    4. Create the first of three Userdump files for the hanging process:
      1. Run the Setup.exe program that is included with the Userdump tool from the C:\Debuggers\Userdump folder.

        When the java process hangs, from a command prompt type the below command: Userdump pid

        where pid is the process ID of the hanging java.exe. To obtain the pid of java.exe, open Task Manager, and then click the Process tab.
      2. This produces java.dmp file in the directory from where the userdump command was executed.
      3. Rename the java.dmp file to java1.dmp.
      4. Wait 2 minutes
    5. Create the second of three Userdump files for a hanging processes, renaming java.dmp to java2.dmp.

    6. Create the third of three Userdump files for the hanging processes, renaming java.dmp to java3.dmp.

    7. If the Java 2 SDK is V1.3.1 SR5 (131-20030618) or above, execute the following command from the Install_Root/java/jre/bin directory (after having copied each java.dmp file). Please execute this command on each java.dmp file created in the previous steps:
      jextract -o user_dmp.sdff [fully qualified path to java.dmp]
  10. Gather netstat data. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

    netstat -an > netstat_after.out
  11. Collect the following data:
    • For WebSphere Application Server V6.0 release:
      • The server.xml file located in the install_root\profiles\profile_name\config\cells\cell_name\nodes\node_name\servers\server_name
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root\profiles\profile_name>\logs\server_name directory
      • Everything in the install_root\profiles\profile_name\logs\ffdc directory
    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The server.xml file located in the install_root\config\cells\nodes\node_name\servers\server_name directory
      • plugin-cfg.xml and http_plugin.log
      • Everything in the install_root\logs\server_name directory
      • Everything in the install_root\logs\ffdc directory
    • For WebSphere Application Server V4.0 release:
      • A XMLConfig full export.
      • plugin-cfg.xml and native.log

      • Everything in the install_root\logs directory from the WebSphere Application Server system.
    • For WebSphere Application Server V3.5 release:
      • A XMLConfig full export.
      • queues, rules, vhosts property files and trace.log.webserver_name.date
      • Everything in the install_root\logs directory from the WebSphere Application Server system.

    • For all releases:
      • all java.dmp files
      • all java_dmp.sdff files
      • netstat*.out files
      • Include the Application Server systemErr.log, systemOut.log, native_stderr.log, and native_stdout.log, if they are located in a different directory
      • Web server's access and error logs
      • If the Web server is remote, send the appropriate file from the Web server system including Web server configuration files and Web server logs. For example for IBM HTTP Server and Apache Web server provide httpd.conf, access and error logs, for SunOne 6 webserver provide magnus.conf, obj.conf, access and error logs.


  12. Follow instructions to send diagnostic information to IBM support.
Alternative method for using drwtsn32 to force a user.dmp
This is not the preferred method, since the java process will be killed

Drwtsn32 can be used to force a user.dmp. This should already be loaded on the system. This will kill the JVM and it will need to be restarted.
  1. Enter the following from a command prompt on the WebSphere Application Server machine and any remote Web server machines:

    netstat -an > netstat_before.out
  2. Determine the pid of the JVM

  3. Issue the following command from a Windows command prompt (where pid is the pid of the JVM that will not produce a javacore):

    drwtsn32 -p pid

    A user.dmp file should be created in the operating_system_root\Documents and Settings\All Users\Documents\DrWatson directory.

  4. Send in the generated user.dmp file and all applicable files and logs from step 11 in above.
  5. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the hang or performance degradation component, search the WebSphere Application Server support site.

Related information
Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Hangs/Performance Degradation
Operating system(s): Windows
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server
Reference #: 1138203
IBM Group: Software Group
Modified date: 2004-12-13