MustGather information for web server hangs or high CPU conditions (Windows)

Known issues to check first


System preparation

If you have not previously setup the Windows error reporting tools to save error information then refer to 'MustGather setup of error reporting dump tools for Microsoft Windows' for information on performing this setup. This setup step must be performed prior to recreating the hang. Make sure you are aware of the directory into which the tool will save the log and dump data.

Remove any existing Windows log and crash files from its output directory (after backing up as appropriate).

Also, please ensure that 'ExtendedStatus On' is enabled in the web server configuration file.

Install ProcDump from Windows Sysinternals to request dumps from hung processes.


AFPA

AFPA must be disabled and IBM HTTP Server restarted in order to collect some of the information below. You must be aware of whether or not AFPA is enabled in the web server configuration file before gathering the hang information. We recommend that AFPA be disabled in advance if the hang is reproducible so that the best set of information is available for analysis. To ensure that AFPA is disabled verify that the LoadModule statement for mod_afpa_cache.so in the configuration file is either commented out or absent, i.e.:
 #LoadModule ibm_afpa_module modules/mod_afpa_cache.so
This module is no longer shipped with IHS starting with Version 9.0.


Obtaining information at the time of the hang or issue

These steps must be performed at the time of the hang

  1. Gather general system and web server information.

    This information is gathered by running the ihsdiag ServerDoc DescribeConfig tool as described by the instructions in the System and web server information tool documentation.
    This will result in a directory of information named 'ServerConfig.timestamp'.
    That directory should be zipped and sent to IBM using the provided instructions after completing the following steps for obtaining additional information.

  2. TCP connection information

    Run 'netstat -an > netstat.txt' at a command-prompt.
    Save the netstat.txt file into the 'ServerConfig.timestamp' directory.

  3. Network configuration

    Run 'ipconfig /all > ipconfig.txt' at a command-prompt.
    Save the ipconfig.txt file into the 'ServerConfig.timestamp' directory.

  4. Server status report

    ('ExtendedStatus On' must be enabled for the running web server.)

    From a web browser, request http://www.example.com/server-status/?showmodule from a web browser.
    (Replace www.example.com with your web server address). If the web server is completely hung, it may not respond. Otherwise, save the server status report as a file in the 'ServerConfig.timestamp' directory.

  5. Task Manager information
  6. Web server state

    You must skip this step if AFPA is enabled, which is why it is important to have disabled it as described earlier in this document.

    Note: On Win2000, this step will terminate the IBM HTTP Server child process when it completes. But this information is critical for determining the cause of the hang.

    Determine the hung process ID, usually the process with +/- ThreadsPerChild number of threads displayed in 'pslist' or task manager (CHILDPID).

    Choose one tool to capture a dump:

    • Preferred:

      Run 'procdump -ma CHILDPID' and note the location of the dump as printed in your terminal. Add the dump files to the ServerConfig output director.

      Replace CHILDPID with the IHS child process PID, as found in the previous step from the Task Manager display.

    • Alternate:

      Run 'DRWTSN32.EXE -p CHILDPID' to create Dr.Watson log and crash files representing the current state of the web server.

      Replace CHILDPID with the IHS child process PID, as found in the previous step from the Task Manager display.

      If a dialog box appears stating that an error log is being created, don't press Cancel. Instead, wait for the text of the button to change to OK. Then press OK.

      The Dr. Watson files can now be found in the output locations configured in the DRWTSN32.EXE program, as described above under System preparation. Example:

            C:\DrWatson_dumps>dir
             Volume in drive C is C_Drive
             Volume Serial Number is 98E7-6D3C
      
             Directory of C:\DrWatson_dumps
      
            08/05/2009  09:32a      <DIR>          .
            08/05/2009  09:32a      <DIR>
            04/05/2010  03:48p           1,118,951 drwtsn32.log
            04/05/2010  03:49p          12,969,444 user.dmp
                           2 File(s)     14,088,395 bytes
                           2 Dir(s)  18,162,450,432 bytes free
            

      The files drwtsn32.log and user.dmp are the files to save.

      Copy the files into the 'ServerConfig.timestamp' directory.

  7. Any additional error or access log files that might be available.

    The access.log and error.log files will be automatically gathered by the ServerDoc DescribeConfig tool used above, but if the configuration file has been changed to specify differently named log files then you should copy these log files from the IHS_install_root\logs\ directory to the 'ServerConfig.timestamp\files\logs' directory.

  8. WebSphere plug-in trace file

    The actual location is specified in plugin-cfg.xml and is generated by configuring LogLevel="Trace".
    Example: c:\WebSphere\AppServer\logs\http_plugin.log
    Copy the file into the 'ServerConfig.timestamp' directory.

  9. Event Viewer information

Recap of information to send to IBM support:

Create a zip file of the 'ServerConfig.timestamp' directory as described in the System and web server information tool documentation. Send this ServerConfig.timestamp.zip containing the following to IBM support for analysis:

Instructions to send diagnostic information to IBM support.