MustGather information for startup problems

IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes

Some of this information can be gathered automatically. Other information is gathered manually. The steps for the startup mustgather are listed below:

  1. Check prerequisites
  2. Check for common problems with known solutions
  3. Determine a minimal configuration
  4. Test OS-specific tuning
  5. Gather webserver information
  6. Disable mod_cgid (unix only)
  7. Enable tracing
  8. Collect traces
  9. Obtain system call trace of startup
  10. Sending information to support

Step 1: Ensure that you are using a supported operating system level

Missing operating system prerequisites frequently cause startup failures.

Consult this information to find the required operating system levels:

Step 2: Check to see if you are encountering a common problem with known solution

Step 2b: Check to see if you are encountering a problem with older environments

Step 3: Determine a minimal configuration which has the problem

There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:

  1. third-party modules
  2. WebSphere plug-in
  3. mod_ibm_ssl
  4. mod_ibm_ldap

Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.

To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.

If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.

OS-specific workarounds

Step 4.1 Linux

Try setting one of these environment variables prior to running apachectl start to see if the problem is resolved:

Platform Mechanism to switch to an alternate C++ run-time library (GSkit 7)
RHEL AS2.1 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
RHEL AS3.0 export LD_ASSUME_KERNEL=2.2.5
export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
Other Linux/x86 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
Linux/390 export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3

The problem symptom can show up like this:

$ bin/apachectl -k start -f conf/ssl.conf
bin/apachectl: line 87: 15476 Aborted           (core dumped) $HTTPD $ARGV

If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.

Step 4.2 HP-UX/PA-RISC

Apply these two HP-UX patches which resolve incompatibilies with the SSL library provided with IBM HTTP Server (GSKit), and confirm that the startup problem is resolved:

Step 5: Gathering information about the system and the IHS installation

Follow the instructions for the System and web server information tool to create a package to upload.

Step 6: Disable mod_cgid until the problem is resolved

This step applies only to IHS 2.0 or above on Unix and Linux systems.

At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.

To disable mod_cgid, comment out this line in the configuration file:

LoadModule cgid_module modules/mod_cgid.so

Also, if you're using the ScriptSock, ScriptLog, ScriptLogBuffer, or ScriptLogLength, make sure they aren't active when mod_cgid is disabled. Such directives can be automatically disabled when mod_cgid is disabled by using a IfModule container, as shown below:

<IfModule mod_cgid.c>
Scriptsock           logs/cgisock
</IfModule>

Step 7: Turn on detailed logging until the problem is resolved

IBM HTTP Server

Edit the configuration file (usually httpd.conf) and set the value of the LogLevel directive to debug. If SSL support is enabled, add the SSLTrace directive to the bottom of the configuration file.

WebSphere plug-in

Edit the configuration file (usually plugin-cfg.xml) and change LogLevel to Trace. For example:

<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>

Step 8: Gathering traces, console messages, and coredumps

Make sure coredumps are allowed:

# ulimit -c unlimited

If the problem is related to SSL, set the following GSKit trace environment variables via <ihsinst>/bin/envvars or the windows control panel:

GSK_TRACE_FILE_NUMBER=1
export GSK_TRACE_FILE_NUMBER

# This following setting should only be used for suspected crashes
GSKTRACE_NOBUFFERING=YES
export GSKTRACE_NOBUFFERING

# 250MB
GSK_TRACE_FILE_SIZE=262144000
export GSK_TRACE_FILE_SIZE

Start IHS to create the failure:

# /path/to/apachectl start

(If necessary, substitute your normal way of starting IBM HTTP Server. Note that apachectl is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.)

Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.

Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called core* or search the system for a file called core* and with a current timestamp. On Solaris, run the coreadm command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run syscorepath -g to see if an alternate location has been specified for coredumps.

If a coredump was generated, follow the instruction here to extract information from it.

Step 9: Get syscall trace of startup

Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.

Platform Command
AIX 5.2 and above truss -o /tmp/ihs_startup -d -f startup-command
AIX 5.1 truss -o /tmp/ihs_startup -f startup-command
Linux strace -o /tmp/ihs_startup -f startup-command
Linux on SUSE for 64-bit zSeries (s390x) strace32 -o /tmp/ihs_startup -f startup-command (RPM: strace-32bit)
Solaris truss -o /tmp/ihs_startup -f startup-command
HP-UX tusc -o /tmp/ihs_startup -f startup-command

Before running this, unset the environment variables GSK_TRACE_FILE and GSKTRACE_NOBUFFERING. Replace startup-command with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file /tmp/ihs_startup.

Step 10: Sending information to IBM

The information to send to IBM includes:

Step Information to send
1
  • nothing (no data collected)
2
  • your statement on which optional features, if any, were required to trigger the startup problem
3
  • whether or not the specified /usr/lib/libstdc++.X file existed
  • whether or not this setting resolved the problem
4
  • the ServerConfig.xxx directory
5
  • nothing (no data collected)
6
  • the IBM HTTP Server and WebSphere plug-in configuration files, after being modified to enable tracing
7
  • the GSKit trace in /tmp/gsktrace (if the problem was related to SSL)
  • a cut-and-paste of all messages that appeared on the console after running apachectl start
  • IHS error log
  • WebSphere plug-in trace file (if the problem was related to WebSphere plug-in)
  • CrashDoc.xxx directory (if a coredump was generated)
8
  • the syscall trace file /tmp/ihs_startup