MustGather information for startup problems

IBM HTTP Server customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes

  • operating system, IHS, and WebSphere plug-in configuration information
  • any messages written to the terminal during startup
  • any messages written to the IHS error log or WebSphere plug-in log during startup
  • results of trying to start IHS with/without the WebSphere plug-in
  • results of trying to start IHS with/without SSL

    Some of this information can be gathered automatically. Other information is gathered manually.

    Step 1: Check to see if you are encountering a common problem with known solution

    IBM HTTP Server 2.0 and above on all Linux and Unix platforms

    If you are attempting to run IBM HTTP Server using the httpd command:

    This is not supported. The apachectl command must be used instead. The httpd command can never be used directly.

    Here are some example console messages that you can see on different platforms when the apachectl command is not used:

    ld.so.1: /opt/IBMIHS20420/bin/httpd: fatal: libaprutil.so.0: open failed: No such file or directory
    
    
    /usr/lib/hpux64/dld.so: Unable to find library 'libaprutil-0.sl.9'.
    
    
    Could not load program
    /opt/IBMIHS/bin/httpd:
            Dependent module libaprutil-0.so could not be loaded.
    Could not load module libaprutil-0.so.
    Error was: No such file or directory
    
    
    # /opt/IBMIHS/bin/httpd -f /opt/IBMIHS/conf/httpd.conf
    Syntax error on line 161 of /opt/IBMIHS/conf/httpd.conf:                
    Cannot load /opt/IBMIHS/modules/mod_cgid.so into server:                
    /opt/IBMIHS/modules/mod_cgid.so: undefined symbol: ihs_os_pipe_put_ex
    

    IHS 1.3 can be started by running httpd directly, but IHS 2.0 must be started by running apachectl. This holds true even for querying the server version. Just do apachectl -v instead of httpd -v.

    IBM HTTP Server 2.0 and above on AIX

    If the console shows an error message such as the following:
    /usr/IBMHttpServer2/bin#./apachectl start            
    [Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for ::                  
    Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf:  
    Listen setup failed
    

    The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".

    This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as

    Listen 80
    
    to
    Listen 0.0.0.0:80
    

    AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.

    IBM HTTP Server on AIX

    If the console shows an error message such as the following:
    Syntax error on line 41 of /usr/HTTPServer/conf/httpd.conf: 
    Cannot load /usr/HTTPServer/libexec/mod_env.so into server: Bad address
    

    "Bad address" is the key part of the message. The module which failed to load could be different.

    This is a known problem on some levels of AIX 5.2 and AIX 5.3, and is corrected by an AIX APAR:

    IBM HTTP Server 1.3.28.x on SuSE SLES 9 or RedHat 3

    If the error log shows the following messages repeatedly:
    /opt/IBMHttpServer/bin/httpd: relocation error:
    /opt/IBMHttpServer/bin/httpd: symbol __pthread_atfork, version GLIBC_2.0 not defined in file libpthread.so.0 with link time reference
    
    (The path to httpd may be different.)

    This is an issue seen with IHS 1.3.28.x on RedHat 3 Update 4/x86 and SLES 9/x86 caused by the vendor dropping some programming interfaces which were available in prior releases. IHS will continually try to fork new child processes but each fork will fail with these messages. As of 20050315 there is no permanent solution. A work-around is to avoid the use of LDAP in /etc/nsswitch.conf.

    The issue with RedHat Linux is being tracked by RedHat support with bug id 67074. The customer can check with RedHat on the status of that report.

    IBM HTTP Server cumulative fix PK16139 (or later) contains a work-around for this problem.

    IBM HTTP Server 1.3.x on Linux for zSeries or pSeries

    If the console shows an error message such as the following:
    lxp-webprod2:/opt/IBMHTTPServer/bin # ./adminctl start      
    /opt/IBMHTTPServer/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
    ./adminctl start: admin http could not be started           
    

    (Messages from apachectl start are similar.)

    The problem is caused by a missing expat library. This library is generally provided by the Linux vendor on the Linux installation media. Consult your Linux vendor to determine what RPM must be installed to provide libexpat.so.0.

    An expat library in directory /usr/lib64/ is not sufficient, since that is a 64-bit library but IBM HTTP Server needs a 31-bit (zSeries) or 32-bit (pSeries) version of the library.

    Note: The expat dependency has been removed for IHS 1.3.28 for Linux/pSeries and Linux/zSeries with cumulative fix PK05084.

    IBM HTTP Server 2.0.x on Linux

    If this message is displayed on the console when trying to start the web server:

    Cannot load /opt/IHS20422/modules/mod_ibm_ssl.so into server: libstdc++.so.2.9: cannot open shared object file: No such file or directory
    

    This is due to a missing C++ compatibility RPM. A page that describes which RPMs must be installed to run WAS 5.1 products on RHEL 3 is at http://www-1.ibm.com/support/docview.wss?uid=swg21164634. The same issue can occur on other Linux distributions.

    IBM HTTP Server 2.0.47.x or 6.0.x on RHAS 4/PPC

    If one of these messages is displayed on the console when trying to start the web server:

    Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server:             
    /opt/IBMIHS/modules/mod_ibm_ssl.so: R_PPC_REL24 relocation at 0x0f6f6d74 for symbol `memset' out of range
    
    or
    Cannot load /opt/IBMIHS/modules/mod_ibm_ldap.so into server:
    /opt/IBMIHS/modules/mod_ibm_ldap.so: R_PPC_REL24 relocation at 0x0e9e0470 for symbol `__assert_fail' out of range
    

    The APAR for this problem is PK10954. IBM HTTP Server L3 support will have to provide a temporary fix for this, as the fix is not yet in the current maintenance levels.

    IBM HTTP Server 2.0 or above on Solaris

    If one of these messages is displayed on the console when trying to start the web server:

    Syntax error on line 847 of /opt/IBMIHS/conf/httpd.conf:
    Cannot load /opt/IBM/WebSphere/Plugins/bin/mod_was_ap20_http.so into server: ld.so.1: httpd: fatal: libgsk7cms.so: open failed: No such file or directory
    
    or
    Syntax error on line 847 of /opt/IBMIHS/conf/httpd.conf:
    Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server: ld.so.1: httpd: fatal: libgsk7cms.so: open failed: No such file or directory
    

    (The missing file could be libgsk5cms.so as well.)

    There is a problem with the GSKit installation. Make sure that GSKit has been installed locally instead of copied from a different system. Provide the output of the following commands:

    ls -l /usr/lib
    find /opt/ibm/gsk5
    find /opt/ibm/gsk7
    

    HP-UX/ia64

    If the console shows an error message such as the following:
    ./apachectl start
    [Mon Mar 07 09:51:00 2005] [crit] (223)Operation not supported: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY)
    no listening sockets available, shutting down
    Unable to open logs
    

    This is caused by a missing or misapplied operating system prerequisite.

    IBM HTTP Server 6.0.1 on HP-UX/ia64 requires the September 2004 update of 11i v2. This bundle appears in swlist as

    BUNDLE11i    B.11.23.0409.3 Required Patch Bundle for HP-UX 11i
    

    In some cases, that bundle can be present but the kernel updates have not been applied. Follow the HP-UX installation instructions in the release notes ( http://docs.hp.com/en/5990-8155/5990-8155.pdf ) and verify that there are no error messages in the installation log. There are different installation requirements based on the level of HP-UX prior to the update.

    If IBM HTTP Server continues to fail at startup with the same message, the HP-UX kernel updates have not been applied correctly. Contact HP support to resolve that issue.

    The critical portions of the startup failure message are "(223)" and "apr_socket_opt_set: (IPV6_V6ONLY)". If a different message is received, the startup failure is expected to have a different cause.

    This is documented in the release notes for 6.0.1.

    If the error log shows messages such as the following:
    /usr/lib/hpux64/dld.so: Unable to find library 'libgsk7cms_64.so'.
    [Mon Mar 07 08:58:16 2005] [notice] SSL0166E: Failure attempting to
    load GSK library.
    Configuration Failed
    

    This is caused by the directory

    /usr/lib
    not being present in the bin/envvars file within the IBM HTTP Server install directory.

    Edit the bin/envvars file within the IHS installation directory and add /usr/lib to the SHLIB_PATH setting. Here is an example value, based on a chosen install location of /opt/IBMIHS:

    SHLIB_PATH="/opt/IBMIHS/lib:/usr/lib:$SHLIB_PATH"
    

    This is documented in the release notes for 6.0.1.

    Step 2: Determine a minimal configuration which has the problem

    There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:

    1. third-party modules
    2. WebSphere plug-in
    3. mod_ibm_ssl
    4. mod_ibm_ldap

    Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.

    To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.

    If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.

    Step 3: SSL-related startup problems on certain systems

    This type of problem only occurs when mod_ibm_ssl is loaded or an https transport is defined in the WebSphere plug-in configuration file.

    3.1 Linux

    Try setting one of these environment variables prior to running apachectl start to see if the problem is resolved:

    Platform Mechanism to switch to an alternate C++ run-time library (GSkit 7)
    RHEL AS2.1 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
    RHEL AS3.0 export LD_ASSUME_KERNEL=2.2.5
    export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
    Other Linux/x86 export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3
    Linux/390 export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3

    The problem symptom can show up like this:

    $ bin/apachectl -k start -f conf/ssl.conf
    bin/apachectl: line 87: 15476 Aborted           (core dumped) $HTTPD $ARGV
    

    If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.

    3.2 HP-UX/PA-RISC

    Apply these two HP-UX patches which resolve incompatibilies with the SSL library provided with IBM HTTP Server (GSKit), and confirm that the startup problem is resolved:

    Step 4: Gathering information about the system and the IHS installation

    Most of this is automated by ServerDoc.jar. Here is an example invocation which saves configuration information. In this example, the ihsdiag package has been unpacked into directory /opt/ihsdiag-1.3.5 and the IHS installation directory is /opt/IBMIHS.

    $ java -jar /opt/ihsdiag-1.3.5/ServerDoc.jar  DescribeConfig /opt/IBMIHS
    Reports, log files, and configuration files have been saved to
    directory
      ServerConfig.200502040927
    If you have additional log files or configuration files, copy them there
    before packing up the directory.
    Web server log and conf files other than the default will have to be copied manually.
    WebSphere plug-in conf and log files will have to be copied manually.
    
    Hint for packing up the directory:
      tar -cf ServerConfig.200502040927.tar ServerConfig.200502040927
      gzip ServerConfig.200502040927.tar
    

    As the instructions indicate, you need to ensure that the proper IHS and WebSphere plug-in configuration files have been copied to the new ServerConfig.xxx directory.

    Step 5: Disable mod_cgid until the problem is resolved

    This step applies only to IHS 2.0 or above on Unix and Linux systems.

    At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.

    To disable mod_cgid, comment out this line in the configuration file:

    LoadModule cgid_module modules/mod_cgid.so
    

    Also, if you're using the ScriptSock, ScriptLog, ScriptLogBuffer, or ScriptLogLength, make sure they aren't active when mod_cgid is disabled. Such directives can be automatically disabled when mod_cgid is disabled by using a IfModule container, as shown below:

    <IfModule mod_cgid.c>
    Scriptsock           logs/cgisock
    </IfModule>
    

    Step 6: Turn on detailed logging until the problem is resolved

    IBM HTTP Server

    Edit the configuration file (usually httpd.conf) and set the value of the LogLevel directive to debug. If SSL support is enabled, add the SSLTrace directive to the bottom of the configuration file.

    WebSphere plug-in

    Edit the configuration file (usually plugin-cfg.xml) and change LogLevel to Trace. For example:

    <Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>
    

    Step 7: Gathering traces, console messages, and coredumps

    Make sure coredumps are allowed:

    # ulimit -c unlimited
    

    If the problem is related to SSL, turn on the GSKit trace:

    # GSK_TRACE_FILE=/tmp/gsktrace
    # export GSK_TRACE_FILE
    # GSKTRACE_NOBUFFERING=YES
    # export GSKTRACE_NOBUFFERING
    

    Start IHS to create the failure:

    # /path/to/apachectl start
    

    (If necessary, substitute your normal way of starting IBM HTTP Server. Note that apachectl is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.)

    Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.

    Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called core* or search the system for a file called core* and with a current timestamp. On Solaris, run the coreadm command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run syscorepath -g to see if an alternate location has been specified for coredumps.

    If a coredump was generated, follow the instruction here to extract information from it.

    Step 8: Get syscall trace of startup

    Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.

    Platform Command
    AIX 5.1 and above truss -o /tmp/ihs_startup -f startup-command
    Linux strace -o /tmp/ihs_startup -f startup-command
    Solaris truss -o /tmp/ihs_startup -f startup-command
    HP-UX tusc -o /tmp/ihs_startup -f startup-command

    Before running this, unset the environment variables GSK_TRACE_FILE and GSKTRACE_NOBUFFERING. Replace startup-command with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file /tmp/ihs_startup.

    Step 9: Sending information to IBM

    The information to send to IBM includes:

    Step Information to send
    1
  • nothing (no data collected)
  • 2
  • your statement on which optional features, if any, were required to trigger the startup problem
  • 3
  • whether or not the specified /usr/lib/libstdc++.X file existed
  • whether or not this setting resolved the problem
  • 4
  • the ServerConfig.xxx directory
  • 5
  • nothing (no data collected)
  • 6
  • the IBM HTTP Server and WebSphere plug-in configuration files, after being modified to enable tracing
  • 7
  • the GSKit trace in /tmp/gsktrace (if the problem was related to SSL)
  • a cut-and-paste of all messages that appeared on the console after running apachectl start
  • IHS error log
  • WebSphere plug-in trace file (if the problem was related to WebSphere plug-in)
  • CrashDoc.xxx directory (if a coredump was generated)
  • 8
  • the syscall trace file /tmp/ihs_startup