IBM HTTP Server customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes
Some of this information can be gathered automatically. Other information is gathered manually.
httpd
command:This is not supported. The apachectl
command must be
used instead. Here are some example console messages that you can
see:
ld.so.1: /opt/IBMIHS20420/bin/httpd: fatal: libaprutil.so.0: open failed: No such file or directory /usr/lib/hpux64/dld.so: Unable to find library 'libaprutil-0.sl.9'. Could not load program /opt/IBMIHS/bin/httpd: Dependent module libaprutil-0.so could not be loaded. Could not load module libaprutil-0.so. Error was: No such file or directory
IHS 1.3 can be started by running httpd directly,
but IHS 2.0 must be started by running apachectl. This holds true
even for querying the server version. Just do apachectl
-v
instead of httpd -v
.
/usr/IBMHttpServer2/bin#./apachectl start [Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for :: Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf: Listen setup failed
The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".
This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 in IHS. That is done by changing a Listen directive such as
Listen 80to
Listen 0.0.0.0:80
AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.
/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: symbol __pthread_atfork, version GLIBC_2.0 not defined in file libpthread.so.0 with link time reference(The path to httpd may be different.)
This is an issue seen with IHS 1.3.28.x on RedHat 3 Update 4/x86 and SLES 9/x86 caused by the vendor dropping some programming interfaces which were available in prior releases. IHS will continually try to fork new child processes but each fork will fail with these messages. As of 20050315 there is no permanent solution. A work-around is to avoid the use of LDAP in /etc/nsswitch.conf.
The issue with RedHat Linux is being tracked by RedHat support with bug id 67074. The customer can check with RedHat on the status of that report.
If this message is displayed on the console when trying to start the web server:
Cannot load /opt/IHS20422/modules/mod_ibm_ssl.so into server: libstdc++.so.2.9: cannot open shared object file: No such file or directory
This is due to a missing C++ compatibility RPM. A page that describes which RPMs must be installed to run WAS 5.1 products on RHEL 3 is at http://www-1.ibm.com/support/docview.wss?uid=swg21164634. The same issue can occur on other Linux distributions.
./apachectl start [Mon Mar 07 09:51:00 2005] [crit] (223)Operation not supported: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY) no listening sockets available, shutting down Unable to open logs
This is caused by a missing or misapplied operating system prerequisite.
IBM HTTP Server 6.0.1 on HP-UX/ia64 requires the September 2004 update of 11i v2. This bundle appears in swlist as
BUNDLE11i B.11.23.0409.3 Required Patch Bundle for HP-UX 11i
In some cases, that bundle can be present but the kernel updates have not been applied. Follow the HP-UX installation instructions in the release notes ( http://docs.hp.com/en/5990-8155/5990-8155.pdf ) and verify that there are no error messages in the installation log. There are different installation requirements based on the level of HP-UX prior to the update.
If IBM HTTP Server continues to fail at startup with the same message, the HP-UX kernel updates have not been applied correctly. Contact HP support to resolve that issue.
The critical portions of the startup failure message are "(223)" and "apr_socket_opt_set: (IPV6_V6ONLY)". If a different message is received, the startup failure is expected to have a different cause.
This is documented in the release notes for 6.0.1.
/usr/lib/hpux64/dld.so: Unable to find library 'libgsk7cms_64.so'. [Mon Mar 07 08:58:16 2005] [notice] SSL0166E: Failure attempting to load GSK library. Configuration Failed
This is caused by the directory
/usr/libnot being present in the bin/envvars file within the IBM HTTP Server install directory.
Edit the bin/envvars file within the IHS installation directory and add /usr/lib to the SHLIB_PATH setting. Here is an example value, based on a chosen install location of /opt/IBMIHS:
SHLIB_PATH="/opt/IBMIHS/lib:/usr/lib:$SHLIB_PATH"
This is documented in the release notes for 6.0.1.
There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:
Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.
To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.
If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.
If the problem only occurs when mod_ibm_ssl is loaded or an
https transport is defined in the WebSphere plug-in
configuration file, then try setting one of these environment
variables prior to running apachectl start
to see if the
problem is resolved:
Platform | Mechanism to switch to an alternate C++ run-time library |
Linux/x86 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3 |
Linux/390 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3 |
The problem symptom can show up like this:
$ bin/apachectl -k start -f conf/ssl.conf bin/apachectl: line 87: 15476 Aborted (core dumped) $HTTPD $ARGV
If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.
Most of this is automated by ServerDoc.jar. Here is an example
invocation which saves configuration information. In this example,
the ihsdiag package has been unpacked into directory
/opt/ihsdiag-1.3.5
and the IHS installation directory is
/opt/IBMIHS
.
$ java -jar /opt/ihsdiag-1.3.5/ServerDoc.jar DescribeConfig /opt/IBMIHS Reports, log files, and configuration files have been saved to directory ServerConfig.200502040927 If you have additional log files or configuration files, copy them there before packing up the directory. Web server log and conf files other than the default will have to be copied manually. WebSphere plug-in conf and log files will have to be copied manually. Hint for packing up the directory: tar -cf ServerConfig.200502040927.tar ServerConfig.200502040927 gzip ServerConfig.200502040927.tar
As the instructions indicate, you need to ensure that the proper IHS and WebSphere plug-in configuration files have been copied to the new ServerConfig.xxx directory.
This step applies only to IHS 2.0 or above on Unix and Linux systems.
At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.
To disable mod_cgid, comment out this line in the configuration file:
LoadModule cgid_module modules/mod_cgid.so
Also, if you're using the ScriptSock
,
ScriptLog
, ScriptLogBuffer
, or
ScriptLogLength
, make sure they aren't active when
mod_cgid is disabled. Such directives can be automatically disabled
when mod_cgid is disabled by using a IfModule
container, as shown below:
<IfModule mod_cgid.c> Scriptsock logs/cgisock </IfModule>
Edit the configuration file (usually httpd.conf
) and
set the value of the LogLevel
directive to
debug
. If SSL support is enabled, add the
SSLTrace
directive to the bottom of the configuration
file.
Edit the configuration file (usually plugin-cfg.xml
)
and change LogLevel
to Trace
. For
example:
<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>
Make sure coredumps are allowed:
# ulimit -c unlimited
If the problem is related to SSL, turn on the GSKit trace:
# GSK_TRACE_FILE=/tmp/gsktrace # export GSK_TRACE_FILE # GSKTRACE_NOBUFFERING=YES # export GSKTRACE_NOBUFFERING
Start IHS to create the failure:
# /path/to/apachectl start
(If necessary, substitute your normal way of starting IBM HTTP
Server. Note that apachectl
is the only supported
mechanism for starting IHS 2.0 and above on Unix and Linux
systems.)
Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.
Check for a coredump. The location can vary by system
configuration and platform. Check in the current directory for a file
called core*
or search the system for a file called
core*
and with a current timestamp. On Solaris, run the
coreadm
command to see if an alternate location or
filename pattern has been specified for coredumps. On AIX 5.2 and
above, run syscorepath -g
to see if an alternate location
has been specified for coredumps.
If a coredump was generated, follow the instruction here to extract information from it.
Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.
Platform | Command |
AIX 5.1 and above | truss -o /tmp/ihs_startup -f startup-command |
Linux | strace -o /tmp/ihs_startup -f startup-command |
Solaris | truss -o /tmp/ihs_startup -f startup-command |
HP-UX | tusc -o /tmp/ihs_startup -f startup-command |
Before running this, unset the environment variables
GSK_TRACE_FILE
and GSKTRACE_NOBUFFERING
.
Replace startup-command
with the command used to
start IHS. After the startup failure occurs, the syscall trace will
be in file /tmp/ihs_startup
.
The information to send to IBM includes:
Step | Information to send |
1 |
|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
/tmp/gsktrace (if the problem was related to SSL)
apachectl start
|
8 |
|