IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes
Some of this information can be gathered automatically. Other information is gathered manually. The steps for the startup mustgather are listed below:
Missing operating system prerequisites frequently cause startup failures.
Consult this information to find the required operating system levels:
Common startup errors on in-service releases of IHS
Check the documentation for error log messages and see if any of them apply to the problem.
ap_cache_cacheable_hdrs_out
" when starting IHS.
Make sure the LoadModule
directive for mod_cache is un-commented and precedes
the LoadModule
directive for mod_mem_cache.
ap_proxyerror
" or
"proxy_hook_scheme_handler
".
Make sure the LoadModule directive for mod_proxy
is un-commented and precedes the
LoadModule directive for mod_proxy_http, mod_proxy_connect, and mod_proxy_ftp.
httpd
command, or there is a problem with the
IHSROOT/bin/envvars
file or a customized
apachectl
script: (IHS 2.0 and above on Linux and Unix platforms)The httpd
command cannot be used directly because
library paths must be set up beforehand so that support libraries are
found in the IHS lib
directory. The
apachectl
command, in conjunction with the
IHSROOT/bin/envvars
file, sets up the library
paths as necessary.
Here are some example console messages that you can
see on different platforms when the httpd
is used
directly or there is a problem with the
IHSROOT/bin/envvars
file or a customer-modified
apachectl
command does not work:
ld.so.1: /opt/IBMIHS20420/bin/httpd: fatal: libaprutil.so.0: open failed: No such file or directory /usr/lib/hpux64/dld.so: Unable to find library 'libaprutil-0.sl.9'. Could not load program /opt/IBMIHS/bin/httpd: Dependent module libaprutil-0.so could not be loaded. Could not load module libaprutil-0.so. Error was: No such file or directory # /opt/IBMIHS/bin/httpd -f /opt/IBMIHS/conf/httpd.conf Syntax error on line 161 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBMIHS/modules/mod_cgid.so into server: /opt/IBMIHS/modules/mod_cgid.so: undefined symbol: ihs_os_pipe_put_ex Syntax error on line 845 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server: /opt/IBMIHS/modules/mod_ibm_ssl.so: undefined symbol: apr_socket_iol_push
IHS 1.3 can be started by running httpd directly,
but IHS 2.0 must be started by running apachectl. This holds true
even for querying the server version. Just do apachectl
-v
instead of httpd -v
.
(98)Address already in use: make_sock: could not bind to address [::]:80
# apachectl start (98)Address already in use: make_sock: could not bind to address [::]:80 no listening sockets available, shutting down Unable to open logs
Listen
directives in httpd.conf.
netstat
or lsof
can be
used for this.
$ netstat -an | grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN # lsof -ni :80 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME httpd 25810 root 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25812 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25814 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25815 nobody 3u IPv4 53644245 TCP *:http (LISTEN)
In IHS configurations where mod_cgid is loaded, some types of
crashes during initialization can leave an httpd process running. In
these cases, the stranded httpd process will be running as the
non-root userid specified by the User
directive and
will have a parent pid of 1. This process should be removed with
kill -TERM.
Possible initialization problems which can leave an httpd process running include:
/usr/IBMHttpServer2/bin#./apachectl start [Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for :: Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf: Listen setup failed
The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".
This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as
Listen 80to
Listen 0.0.0.0:80
AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.
This may be caused by IPv6 resolver lookups. Try this possible solution, which is to skip IPv6 lookups for IHS.
NSORDER=local4,bind4 export NSORDER
After installing IBM HTTP Server 6.1 onto a Windows 64-bit OS (Windows 2003 or Windows 2008),
the web server may not start
when using the 'Start HTTP Server'
shortcut icon on the 'Start'
menu.
You may also see a message displayed that is similar to the following:
"The item 'apache.exe' that this shortcut refers to has been changed or moved, so this shortcut will no longer work properly."
When IBM HTTP Server 6.1 is installed to the default location under C:\Program Files
on a 64-bit Windows OS, the shortcut icons are created incorrectly.
When this occurs, the icons are created with values such as:
Target: "C:\Program Files(x86\IBM\HTTPServer,\)\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:Program Files(x86/IBM/HTTPServer,\)\bin"
There are several problems in these strings:
,\
after 'HTTPServer' should not be present
Start in:
' value should have a forward slash after the C:
These can be resolved by editing the values as follows:
Start->All Programs->IBM HTTP Server V6.1
' and right-click 'Start HTTP Server
'
Target:
' and 'Start in:
' values.
Target: "C:\Program Files (x86)\IBM\HTTPServer\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:/Program Files (x86)/IBM/HTTPServer/bin"If your actual paths are different, then adjust as needed. You simply need to edit them to be valid paths.
Start in:
path should work with either forward slashes or backslashes.)
'Stop HTTP Server'
shortcut.
You should also check the following items.
These may have similar path problems that need to be corrected. Edit them to be valid paths as needed.
(The base part of the values should look similar to the good values shown above.)
httpd.conf
file entries:
ServerRoot
'
DocumentRoot
'
Additional information can be found in the 'Restriction' section of the Installing Web server plug-ins InfoCenter page.
Since the IHS problem is in the install of the base 6.1 package, it cannot be resolved via a fixpack.
The solution is to perform the edits described above, or to install to a path that doesn't contain parentheses in the path string.
The problem does not occur with the newer IBM HTTP Server V7.
error while loading shared libraries: libexpat.so.0
(IHS 1.3.x on Linux)
Cannot load /opt/IHS20422/modules/mod_ibm_ssl.so into server: libstdc++.so.2.9:
(IHS 2.0.x on Linux)
R_PPC_REL24
fatal: libgsk7cms.so: open failed: No such file or directory
(IHS 2.0 or above on Solaris)
Console Error: file vhost.c, line 232, assertion "rv == APR_SUCCESS"
(IBM HTTP Server 6.0 or above on Solaris)
Console Error: Unresolved symbol: getaddrinfo (code)
(IHS on HP-UX)
Console Error: [crit] (223)Operation not supported: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY)
(IHS on HP-UX)
Unable to find library 'libgsk7cms_64.so'
(IHS on HP-UX)
Cannot load /usr/HTTPServer/libexec/mod_env.so into server: Bad address
(IHS on AIX)Syntax error on line 41 of /usr/HTTPServer/conf/httpd.conf: Cannot load /usr/HTTPServer/libexec/mod_env.so into server: Bad address
"Bad address" is the key part of the message. The module which failed to load could be different.
This is a known problem on some levels of AIX 5.2 and AIX 5.3, and is corrected by an AIX APAR:
/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: symbol __pthread_atfork, version GLIBC_2.0 not defined in file libpthread.so.0 with link time reference(The path to httpd may be different.)
This is an issue seen with IHS 1.3.28.x on RedHat 3 Update 4/x86 and SLES 9/x86 caused by the vendor dropping some programming interfaces which were available in prior releases. IHS will continually try to fork new child processes but each fork will fail with these messages. As of 20050315 there is no permanent solution. A work-around is to avoid the use of LDAP in /etc/nsswitch.conf.
The issue with RedHat Linux is being tracked by RedHat support with bug id 67074. The customer can check with RedHat on the status of that report.
IBM HTTP Server cumulative fix PK16139 or later contains a work-around for this problem.
error while loading shared libraries: libexpat.so.0
(IHS 1.3.x on Linux) lxp-webprod2:/opt/IBMHTTPServer/bin # ./adminctl start /opt/IBMHTTPServer/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory ./adminctl start: admin http could not be started
(Messages from apachectl start
are similar.)
The problem is caused by a missing expat library. This library is generally provided by the Linux vendor on the Linux installation media. Consult your Linux vendor to determine what RPM must be installed to provide libexpat.so.0.
An expat library in directory /usr/lib64/
is not
sufficient, since that is a 64-bit library but IBM HTTP Server needs a
31-bit (zSeries) or 32-bit (pSeries) version of the library.
Note: The expat dependency has been removed for IHS 1.3.28 for Linux/pSeries and Linux/zSeries with cumulative fix PK05084 or later.
Syntax error on line 223 of /opt/IHS/conf/httpd.conf: Cannot load /opt/IHS/libexec/mod_auth_dbm.so into server: /opt/IHS/libexec/mod_auth_dbm.so: undefined symbol: dbm_fetch
This can also occur with mod_rewrite.so.
The startup failure is caused by the absence of libgdbm.so.2 on the system.
For 64-bit SLES distributions, install the compat-32bit RPM to obtain this file.
For other Linux distributions, the Linux vendor will need to provide the name of the necessary RPM.
Cannot load /opt/IHS20422/modules/mod_ibm_ssl.so into server: libstdc++.so.2.9:
(IHS 2.0.x on Linux)If this message is displayed on the console when trying to start the web server:
Cannot load /opt/IHS20422/modules/mod_ibm_ssl.so into server: libstdc++.so.2.9: cannot open shared object file: No such file or directory
This is due to a missing C++ compatibility RPM. A page that describes which RPMs must be installed to run WAS 5.1 products on RHEL 3 is at http://www-1.ibm.com/support/docview.wss?uid=swg21164634. The same issue can occur on other Linux distributions.
R_PPC_REL24
If one of these messages is displayed on the console when trying to start the web server:
Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server: /opt/IBMIHS/modules/mod_ibm_ssl.so: R_PPC_REL24 relocation at 0x0f6f6d74 for symbol `memset' out of rangeor
Cannot load /opt/IBMIHS/modules/mod_ibm_ldap.so into server: /opt/IBMIHS/modules/mod_ibm_ldap.so: R_PPC_REL24 relocation at 0x0e9e0470 for symbol `__assert_fail' out of range
The APAR for this problem is PK10954.
fatal: libgsk7cms.so: open failed: No such file or directory
(IHS 2.0 or above on Solaris)If one of these messages is displayed on the console when trying to start the web server:
Syntax error on line 847 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBM/WebSphere/Plugins/bin/mod_was_ap20_http.so into server: ld.so.1: httpd: fatal: libgsk7cms.so: open failed: No such file or directoryor
Syntax error on line 847 of /opt/IBMIHS/conf/httpd.conf: Cannot load /opt/IBMIHS/modules/mod_ibm_ssl.so into server: ld.so.1: httpd: fatal: libgsk7cms.so: open failed: No such file or directory
(The missing file could be libgsk5cms.so as well.)
There is a problem with the GSKit installation. Make sure that GSKit has been installed locally instead of copied from a different system. Provide the output of the following commands:
ls -l /usr/lib find /opt/ibm/gsk5 find /opt/ibm/gsk7
file vhost.c, line 232, assertion "rv == APR_SUCCESS"
(IBM HTTP Server 6.0 or above on Solaris)If this message, or similar, is displayed on the console when trying to start the web server:
[crit] [Thu Jan 25 15:01:15 2007] file vhost.c, line 232, assertion "rv == APR_SUCCESS" failed
Ensure that dns
is in the search order for
hosts
in /etc/nsswitch.conf
. Working
examples:
# simple hosts: files dns # more complex hosts: files dns nis [NOTFOUND=return] files
The failure can occur with certain VirtualHost
directives when dns
in not the search order. An example
is <VirtualHost _default_:portnumber>
.
Unresolved symbol: getaddrinfo (code)
(IHS on HP-UX)If this message, or similar, is displayed on the console when trying to start the web server:
$ ./apachectl start /usr/lib/dld.sl: Unresolved symbol: getaddrinfo (code) from /usr/IBMIHS/lib/libapr-0.sl.9 ./apachectl[73]: 17367 Abort(coredump)
This has been seen on HP-UX 11 (11.00). That release of HP-UX has not been supported in a number of releases, and IBM HTTP Server 6.0 won't start up on that level. Some older releases of IBM HTTP Server may start up, but they haven't been tested on this level of HP-UX.
[crit] (223)Operation not supported: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY)
(IHS on HP-UX)./apachectl start [Mon Mar 07 09:51:00 2005] [crit] (223)Operation not supported: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY) no listening sockets available, shutting down Unable to open logs
This is caused by a missing or misapplied operating system prerequisite.
IBM HTTP Server on HP-UX/ia64 requires the September 2004 update of 11i v2. This bundle appears in swlist as
BUNDLE11i B.11.23.0409.3 Required Patch Bundle for HP-UX 11i
In some cases, that bundle can be present but the kernel updates have not been applied. Follow the HP-UX installation instructions in the release notes ( http://docs.hp.com/en/5990-8155/5990-8155.pdf ) and verify that there are no error messages in the installation log. There are different installation requirements based on the level of HP-UX prior to the update.
If IBM HTTP Server continues to fail at startup with the same message, the HP-UX kernel updates have not been applied correctly. Contact HP support to resolve that issue.
The critical portions of the startup failure message are "(223)" and "apr_socket_opt_set: (IPV6_V6ONLY)". If a different message is received, the startup failure is expected to have a different cause.
This is documented in the release notes for 6.0.1.
Unable to find library 'libgsk7cms_64.so'
(IHS on HP-UX)/usr/lib/hpux64/dld.so: Unable to find library 'libgsk7cms_64.so'. [Mon Mar 07 08:58:16 2005] [notice] SSL0166E: Failure attempting to load GSK library. Configuration Failed
This is caused by the directory /usr/lib
not being present
in the IHSROOT/bin/envvars
within the IBM HTTP
Server install directory.
Edit the IHSROOT/bin/envvars
file within the
IHS installation directory and add /usr/lib to the SHLIB_PATH setting.
Here is an example value, based on a chosen install
location of /opt/IBMIHS:
SHLIB_PATH="/opt/IBMIHS/lib:/usr/lib:$SHLIB_PATH"
This is documented in the release notes for 6.0.1.
There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:
Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.
To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.
If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.
Try setting one of these environment
variables prior to running apachectl start
to see if the
problem is resolved:
Platform | Mechanism to switch to an alternate C++ run-time library (GSkit 7) |
RHEL AS2.1 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3 |
RHEL AS3.0 | export LD_ASSUME_KERNEL=2.2.5 |
Other Linux/x86 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.2-2.so.3 |
Linux/390 | export LD_PRELOAD=/usr/lib/libstdc++-libc6.1-2.so.3 |
The problem symptom can show up like this:
$ bin/apachectl -k start -f conf/ssl.conf bin/apachectl: line 87: 15476 Aborted (core dumped) $HTTPD $ARGV
If this resolves the problem, that is your permanent solution, and gathering further information is not necessary.
Apply these two HP-UX patches which resolve incompatibilies with the SSL library provided with IBM HTTP Server (GSKit), and confirm that the startup problem is resolved:
Follow the instructions for the System and web server information tool to create a package to upload.
This step applies only to IHS 2.0 or above on Unix and Linux systems.
At initialization, mod_cgid creates a separate child process to handle CGI requests. During some types of startup failures, this child process is left stranded when IHS crashes. mod_cgid should be disabled until the problem is resolved so that the mod_cgid child process does not have to be manually terminated, and so that it does not appear that IHS startup has hung since an httpd process (the mod_cgid child) will remain running.
To disable mod_cgid, comment out this line in the configuration file:
LoadModule cgid_module modules/mod_cgid.so
Also, if you're using the ScriptSock
,
ScriptLog
, ScriptLogBuffer
, or
ScriptLogLength
, make sure they aren't active when
mod_cgid is disabled. Such directives can be automatically disabled
when mod_cgid is disabled by using a IfModule
container, as shown below:
<IfModule mod_cgid.c> Scriptsock logs/cgisock </IfModule>
Edit the configuration file (usually httpd.conf
) and
set the value of the LogLevel
directive to
debug
. If SSL support is enabled, add the
SSLTrace
directive to the bottom of the configuration
file.
Edit the configuration file (usually plugin-cfg.xml
)
and change LogLevel
to Trace
. For
example:
<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>
Make sure coredumps are allowed:
# ulimit -c unlimited
If the problem is related to SSL, set the following GSKit trace environment variables via <ihsinst>/bin/envvars or the windows control panel:
GSK_TRACE_FILE_NUMBER=1 export GSK_TRACE_FILE_NUMBER # This following setting should only be used for suspected crashes GSKTRACE_NOBUFFERING=YES export GSKTRACE_NOBUFFERING # 250MB GSK_TRACE_FILE_SIZE=262144000 export GSK_TRACE_FILE_SIZE
Start IHS to create the failure:
# /path/to/apachectl start
(If necessary, substitute your normal way of starting IBM HTTP
Server. Note that apachectl
is the only supported
mechanism for starting IHS 2.0 and above on Unix and Linux
systems.)
Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.
Check for a coredump. The location can vary by system
configuration and platform. Check in the current directory for a file
called core*
or search the system for a file called
core*
and with a current timestamp. On Solaris, run the
coreadm
command to see if an alternate location or
filename pattern has been specified for coredumps. On AIX 5.2 and
above, run syscorepath -g
to see if an alternate location
has been specified for coredumps.
If a coredump was generated, follow the instruction here to extract information from it.
Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.
Platform | Command |
AIX 5.2 and above | truss -o /tmp/ihs_startup -d -f startup-command |
AIX 5.1 | truss -o /tmp/ihs_startup -f startup-command |
Linux | strace -o /tmp/ihs_startup -f startup-command |
Linux on SUSE for 64-bit zSeries (s390x) | strace32 -o /tmp/ihs_startup -f startup-command (RPM: strace-32bit) |
Solaris | truss -o /tmp/ihs_startup -f startup-command |
HP-UX | tusc -o /tmp/ihs_startup -f startup-command |
Before running this, unset the environment variables
GSK_TRACE_FILE
and GSKTRACE_NOBUFFERING
.
Replace startup-command
with the command used to
start IHS. After the startup failure occurs, the syscall trace will
be in file /tmp/ihs_startup
.
The information to send to IBM includes:
Step | Information to send |
1 |
|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
|
8 |
|