[emerg] (9)Bad file number: fcntl: F_SETLKW:
Error getting accept lock, exiting! Perhaps you need to use the
LockFile directive to place your lock file on a local disk!
The "Bad file number" message indicates that the IBM HTTP Server file mutex descriptor has been closed incorrectly. This has been encountered on Solaris with IBM HTTP Server 1.3 and a third-party module.
Make a change to httpd.conf to work around this problem; add
AcceptMutex sysvsem
to the bottom of httpd.conf. Also,
there are likely changes to the Solaris kernel semaphore tuning.
Refer to these notes.
[warn] (22)Invalid argument: setsockopt: (TCP_NODELAY)
[error] (22)Invalid argument: getsockname
[warn] (22)Invalid argument: apr_socket_opt_set: (TCP_NODELAY)
[info] (22)Invalid argument: apr_socket_addr_get(APR_LOCAL)
This message is logged when the client drops the connection right after IBM HTTP Server has started processing it but before IBM HTTP Server has started reading the request. It is to be expected that some subset of clients will drop the connection at unexpected times (this could be a real user at a web browsers pressing the stop button), and this is one of the types of messages that can be logged because of that.
Unfortunately, the error returned from the kernel (EINVAL) has a text description that makes the failure look like a programming error, but that is how these calls are rejected when the TCP connection has already been dropped by the client.
Some load balancers test availability of the server on a regular basis in such a way that IBM HTTP Server often encounters a failure on one of these calls. Current maintenance levels of IBM HTTP Server 1.3 have lowered the severity of these messages and included the client IP address in the log message.
(info) (9) bad file number: core_output_filter: writing
data to the network.
The significance with this message is the text "bad file number." If
it says something like "Broken pipe" or "Connection reset", it is a
different issue.
To correct some problems which lead to the "bad file number" message, the customer needs PQ85834 or later for 2.0.42.2, or 2.0.47.
[info] [client 1.2.3.4] (32)Broken pipe: client stopped
connection before rflush completed
[info] (32)Broken pipe: core_output_filter: writing data to the network
This message is logged when the client drops the connection while IBM HTTP Server is writing the response. It is to be expected that some subset of clients will drop the connection while IBM HTTP Server is writing to it. The user could have pressed the stop button or closed the browser or there could have been a temporary network problem.
Another normal condition where this may occur is when a .pdf file is requested from a browser with the Adobe plug-in installed. The browser first requests the entire file, then when it sees the type of response the plug-in takes over, cancels the initial request (though IBM HTTP Server may not have written the entire response yet), then re-issues the request for the .pdf file in the form of a range request.
A rare condition where these messages may occur is if there is some application layer issue (e.g., browser incompatibility with application response) which leads to the browser abruptly dropping the connection to IBM HTTP Server when it receives certain response data. In order to diagnose the problem, network traces or mod_net_trace are normally needed so that IBM HTTP Server support can see what was sent to the browser. For SSL connections, mod_net_trace is preferred since it traces the data in unencrypted form.
[debug] D:\Build\WWWIHS1328\apache\ibm\modules\afpa\mod_ibm_afpa.c(595): (2)No such file or directory: afpa_init: hafpa = 0, cfg->init_afpa = 1, cfg->bBaseServerConfig = 1, cfg->afpaEnabled = 0
This is a debug message and is not formatted for customer consumption. The key issue here is the value printed for "cfg->afpaEnabled." If the value is 0, then AFPA is disabled. Otherwise, AFPA is enabled.
[error] File does not exist: /opt/IBMHTTPServer/htdocs/filename
This message indicates the requested filename, or a directory along the way to the requested filename, does not exist. In the latter case, for IHS 2.x and later, the path displayed stops at the directory name that could not be found.
If a request is intended for the WebSphere Plugin, but isn't handled due to an incomplete plugin-cfg.xml, you might see this message ending in one of your defined application context roots.
/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: undefined symbol: apr_table_compress
This is a symptom of IBM HTTP Server finding the wrong version of the
libapr-0.so
at runtime. libapr-0.so
is
provided with the IBM HTTP Server installation in the lib/
subdirectory, and is found by a platform-specific environment variable
in IHSROOT/bin/envvars
.
IHSROOT/bin/envvars
includes the libs/
subdirectory if you have altered your installation.
apachectl
script, or are
using some other custom method to start IHS, make sure it's sourcing
the proper IHSROOT/bin/envvars
file
/opt/IBMHttpServer/bin/httpd: relocation error:
/opt/IBMHttpServer/bin/httpd: symbol __pthread_atfork, version GLIBC_2.0 not defined in file libpthread.so.0 with link time reference
This is an issue seen with IBM HTTP Server 1.3.28 on RedHat 3 Update 4/x86 and SLES 9/x86 caused by the vendor dropping some programming interfaces which were available in prior releases. IBM HTTP Server will continually try to fork new child processes but each fork will fail with these messages. A local circumvention is to avoid the use of LDAP in /etc/nsswitch.conf. Cumulative e-fix PK16139 and later have changed IBM HTTP Server to avoid the problem.
[notice] caught SIGTERM, shutting down
This message is a normal shutdown message for IBM HTTP Server. If a user or automated task (e.g., cron job) does "apachectl stop" or sends SIGTERM to the IBM HTTP Server parent process, this message will be written.
Note: The SIGTERM "signal" shouldn't be confused with other signals that indicate problems, such as SIGSEGV, SIGBUS, SIGABRT, and SIGILL. SIGTERM is the normal mechanism to tell a program to terminate on Unix, so apachectl sends SIGTERM to the IBM HTTP Server parent process to tell it to go away (and to clean up other IBM HTTP Server processes).
[warn] child process 26354 still did not exit, sending a SIGTERM
and/or [error] child process 26354 still did not exit, sending a SIGKILL
A child process is not exiting as quickly as expected during shutdown or non-graceful restart processing. During shutdown or non-graceful restart, the parent will tell the child processes to exit. If after a short amount of time a child has still not exited, the parent will send the SIGTERM signal to the child process again and write the message to the log. If after another interval the child process still does not exit, then the parent process forcefully brings down the child by sending the SIGKILL signal to the child process.
Stopping or non-gracefully restarting the web server while it is actively processing client requests will result in this type of message.
[error] could not make child process 11540 exit, attempting
to continue anyway
A child process hasn't exited even after sending the kill signal to the process. Web server termination has to continue even though the process hasn't exited, because there is no further action that can be taken automatically.
If the process exits within a few more seconds, the expected cause of the message is high system load at the time of termination, and there is no operational problem.
If the process does not exit on its own within one minute, the
expected cause is that a thread in that process is blocked in the
operating system kernel. Operating system support may need to
investigate if this is a recurring problem. IBM HTTP Server support
can investigate if hang
documentation is provided. When running the hang collector tool,
specify auto
for the parent process id and -
for the
non-SSL port.
[alert] No active workers found... Apache is
exiting!
Check for error messages earlier in the error log. One or more of those error messages should explain the problem.
[Wed Sep 10 15:20:03 2003] [crit] the listener thread didn't exit
This is sometimes written when a child process exits. It means that one of the threads in a child process didn't exit in the cleanest possible manner, but it doesn't indicate any operational problem. This is a debug message in current levels of IBM HTTP Server.
[Tue Sep 21 23:55:02 2004] [emerg] (22)Invalid argument:
apr_proc_mutex_unlock failed. Attempting to shutdown process
gracefully.
This message can be ignored if it occurs at the time of a web
server restart (apachectl graceful
or apachectl
restart
). It commonly occurs when the accept mutex type is
SysV sem and there are multiple listening sockets.
The log level of this message has been changed to
debug
if it appears during restart with the following
maintenance levels:
Release | Fix level |
---|---|
2.0.42.2 | PK07831 or later |
2.0.47.1 | PK07831 or later |
6.0 | 6.0.2.1 or later |
[warn] new file descriptor 1025 is too large; you probably
need to rebuild Apache with a larger FD_SETSIZE (currently 1024)
To resolve the problem:
[error] (24)Too many open files: file permissions deny server
access: /opt/IBMHttpServer/htdocs/en_US/index.html
[error] (24)Too many open files: apr_accept: (client socket)
[error] [client 1.2.3.4] (11)Resource temporarily unavailable: couldn't spawn child process: /path/to/cgi
This message occurs when the operating system cannot spawn
(fork()
) another child process for a CGI request. The
request will fail with a 500 error.
For more information, refer to fork() failures.
[error] (12)Not enough space: fork: Unable to fork new
process
This message occurs when the parent process tries to create a new
child process using the fork()
system call, but the
operating system returns an error. Clients may be delayed until an
existing child process finishes handling an existing connection.
For more information, refer to fork() failures.
[Sat Dec 13 11:52:48 2003] [warn] long lost child came home!
(pid 11380)
This can occur with piped loggers (e.g., rotatelogs) during a graceful restart. One of these messages may appear for every piped logger.
With levels of IBM HTTP Server 2.0 prior to interim fix PK01070, it can also occur during steady state for httpd processes which begin exiting due to MaxSpareThreads or MaxRequestsPerChild processing, but take a long time to exit.
[alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[alert] (12)Cannot allocate memory: apr_thread_create: unable
to create worker thread
This is a tuning issue. The operating system failed a request by the web server to create another thread. The operating system can be tuned to allow more threads, or the web server can be tuned to create fewer threads per process so that it doesn't encounter this issue.
Configration to check:
It may be too high for current operating system limits on memory or processes or threads. If the operating system limits cannot be changed, reduce the value of ThreadsPerChild and adjust other MPM directives accordingly.
IHSROOT/bin/envvars
file
This controls how much of the address space can be used for heap storage, and heap storage is the main requirement on AIX for being able to create a new thread.
The MAXDATA setting within LDR_CNTRL should be at least 0x60000000 in order to allow a large number of threads. Example:
LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...
For every thread created, Linux allocates the stack size as displayed by ulimit -s in the processes address space. If ThreadsPerChild * (ulimit -s) is greater than 2GB the address space will be exhausted (OS defaults may be as high as 10MB).
If a very large number of threads are required, reduce the stack size for the IBM HTTP Server environment. A stack size of 128K should be sufficient for IBM HTTP Server and the WebSphere plug-in. If third-party modules are used in the configuration, a larger stack may be required.
max_thread_proc
kernel setting
is at least five more than the setting of ThreadsPerChild.
max_thread_proc
is the maximum number of threads in a
process. A related kernel setting is nkthread
, the
maximum number of threads in the system.
max_thread_proc
and nkthread
can be
displayed or modified by the sam
program. With some
levels of HP-UX, changing the value requires a kernel rebuild and a
reboot of the system.
[notice] The LDR_CNTRL environment variable is not set;
httpd processes may fail under load due to memory limitations.
[notice] Verify that IBM HTTP Server was started with apachectl
and that the envvars file sets LDR_CNTRL.
This message is applicable only to IBM HTTP Server 2.0 and above on AIX.
By default, large amounts of heap memory will not be possible for
the httpd processes. The default
IHSROOT/bin/envvars
file installed with the
product uses the LDR_CNTRL environment variable to allow a generous
amount of heap memory to be allocated. If the setting is removed or
commented-out, configurations with large numbers of threads or other
uses of large amounts of heap memory, such as in-process caches, will
not work.
Customers may need to change the LDR_CNTRL value for their environment. In some cases it can be removed altogether, with no impact as long as the current configuration is mantained. If the configuration is modified in the future such that larger amounts of heap memory are required, the LDR_CNTRL setting may need to be restored or modified.
[crit] (17)File exists: unable to create scoreboard
"/opt/IBMIHS/logs/xxxx" (name-based shared memory
failure)
The xxxxx portion of the message varies based on the value of the ScoreBoardFile directive.
Either there are multiple web server instances configured with the same ScoreBoardFile directive or there is a single instance configured with ScoreBoardFile and it did not shutdown cleanly.
The recommended solution in either case is to remove the ScoreBoardFile directive and restart.
The only situation where ScoreBoardFile is needed is if a third-party application which reads the web server shared memory is in use and that application specifies that the ScoreBoardFile directive is required. That is very rare.
[error] server reached MaxClients
setting, consider raising the MaxClients setting
At the time the message was written, there were not any worker threads, or child processes in V1.3, available to process additional requests. I.e., all child processes/threads are already in use handling existing requests. The message is only written once during the life of the server, but the peak condition may happen any number of times after the message is logged.
mod_mpmstats for IBM HTTP Server V2.0 and above provides better reporting of this condition; in particular, it can report when the MaxClients condition has been reached as often as once every 90 seconds.
The web server does not take any action when the MaxClients condition is reached. In many cases the condition will be corrected when a temporary application slow-down is resolved and threads, or child processes in V1.3, can finish their current requests and become free to handle new connections. If there is no suspected application slowdown, the customer may need to increase MaxClients to increase the peak capacity of the web server. This type of tuning is described in the IBM HTTP Server Performance Tuning document. As described in that information, the customer can get a good idea of how many of the IBM HTTP Server child processes/threads are in use at a given time by activating mod_status and checking the server-status page, or by monitoring with mod_mpmstats.
This is an important message to watch out for when there is a symptom such as a hung server. In some types of hang conditions (e.g., all IBM HTTP Server child process/threads tied up waiting for the application server to respond, and the application server isn't responding because the application processing the request is hung), this "MaxClients" message will be written to the error log.
[info] server seems busy, (you may need to
increase StartServers, or Min/MaxSpareServers), spawning 8 children, there
are 4 idle, and 24 total children
[info] server seems busy, (you may need to increase
StartServers, ThreadsPerChild or Min/MaxSpareThreads), spawning 8
children, there are around 20 idle threads, and 5 total children
This message appears when a lot of child processes have to be created at about the same time to handle increased load. If it occurs very rarely, it may be triggered by a an infrequent but drastic increase in load. If it occurs relatively frequently, the process management tuning needs to be changed, as follows:
[crit] (67)Address already in use: make_sock: could not bind
to address nnn.nnn.nnn.nnn port 80
This can occur at startup or restart if some other process is listening to the specified port and optional address.
[emerg] (69)Network is down: apr_accept: giving up.
This can be reported on some platforms if the network layer or a specific interface is no longer active. IBM HTTP Server will exit if all child processes receive this same error.
This problem has been encountered on AIX when performing a load balancer takeover. An AIX APAR fix is available to disable reporting the action to applications via the ENETDOWN error:
Refer to the text of the appropriate AIX APAR for a temporary circumvention.
Ouch! malloc failed in malloc_block()
This occurs if a storage allocation fails. The child process then exits.
The storage allocation can fail due to reaching an OS limit for memory, or because a plug-in module is asking for an inappropriate amount of storage.
In later IBM HTTP Server 1.3 (PQ87084 for 1.3.19.6 or 1.3.26.2, or IBM HTTP Server 1.3.28), the number of bytes for the failing allocation is traced to make it easier to decide the condition.
This message is issued when IHS has mapped an incoming request to the filesystem, but IHS has been configured to not allow files from the specified path to be served to clients.
By default, IHS ships with a configuration that doesn't allow
files outside of the default document root to be
served. When new directories are added into the URL-space via
DocumentRoot
(possibly in a new VirtualHost) or Alias
,
a corresponding <Directory>
container should be
added to the configuation to establish the proper access control.
An example <Directory> container can be viewed in the httpd.conf.default prefixed by the following text:
# This should be changed to whatever you set DocumentRoot to.
Below is an example <Directory> container which allows anyone to access the newly added files:
<Directory /usr/local/app1> Options Indexes FollowSymLinks Order allow,deny Allow from all </Directory>
[error] [client 127.0.0.1] request failed: error reading the
headers
This message can be triggered by one of the following protocol errors encountered while reading a request from the client:
The exact cause is written in the error message sent to the client,
but it is not logged anywhere unless the "error-notes" note for the
request is logged via mod_log_config. (This would be rather
voluminous and is not recommended except for brief testing periods.)
The note can be logged by adding %{error-notes}n
to your
log format string.
Even without logging the error-notes note, you can consult the access log to see the URL of the request which failed and see what error message was returned to the browser. Look for requests which failed with 400 with the same timestamp as the messages in the error log.
[error] SSL0102S: GSK could not initialize, Could not open keyfile.
The InfoCenter lists this message as SSL0102E
instead of SSL0102S
.
Ensure that the KeyFile
directive points to a file
that can be read by the user starting IHS, including read and execute
permission for all directories between the root directory and the
directory containing the KDB file.
If the key file is stored on a non-native or non-local filesystem, problems with file locking in those filesystems can result in this error. Try copying the key file to a local filesystem using the native filesystem type, modifying the Keyfile directive to point to the new location, and attempting to start IBM HTTP Server again. If it works with the new location, consult OS or filesystem support to diagnose the failure accessing the file from the original location.
A customer encountered this on Linux when attempting to use a
key file on an NFS volume and the rpc.statd
daemon
was not running, resulting in errors when attempting to lock the
file.
[error] SSL0104S: GSK could not initialize,
Invalid password for keyfile.
The InfoCenter lists this message as SSL0104E
instead of SSL0104S
.
For each KDB file IHS is configured to use, IHS will also try to open a file with the same file extension changed to .sth. If IHS cannot find, open, or read this file the above error will be reported.
Solution: If the stash file does not exist, it can be created in Ikeyman
Be sure the userid and group that IHS is configured to run under have read and execute access to every intermediate directory and to the .sth file itself.
[error] SSL0108E: Initialization error, GSK internal
error.
If using 2.0.42 on Linux/PPC: This can be caused by GSkit getting installed without the OpenSSL functions. The root cause shows up in an strace of server startup like this:
open("/usr/local/ibm/gsk6//icc/icclib/libicclib.so", O_RDONLY) = -1 ENOENT (No such file or directory)
To resolve: In the directory where IBM HTTP Server was unpacked there should be files iccrte.tar and osslrte.tar, as well as gskit.sh and the GSkit RPM. Run this to re-install GSkit with the OpenSSL function:
# rpm --erase gsk6bas # export ICC_PKG_DIR=<any_non_empty_string> # ./gskit.sh 1
This bypasses a license dialog. If you install IBM HTTP Server normally via the GUI, you'll have a chance to see the license and accept it.
The message can appear on other platforms when there is a problem with the GSKit installation or there is a system library compatibility issue.
[error] SSL0209E: SSL Handshake Failed,
ERROR processing cryptography.
The message can be ignored. The crypto accelerator operation was aborted during shutdown.
The message can be ignored. The expected situation is
[warn] SSL0222W: SSL Handshake Failed, No ciphers specified.
SSL0222W occurs when the client and server don't share any SSL ciphers, which may be a result of overly-restrictive
values for SSLProtocolDisable
or SSLCipherSpec
[error] SSL0223E: SSL Handshake Failed, No certificate.
Verify that at least one personal certificate exists in the KDB
file specified by the KeyFile
directive. If no certificate
is marked as default (an asterisk appears next to its label), your configuration
must include an SSLServerCert
directive for each virtual host
containing an SSLEnable
directive.
Note: Some levels of IHS can report this error instead of SSL0222W.
[warn] SSL0226I: SSL Handshake Failed, I/O error during
handshake.
The communication between the client and the server failed. This is a common error when the client closes the connection before the handshake has completed.
If a load balancer is configured to try to establish a TCP connection to the SSL port at intervals, the SSL0226I message can occur repeatedly, at the configured interval.
[warn] SSL0265W: Client did not supply a certificate.
This message is received when SSLClientAuth required is specified, but IBM HTTP Server did not receive a client certificate during the SSL handshake. Additionally a 403 Forbidden error will appear at the browser. If the browser has a certificate installed, verify that the certificate authority (CA) which created the client certificate has a signer certificate installed in IBM HTTP Server's key database (.kdb) file. If not, use iKeyman to import the signer certificate information from the client certificate.
[error] SSL0404E: I/O failed, insufficient storage.
IHS 2.0.47.1 requires cumulative fix PK07831 or later to be compatible with GSKit versions 7.0.3.12 and higher.
[crit] SSL0600S: Unable to connect to session ID
cache
[crit] (146)Connection refused: SSL0600S: Unable to connect to
session ID cache
[crit] (13)Permission denied: SSL0600S: Unable to connect to session
ID cache
piped log program '(null)' failed unexpectedly
This message can appear with IBM HTTP Server 2.0 and above at shutdown or restart time when piped log programs such as rotatelogs are configured. It does not result in an operational problem.
[error] [client nnn.nnn.nnn.nnn] (13)Permission denied:
access to some url failed because search permissions are missing on a
component of the path
To find the exact permission problem, first determine which file
should be served for the specified url. As an example, if the url is
/
, this would usually result in serving file index.html
in the IBM HTTP Server DocumentRoot
directory. With the following
configuration in httpd.conf
DocumentRoot /opt/local/HTTPServer/htdocs/en_US ... User nobody Group nobody
the actual file to be served would be
/opt/local/HTTPServer/htdocs/en_US/index.html
, and the
user/group nobody
/nobody
must be able to
read and search each component of the path up through index.html. So
run these commands and verify from the permissions of each component
of the path that user/group nobody
can read and search
that component.
ls -ld / ls -ld /opt ls -ld /opt/local ls -ld /opt/local/HTTPServer ls -ld /opt/local/HTTPServer/htdocs ls -ld /opt/local/HTTPServer/htdocs/en_US ls -l /opt/local/HTTPServer/htdocs/en_US/index.html
If the directory where you choose to install IBM HTTP Server is not within a directory created by the system, you may need to add read and search permissions to that directory so that the web server user/group can access the files.
[error] [client nnn.nnn.nnn.nnn] (13)Permission denied: unable to connect to cgi daemon after multiple tries: /path/to/cgi
This message can be received for CGI requests with IBM HTTP Server
2.0 or above on Unix and Linux systems. mod_cgid
implements the CGI feature in these levels of IBM HTTP Server. At IBM HTTP Server
initialization, mod_cgid creates a Unix socket in the filesystem which
is used when executing CGI scripts. The default name for this Unix
socket is logs/cgisock
within the ServerRoot
directory. (Example name: /usr/IBMIHS/logs/cgisock
) The name
of this Unix socket in the filesystem can be changed with the
ScriptSock
directive. Example:
ScriptSock /tmp/scriptsock
There are two requirements for the filesystem path of this Unix socket:
ServerRoot
directive, the ScriptSock
directive must be used in each configuration file to specify a
unique name. Otherwise, security exposures can occur if different web
servers have different credentials, since CGI requests may end up
being executed by the wrong IBM HTTP Server instance. With cumulative
e-fix PK01070 or later for IBM
HTTP Server 2.0.x, mod_cgid will refuse to execute a CGI request if
this configuration error exists, since unexpected privileges could be
used otherwise.
User
and Group
directives).
Assume the following configuration:
User nobody Group nobody ScriptSock /usr/IBMIHS/logs/scriptsock
If the permission denied error is written to the error log when a CGI request is made, the expected cause is that there is a filesystem permissions issue which prevents the web server user id from accessing the Unix socket. Perform the following checks:
Here is an example where the logs directory has been made readable only by root. In this case, the permission denied error would occur when trying to process a CGI request. The fix would be to use the ScriptSock directive to specify a Unix socket file that the web server user id could access, or to make the logs directory readable by the web server user id.
[trawick@b80-2 /]$ ls -ld /usr drwxr-xr-x 63 bin bin 1536 Mar 7 16:15 /usr [trawick@b80-2 /]$ ls -ld /usr/IBMIHS drwxr-xr-x 22 root system 512 May 19 2004 /usr/IBMIHS [trawick@b80-2 /]$ ls -ld /usr/IBMIHS/logs drwx------ 2 root system 512 May 19 2004 /usr/IBMIHS/logs
If the problem persists, send in the following information to IBM support:
[error] [client 127.0.0.1] Invalid method in request unprintable-chars
This problem is almost always caused by an SSL request being handled by a virtual host which is not SSL-enabled. It will be accompanied by a message similar to the following in the access log:
127.0.0.1 - - [15/Apr/2005:11:42:51 -0400] "\x80\x8c\x01\x03\x01" 501 310 - -
Here are some common points of confusion which can lead to this issue:
One cause for this is a defect in IBM HTTP Server 2.0.x, resolved by PK01070 or later, which respected only the first returned IP address for the host name.
Another cause for this is when the DNS used by IBM HTTP Server returns different information than the DNS used by the client.
To diagnose the configuration problem, make these configuration changes:
%A
) to the access log
format so that the IP address used by the client on the failing
request can be seen.
SetEnv vhostname MAIN
to the main scope of
httpd.conf.
SetEnv vhostname UNIQUE-NAME
to each
VirtualHost container. Make sure UNIQUE-NAME is unique for
each virtual host.
%{vhostname}e
) to the access log format.
Example log format with both of these changes made:
LogFormat "%h %l %u %t \"%r\" %>s %b %A %{vhostname}e" common
Example setting of vhostname:
... SetEnv vhostname MAIN ... <VirtualHost 192.168.1.15:8443> ... SetEnv vhostname vhost8443 </VirtualHost> <VirtualHost 192.168.1.1:443> ... SetEnv vhostname vhost443 </VirtualHost>
Now restart IBM HTTP Server and try the request again. Check the access log for the destination IP address and the vhost name:
127.0.0.1 - - [15/Apr/2005:11:58:24 -0400] "\x80\x8c\x01\x03\x01" 501 310 127.0.0.1 MAIN
In this example, the IP address in next to last column is the IP address used by the client. So you have to check if there is an SSL-enabled virtual host which specifies that IP address (or *) on the <VirtualHost > directive.
The last column is the virtual host which was actually chosen. In this example, it is the main scope of httpd.conf, because the IP address used by the client did not match the IP address on any <VirtualHost > directives. Whichever virtual host was chosen, verify that there is no SSLEnable directive for it, and decide if the client sent the request to wrong vhost (due to bad link?) or if the SSLEnable directive is supposed to be specified or if another VirtualHost container is appropriate.
[error] [client nnn.nnn.nnn.nnn] Directory index forbidden by rule: /usr/IBMIHS/docroot/path/
A request has been issued from the browser, and that request maps
to an actual directory on the web server machine under DocumentRoot,
and there is no active DirectoryIndex directive/document to return a
document such as index.html
to the browser, so IBM HTTP
Server tries to send a directory listing back to the client. However,
the Indexes option is not active for the directory so an
error is returned.
To resolve the problem, you need to determine what response was supposed to be returned on this request.
If the request is not expected, check for links on your site which specify that URL.
If a directory listing was supposed to be returned, enable the Indexes option for that directory. Example:
<Directory /usr/IBMIHS/docroot/path/> Options +Indexes </Directory>
If a static file such as index.html
was supposed to be
returned, enable that as the DirectoryIndex
for that
directory. Example:
<Directory /usr/IBMIHS/docroot/path/> DirectoryIndex index.html </Directory>
If the request is supposed to map to a WebSphere request, check the WebSphere plug-in configuration file and make sure there is a rule to send requests for that URL to WebSphere.
[warn] [client nnn.nnn.nnn.nnn] mod_was_ap20: /some/URL not found
This message is issued by the WebSphere plug-in, and it indicates a plug-in processing error, due either to the plug-in implementation or to the plug-in configuration. A common cause of this error is fixed by APAR PK01215 (WebSphere plug-in). It can affect ErrorDocument processing as well as some other types of IBM HTTP Server processing. A WebSphere plug-in update including that fix needs to be applied before the error can be investigated further.
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
[emerg] (28)No space left on device: Couldn't create accept lock
Check your system sysv semaphore usage and limits
[error] [client aaa.bbb.ccc.ddd] internal error: bad expires code: (null), referer: http://www.example.com/welcome.html
This problem with IBM HTTP Server 2.0.42.x/2.0.47.x is limited to extra error message logging. It is resolved by any maintenance after PQ85834. The customer should apply the current recommended update.
[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2513
setuid() can return error code 11 when the NPROC rlimit for the user would be exceeded. Edit /etc/security/limits.conf or use ulimit to increase limit on number of processes.
[error] mod_ibm_ldap: unable to authenticate the web server for realm 'Example LDAP Realm': Encoding error.
If this is Solaris, verify that the SUNWuiu8 package is installed:
# pkginfo SUNWuiu8 system SUNWuiu8 Iconv modules for UTF-8 Locale
If this is AIX and IHS is launched with an environment containing LC_MESSAGES=c@lft
(e.g. via crontab or /etc/inittab) then either unset LC_MESSAGES or check for the AIX APAR
listed below:
APAR | Release |
---|---|
IY59922 | 5.1 |
IY56520 | 5.2 |
IY59458 | 5.3 |
To circumvent the problem without applying the AIX fix, add the
following to the bottom of IHSROOT/bin/envvars
:
# work around AIX setlocale() problem fixed by IY59922/IY56520/IY59458 unset LC_MESSAGES
[crit] Unable to set default LDAPCodePages directory relative to ServerRoot.
If the ServerRoot directive has been changed from the default value, copy (or symlink) the codepages directory from the installation path into the new ServerRoot
[warn] proxy: No protocol handler was valid for the URL /example/url.js. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
The usual cause of this is the lack of a LoadModule directive to load mod_proxy_http.so when supporting HTTP proxy. Uncomment, or add, the following directive:
LoadModule proxy_http_module modules/mod_proxy_http.so
If the specified URL is handled via FTP, activate mod_proxy_ftp.so instead.
This message can also be seen with some levels of IBM HTTP Server when SSL is used for the proxy connection, and an error occurs setting up the connection. If the necessary proxy modules are already loaded and the proxy connections are https, apply one of the following fixes:
[notice] suEXEC mechanism enabled (wrapper: /usr/IBMIHS/bin/suexec)
This message indicates that suEXEC could be used if configured in httpd.conf. In order to configure it, the SuexecUserGroup directive is coded. See this suexec how-to for more information.
Cannot load IHS_PATH/afpaplugin.dll into server: The specified module could not be found.
This error message may occur when the customer is trying to use IBM HTTP Server with the Websphere Application Server high speed external caching feature and has configured the wrong version of afpaplugin.
The afpaplugin.dll file, which is a component of Webshpere Application Server, is only for use with IBM HTTP Server 1.3.x. The afpaplugin_20.dll file, which is installed via the Websphere 6.x Plugins installation, is required when using IBM HTTP Server 2.0.x and above.
When using IBM HTTP Server 2.0.42 or 2.0.47, cumulative e-fix PQ94086 or later must be applied. With IBM HTTP Server 6.x no additional fix pack is required.
Other symptoms that might be experienced are slow IHS response time or the appearance that IHS has hung.
Try adding Win32DisableAcceptEx to the IHS configuration file and restart IHS.
Note: AFPA must be disabled before adding the Win32DisableAcceptEx directive. See technote Disabling AFPA (fast cache accelerator) in IBM HTTP Server for details of how to disable AFPA.
apache.exe: Could not determine the server's fully qualified domain name. using IP_ADDR for server name.
If this message appears in Event Viewer, and you're using the IHS 6.1 Admin Server,
add the 'ServerName yourhostname
' directive to <ihsinst>/conf/admin.conf
.
[error] [client ip-address]
(errno)error-string (errno2=errno2): SAF
user userid: authentication failure for
"request-uri": Password Mismatch
The server logs this message when an invalid password is specified. Common error conditions are:
errno value | errno2 value | meaning |
---|---|---|
111 | 0xnnnn0000 | The password is of a valid length but is not valid. |
121 | 0xnnnn02A7 (JRPasswordLenError) | SAF reports that the password has an invalid length. (mod_auth_saf did not make this determination.) |
[error] [client ip-address]
(errno)error-string (errno2=errno2): SAF
user userid not found: request-uri
The server logs this message when an invalid user id is specified. Common error conditions are:
errno value | errno2 value | meaning |
---|---|---|
143 | 0xnnnn05DD | The user id is of a valid length but is not defined to SAF. |
121 | 0xnnnn02A6 (JRUserNameLenError) | SAF reports that the user id has an invalid length. (mod_auth_saf did not make this determination.) |
[debug] (168)user [username]: SAF authentication failure 401-168
for [request-uri]: password has expired
The server logs this message when a user has entered a correct SAF password, but the password has expired. A call to the function __passwd() returned EMVSEXPIRE. Access is denied.
To enable updating of expired SAF passwords using the server, code the directive "AuthSAFExpiration". For example:
AuthSAFExpiration on
You may also tailor the message to be presented at the client by coding some brief text instead of "on". For example:
AuthSAFExpiration "EXPIRED PW: oldpw/newpw/newpw"
You can also use the optional AuthSAFReEnter directive to further tailor the next password message issued at the client. For example:
AuthSAFReEnter "New PW again:"
[crit] [client ip-address] (157)EDC5157I An
internal error has occurred. (errno2=0x090C02AF): user
userid: SAF authentication failure 500 for request-uri:
function is not supported
The server logs this message when a call to the function __passwd() returns EMVSERR.
Check the errnoJr value for the exact cause. The most common cause, as seen in the example above where errno2 is 02AF, is that you have not marked for program control one or more modules that the server loads. You can use the command "extattr +ap [modulename]" to mark a module program controlled.
[info] [client ip-address] (errno-value)errno-description
(errno2=errno2-value) user userid: saf_check_pw=401 for request-uri:
Other SAF failure; errno2=yyyyzzzz
The server logs this message when a call to the function __passwd() returns an error which the server does handles with specific messages. The combination of errno and errno2 explain the problem.
UNIX System Services Messages and Codes describes all errno and errno2 values. C/C++ Run-Time Library Reference contains documentation for the __passwd() function.
[error] [client ip-address] access to [request-uri] failed: [requirement]
The server logs this message when it denies access to a resource based on the options in a "Require" directive in httpd.conf.
If access should have been granted, review the options in the "Require" directive.
[info] [client ip-address] access to [request-uri] failed; reason:
user [username] is not in the "Require" list
The server logs this message when it denies access to a resource because a "Require" directive in httpd.conf does not specifically include the username or a group containing the username.
If access should have been granted, review the options in the "Require" directive.
[info] (xxx)rc=[return code] attempting to reset UID
The server logs this message when it has run a request under SAFRunAs control and is attempting to delete the SAF security environment and return to the server's default user ID (UID).
See the documentation for errno xxx and the pthread_security_np() function in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.
[warn] [client 9.37.242.143] [16777781] auth_ldap authenticate:
user [xxxx] authentication failed; URI /ldapdir/index.html [User not
found][No such object]
The server logs this message if auth_ldap cannot find the requested dn or user, but mod_ldap is not authoritative.
The following messages will also be logged if no module is authoritative and the user is not found by any authorization module:
[crit] [client 9.37.242.143] check_user_id hook called [crit] [client 9.37.242.143] configuration error: couldn't check user. No user file?: /ldapdir/index.html
See http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_auth_ldap.html#authldapauthoritative.
Each authorization module only authenticates based on its own
knowledge. If directive AuthLDAPAuthoritative off
is
specified, mod_auth_ldap will not reject the
request, but will allow it to be passed to other possible
authorization modules. Normally, mod_auth_ldap is the only
authorization module configured for a request and
AuthLDAPAuthoritative
is set to on.
[warn] [client 9.76.147.159] [43]
auth_ldap authenticate: user [username] authentication failed; URI
[path/to/ldap/directory] [LDAP: ssl connections not
supported][N/A]
The server logs this message if an LDAP session was requested but failed.
If an SSL connection to the LDAP server is required, the url specified in AuthLDAPURL must begin with ldaps://, non-SSL must begin with ldap://. Additional messages which describe the problem may have appeared earlier in the error log.
[error] (121)EDC5121I Invalid argument.
(errno2=0x07200316): LDAP cache: error while creating a
shared memory segment: EDC5121I Invalid argument.
(errno2=0x07200316)
[error] (153)EDC5153I Catalog obtain error.
(errno2=0xC5C7082A): LDAP cache: error while creating a shared memory segment: EDC5153I Catalog obtain error.
(errno2=0xC5C7082A)
Ensure that the size specified by LDAPSharedCacheSize is valid.
Ensure that the file specified in LDAPSharedCacheFile is valid.
Delete the file specified in LDAPSharedCacheFile and try starting IBM HTTP Server again.
If a LDAPSharedCacheFile directive is specified remove it; if it is not specified then add it. If the directive is specified, named shared memory will be used; otherwise, anonymous shared memory will be used.
[crit] [return code] LDAP cache: failed to set mutex
permissions
The server may log this message if the User or Group directive in httpd.conf does not have appropriate permissions.
See the documentation for semctl() IPC_SET operation in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.
[crit] LDAP: Invalid LDAPTrustedCAType
directive - KDB_FILE or SAF_KEYRING type required or mismatched
The server logs this message if the KDB file, SAF keyring, or password file are invalid.
.kdb
and the associated associated password file must be
have the same name as the kdb file, but with a file extension of
.sth
.[warn] LDAP: SSL initialization failed
[notice] LDAP: SSL support unavailable
The server logs one or both of these messages if SSL was not requested, or fails. If SSL support is required, check for additional messages in the log and directives such as LDAPTrustedCA and LDAPTrustedCAType.
Contact the supplier of any third-party modules in the configuration in order to address these messages.
SemCounter ERROR: Cannot remove semaphore: 655433
[notice] msgsnd() FAILED can't send, ipc msg queue is full
(EAGAIN=Resource temporarily unavailable)
[error] [client 1.2.3.4] (13)Permission denied: Could not open file /apps/opt/IBMHTTPD-SSO/logs/filter.52954
1172793600:28168: CSmSharedSegment::smalloc - attached to shared memory segment 5111808 using key 0x6910a016
Several web server functions depend on the fork()
system call to create a new process. Specific requests can fail or
web server performance can be degraded if this system call fails.
The specific cause for failure is usually one of the following:
More information may be available in the operating system
documentation for the fork()
subroutine.
AIX
The AIX documentation for fork()
is
http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/libs/basetrf1/fork.htm.
The applicable limit is maxuproc
. The following
commands show how to display the current value and set it to a new
value:
# lsattr -El sys0 -a maxuproc maxuproc 2048 Maximum number of PROCESSES allowed per user True # chdev -l sys0 -a maxuproc=4096 sys0 changed