IBM HTTP Server - Resolving problems with too many open files

Unix and Linux systems have a per-process limit on the number of open files. In some configurations, particularly with a large number of threads per child process, web server operations can fail due to reaching the limit. Sockets can no longer be created, client connections can no longer be accepted, files can no longer be opened, etc.

The problem is usually interimittent; the web server is continually closing files or sockets that it has opened, so the limit is reached briefly under the higher load, and the file descriptor usage can then lower, in a cycle.

Here are just a few examples of messages which can appear in the error log when this limit is reached:

[crit] (24)Too many open files: SSL0600S: Unable to connect to session ID cache
[error] (24)Too many open files: apr_accept: (client socket)
(24)Too many open files: file permissions deny server access: /opt/IBMHttpServer/htdocs/en_US/index.html

Steps to resolve the problem:

1. Stop the web server.
2. Display the current per-process hard and soft limits.

       # ulimit -H -n
       1024
       # ulimit -n
       256
       

3. Set these limits to a higher value.

       # ulimit -H -n 4096
       # ulimit -n 4096
       

4. In the same shell environment, start IBM HTTP Server.

Consult your operating system documentation to determine how to make the change permanent.

Additional notes

• When the limit is hit during the setup for a new client connection (the apr_accept error message above) the affected child process will perform a graceful shutdown; this graceful shutdown allows any pending client requests to complete uninterrupted.
• Take the sum of the following terms for a rough estimate for the number of open files that may required for a given web server child process

  Hint: The value can regularly be well over ThreadsPerChild*2

  • ThreadsPerChild
  • Number of Listen directives
  • Number of of CustomLog/ErrorLog directives
  • maximum number of backend connections via the WebSphere Plugin
  • maximum outgoing connections for mod_proxy or mod_ibm_ldap
  • 5 (fixed overhead)