ORB reader threads causing slow performance and OutOfMemoryErrors
 Technote (troubleshooting)
 
Problem(Abstract)
On a cell with 30 nodes, performance is extremely slow. A simple save through the administrative console can take 20 minutes.
 
Symptom
The ORB thread pool reaches 50 for each JVM but thread dumps show there are hundreds of ORB readers threads. The system eventually cores with OutOfMemory conditions.
 
Cause
The ORB thread pool setting only refers to worker threads. The ORB uses a Java thread for processing each inbound connection request it receives. As the number of concurrent requests increases, the storage consumed by a large number of reader threads will increase and may become a bottleneck in resource-constrained environments. The number of Java threads created may cause out-of-memory exceptions if the number of concurrent requests exceeds the system's available resources.

This means that the number of threads will increase beyond the ORB max thread pool size, which is a setting for the ORB worker thread pool, not reader thread pool.

 
Resolving the problem
To help reduce the number, the ORB may be configured to use JNI reader threads where a finite number of reader threads (implemented using native OS threads instead of Java threads) are created when the ORB is initialized.

Specify this property using the administrative console.
These can also be set in the Admin Console > Application Servers > [your_server] > ORB Service > Custom Properties.

Name: com.ibm.CORBA.ORBPluginClass.com.ibm.ws.orbimpl.transport.JNIReaderPoolImpl
Value: ""

Name: com.ibm.CORBA.numJNIReaders
Value: 1024 or less. Default is 4

As Java command line parameters:
-Dcom.ibm.CORBA.ORBPluginClass.com.ibm.ws.orbimpl.transport.JNIReaderPoolImpl with no value.
-Dcom.ibm.CORBA.numJNIReaders=xx where xx is the number of JNIReader threads

Note: Make sure the JNI library is in the WebSphere® Application Server bin directory.
For Intel platforms, the library is Selector.dll and for UNIX™, it is libSelector.a or libSelector.so. For UNIX, if the prefix "lib" is missing, the file should be renamed.
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Object Request Broker (ORB)
Operating system(s): AIX
Software version: 6.1
Software edition:
Reference #: 1204384
IBM Group: Software Group
Modified date: Sep 28, 2007