Tivoli Performance Viewer is very slow on systems with a large number of nodes and/or servers
 Technote (troubleshooting)
 
Problem(Abstract)
In a cell configured to have 30 nodes, IBM® Tivoli® Performance Viewer was configured to monitor only 6 out of the 30 nodes in a cell. It takes 15 to 20 minutes for the initial view to come up and takes a very long time for the screen to repaint on a refresh.

Sometimes java.lang.OutOfMemoryErrors occur.
 
Cause
This is a side-effect of the following problem.

By default, the ORB uses a Java™ thread to process each inbound connection request it receives. Each of the connection requests is processed by an ORB reader thread. As the number of concurrent requests increases, the Java heap memory consumed by the reader threads increases and may become a bottleneck in resource-constrained environments. The number of Java threads created may cause out-of-memory exceptions if the number of concurrent requests exceeds the system's available resources.

This means that the number of threads will increase beyond the ORB maximum thread pool size, which is a setting for the ORB worker thread pool, not reader thread pool.

 
Resolving the problem
To help reduce the number the ORB may be configured to use JNI reader threads where a finite number of reader threads (implemented using native OS threads instead of Java threads) are created when the ORB is initialized via the ORB custom properties, com.ibm.CORBA.ORBPluginClass.com.ibm.ws.orbimpl.transport.JNIReaderPoolImpl and com.ibm.CORBA.numJNIReaders. These parameters may be specified as ORB Service custom properties or as command line arguments for the IBM WebSphere® Application Server.
NOTE: Make sure the JNI library is in the Application Server bin directory.
For Intel® platform, the library is Selector.dll and for UNIX®, it is libSelector.a or libSelector.so. For UNIX, if the prefix "lib" is missing, the file should be renamed

Specifying the parameters using the ORBService Custom Properties
To specify these properties using the Administrative Console, navigate to
Application Servers > [your_server] > ORB Service > Custom Properties

Name:com.ibm.CORBA.ORBPluginClass.com.ibm.ws.orbimpl.transport.JNIReaderPoolImpl
Value: ""

Name: com.ibm.CORBA.numJNIReaders

Value: 1024 or less. Default is 4


Specifying the parameters using the Generic JVM Arguments
To specify these parameters as Java command line parameters,

  1. Expand "Servers", choose "Application Servers" and select the problem server.
  2. Under "Additional Properties", click "Process Definition"
  3. Under "Additional Properties", select "Java Virtual Machine"
  4. Enable the JVM Verbose class loading option
  5. Check the Verbose class loading checkbox
  6. Enter the following system properties into the Generic JVM Argument field:
-Dcom.ibm.CORBA.ORBPluginClass.com.ibm.ws.orbimpl.transport.JNIReaderPoolImpl with no value.
-Dcom.ibm.CORBA.numJNIReaders=xx where xx is the number of reader threads


Another possible performance improvement can be made by setting the location of wstemp directory to a "ramdisk". This reduced the time it took the Tivoli Performance Viewer to start from 15 minutes to just seconds as well as improving the servers performance.
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
Historical Number
03245
370
000
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > PMI/Performance Tools
Operating system(s): AIX
Software version: 5.1.1
Software edition:
Reference #: 1204385
IBM Group: Software Group
Modified date: Apr 22, 2005