Tuning interruptible processes

Use this task to improve the performance of interruptible business processes.

Why and when to perform this task

Interruptible processes include user-interaction, they use database and messaging subsystems for storing persistent states and reliable communications. The performance of interruptible processes can be improved by tuning the database, the messaging system, and the application server settings.

Steps for this task

  1. Balance the hardware.
    A general rule of thumb before starting to tune the system, is to verify that the computer used is well balanced, so that the available resources, CPU, memory, and input/output have the right relationship. A computer with one (or many) very fast CPUs but low memory or low input/output performance will be hard to tune.
    1. Allocate enough disks.
      For interruptible processes good input/output performance and multiple, fast disk drives are as important as enough processing power and a sufficient amount of memory.

      Separate disks are preferred over using RAID arrays because that gives more fine grained tuning capabilities. When using RAID arrays the optimum number of disks is higher, because of the redundancy required for fail-over capabilities.

      Ideally a system that runs interruptible processes has six or more disk drives:
      • One disk for the operation system and the swap space (page file on Windows, paging space on AIX, swap space on Solaris, paging space on HP-UX
      • One disk for WebSphere and its transaction log
      • One disk for the transaction log of the message queuing system
      • One disk for the persistent queue storage of the message queuing system
      • One disk for the transaction log of the database management system
      • One (or more!) disks for the process choreographer database in the database management system. If more disks are available the instance table should be distributed on two or more disks.
    2. Allocate enough memory.
      For a Windows system with 3 GB of physical memory, a local database, and a local WebSphere MQ queue manager, the following memory allocation is recommended:
      • 256 MB for Windows
      • 1024 MB for WebSphere application server
      • 256 MB for WebSphere MQ
      • 1.5 GB for the database
      For an AIX system with 8 GB of physical memory, a local database, and local WebSphere MQ queue manager, the following memory allocation is recommended:
      • 512 MB for AIX
      • 1024 MB for WebSphere application server
      • 512 MB for WebSphere MQ
      • 6 GB for the database

      Note: The recommended memory for the application server, when running process choreographer, is more than the default value. How to tune the application server heap size is described later, when tuning thethe application server.

    3. Move workload to other machines.
      Consider which applications or subsystems could be moved to other machines. For example, the Web server, other applications running on the application server, hosting the process choreographer database on a dedicated machine.
  2. Initial database settings
    1. Separating log and data:
      Putting the database log file on a disk drive separate from the data tends to improve performance, provided that a large enough number of disk drives is available. If only a small number of disks are available then distributing the table spaces, as described in the previous section, is usually more beneficial than putting the database log on a separate drive.
      For example, on windows, using DB2, you can change the location of the log files for the database named BPEDB to the directory F:\db2logs, by entering the following command:
      db2 UPDATE DB CFG FOR BPEDB USING NEWLOGPATH F:\db2logs
    2. Creating table spaces:
      After creating the database, it is highly suggested to explicitly create table spaces. Example scripts to create table spaces are provided by process choreographer in the ProcessChoreographer subdirectory of your WebSphere installation. These scripts can be customized to accommodate the needs of a particular scenario. When creating the table spaces the goal is to distribute input/output operations over as many disk drives as is available to DB2. By default these scripts create the following table spaces:
      • The STAFFQRY table space contains the tables that are used to temporarily store staff query results obtained from staff registries like LDAP. When using many person activities in business processes then tables in this table space is frequently accessed.
      • The AUDITLOG table space contains the audit trail tables. If auditing is turned off no access to tables in this table space happens. Depending on the degree of auditing used access to tables in this table space may be significant though.
      • The COMP table space is used for the compensation tables. If compensation is not used within business processes no access to tables in this table space happens. Depending on the percentage of compensatible processes and activities, the tables in this table space may require high input/output bandwidth.
      • The INSTANCE table space holds the process instance tables. It is always used intensively, regardless of the kind of interruptible process that is run. It is suggested, where possible, to spread this table space over multiple disk drives.
      • The WORKITEM table space holds the tables required for work item processing. Work items are used by people interacting with business processes. Depending on the amount of people activities in the business processes used access to the tables in this table space may vary from a low access rate to significant input/output. The access rate will not be zero even when no people activities are used, because work items are also generated to allow administration of interruptible processes.
      • The SCHEDTS table space contains the table used by the WebSphere scheduling component that is used by process choreographer. Due to the caching mechanisms used in the scheduler, access to tables in the scheduler table space is usually low.
      The way to get the database for process choreographer is to run the script createDatabseDb2.ddl creating the database, the table spaces and the tables. This way is recommended by the Business Process Container Install Wizard but you cannot use it if you want the data to be distributed across different disk. To create a database for high performance, perform the following:
      1. Create the database only. For example, to create a DB2 database in the directory D:\BPE for process choreographer, enter the command:
        CREATE DATABASE BPEDB ON D:/BPE USING CODESET UTF-8 TERRITORY en-us;
      2. Create the table spaces on the desired disks. For example, the following script is based on the file createTablespaceDb2.ddl located in the ProcessChoreographer subdirectory of your WebSphere installation. It creates table spaces using three different disk drives on a Windows system:
        -- Scriptfile to create tablespaces for DB2 8.1
        -- to run the script call
        --      db2 connect to BPEDB
        --      db2 -tf createTablespaceDb2.ddl
        
        CREATE TABLESPACE TEMPLATE MANAGED BY SYSTEM USING( 'D:/BPE/TEMPLATE' );
        
        CREATE TABLESPACE STAFFQRY MANAGED BY SYSTEM USING( 'D:/BPE/STAFFQRY' );
        
        CREATE TABLESPACE AUDITLOG MANAGED BY SYSTEM USING( 'E:/BPE/AUDITLOG' );
        
        CREATE TABLESPACE COMP MANAGED BY SYSTEM USING( 'D:/BPE/COMP' );
        
        CREATE TABLESPACE INSTANCE MANAGED BY SYSTEM USING( 'D:/BPE/INSTANCE', 'E:/BPE/INSTANCE');
        
        CREATE TABLESPACE WORKITEM MANAGED BY SYSTEM USING( 'F:/BPE/WORKITEM' );
        
        CREATE TABLESPACE SCHEDTS MANAGED BY SYSTEM USING( ' F:/BPE/SCHEDTS' );
        
      3. Create the process choreographer tables using the script provided for your database in the ProcessChoreographer directory. For example, for DB2, use the file createSchemaDb2.ddl.
    3. Tuning the database
      Use a capacity planning tool for your initial database settings.
      If you are using DB2, start the DB2 configuration advisor from the DB2 Control Center by selecting DB2 configuration advisor from the context menu of the process choreographer database. Make sure you:
      • Do not allocate more memory to DB2 than is physically available to it without swapping.
      • Select a Mixed workload (queries and transactions).
      • For Typical transaction, select more than 10, to indicate that long transactions are used.
      • For Database administration priority, it is recommended to tune the database for faster transaction performance and slower recovery.
      • For Data, the initial database tuning must performed without data in the database.
      • Number of applications determines limits the maximum number of parallel connections that can be made to the application server. Consider the following guidelines:
        • The amount of parallelism that is possible is limited by memory and CPU resources. Details are discussed in the section on WebSphere tuning below. As a base line on a 2-way Intel based server with 1.7 GHz CPU clock speed and 3 GB of memory N=50 parallel connections to the database produced good results. This value can also be used as a starting point on larger nodes.
        • To be able to offer P parallel JDBC connections to the application server, DB2 should be configured for N local applications, where N = 1.1 * P. How to estimate a realistic value for P is described later.
        • If process choreographer is running on the same physical machine as the database, then it does not need any remote database connections. Remote connections may be required for remote database management though, in which case it is advisable to use a low value instead of zero.
        • If process choreographer and DB2 are installed on separate machines then the number of remote applications has to be set according to the rule described above for local connections.
      • For Isolation level, it is recommended to select read stability.
      At the end the configuration advisor will display the changes it is suggesting. You can either apply the changes now, or save them to a file to apply later.
  3. Settings to consider before creating your queue manager.
    To achieve the best performance for interruptible processes, you must tune the message queuing system for maximum performance of persistent messages. It is recommended to use the IBM WebSphere MQ messaging product rather than the embedded messaging provider that comes with this product.
    1. Perform Tuning Java messaging service to tune things like:
      • Log file pages
      • Log buffer page
      • Log primary files
      • Log secondary files
      • Log default path
      • Maximum channel
      • Channel application bind type
      The default locations for both the persistent queue data and the MQ logs is the MQ install directory. It is strongly recommended to put the data storage for the persistent queues and the WebSphere MQ logs on different disk drives. By changing the Log default path parameter to another disk drive, the location for the MQ logs can be changed. You must do this before creating the queue managers for process choreographer.
    2. Tuning WebSphere MQ service properties for process choreographer:
      These values have to be set before creating the queue manager(s) used by process choreographer. It is recommended to set the parameters to their respective maximum:
      • Log file pages: 16384

        Note: On Windows systems, not all versions of WebSphere MQ allow you to set the number of log file pages to 16384 using the MQ administration tools. A workaround is to change the value of the Windows registry key:

        HKEY_LOCAL_MACHINE\SOFTWARE\IBM\MQSeries\CurrentVersion\Configuration\LogDefaults
        to 16384.

      • Log primary files: 62
      • Log secondary files: 1
      • Log buffer pages: 512
    3. The following queue manager properties are recommended:
      • Maximum channels: Use the default
      • Channel application bind type: FASTPATH
  4. Tuning the application server
    1. Estimate the application server resources you need for process choreographer:
      Each business process container requires the following resources to run properly (names in brackets are the default names used in a standard installation of process choreographer):
      • Four listener ports:
        • One for internal messages (BPEInternalListenerPort).
        • One for internal messages (BPEInternalListenerPort).
        • One for external API messages (BPEApiListenerPort).
        • One for unprocessed messages (BPEHoldListenerPort).
      • Two queue connection factories:
        • One for getting message (BPECF).
        • One for putting messages (BPECFC).
      • One data source to read and write business process state information to a database (jdbc/BPEDB).
      If you are using interruptible processes (for example, processes involving human interaction), then you must also allow the following resources for each possible concurrent navigation of an interruptible process:
      • One JDBC connection.
      • One MDB session on the listener port.
      • Two JMS connections on the factory (one each for get and put).
      • Two JMS sessions on the factory (one each for get and put).
      In addition:
      • If you are using the process choreographer API to start processes or to interact with running processes, additional database connections will be required.
      • Allow additional JMS sessions to allow for system overloads that trigger process choreographer to enter quiesce mode.
      • Keep in mind that it does not make sense, from a performance tuning point-of-view, for the number of MDB sessions and JMS connections that you allow in the application server to exceed the number of JDBC connections than the database makes available.
      Your estimate for the maximum number of parallel JDBC connections required at any moment, provides the value for P, which is used in following steps.
    2. To tune the JDBC settings, perform the following:
      1. For the JDBC provider, set Max Connections to the value P, as described in Connection pool settings.
      2. For the Data source, it is recommended to set the SQL Statement cache size to 500, as described in Data source settings.
    3. Configuring a queue connection factory, WebSphere MQ JMS provider describes how to change the connection factory settings using the administrative console. Which settings you must tune for process choreographer depends on the platform you are using:
      • On Windows systems,
        1. For the JMS Connection Factory for 'get', BPECF, set both Max Connections (Connection Pool) and Max Sessions (Session Pool) to the previously calculated value, P.
        2. For the JMS Connection Factory for 'put', BPECFC, set both Max Connections (Connection Pool) and Max Sessions (Session Pool) to the previously calculated value, P.
      • On UNIX and Linux systems, it is not possible for multiple WebSphere MQ clients to connect in binding mode
        1. For the JMS Connection Factory for 'get' and 'put', BPECF, set both Max Connections (Connection Pool) and Max Sessions (Session Pool) to the previously calculated value, P, multiplied by two.
    4. Configuring the message listener service describes how to change the message listener service properties using the administrative console. To tune the JMS message listener service fro process choreographer, set the following properties:
      • For the internal listener port used by process choreographer (default name: BPEInternalListenerPort), set Max Sessions to the previously calculated value P.
      • For the Maximum Size for the thread pool for the message listener, set a value (P + number of sessions for other listeners) to ensure that the pool has enough threads for each session that is configured for all the listener ports.
    5. Tuning the JVM heap size
      To change the JVM heap size, see Java virtual machine settings. For most production systems that use process choreographer the default application server heap size is too small, and must be increased because of all the extra resources that process choreographer and process applications need, foe example, JDBC connections and JMS connections. Generally:
      • 256 MB is too low, and results in poor performance.
      • 512 MB is recommended as an initial heap size that will be enough for many systems.
      • 1024 MB is considered a sensible upper limit.

      Make sure that the total memory allocated to applications is no larger than the amount of physical memory available, otherwise paging will significantly reduce your system's performance, and it might even impact the stability of your system.

      If your database is running on the same machine as your application server, any extra memory allocated for heap size, will probably have to be subtracted from the memory allocated to the database. Since it is recommended to allocate all remaining physical memory to the database, use the following equation to rebalance the allocation for the database:

      Memory for database = (Physical memory - Memory for operating system - Memory for application server - Memory for WebSphere MQ)

  5. Fine tuning WebSphere MQ to utilize more CPU.
    If, after having completed the previous tuning actions, your CPU is less than 80% utilized, or the CPU utilization graph is bursty, it is worth experimenting with the custom property non.asf.receive.timeout. Configuring the message listener service describes how to change this property using the administrative console.

    Set the value of the non.asf.receive.timeout property to the average time in milliseconds required to process a message. When applied to process choreographer that corresponds to the time required to process an activity in an interruptible process.

    Tests in a lab environment gave the following results:

    • Values below 50 ms resulted in constant utilization of the CPU without a positive impact on throughput.
    • The best results were obtained using 500 ms for a slower machine and 1200 ms for a faster machine.
    • Higher values did not resolve bursty behavior and did not improve throughput.
    • Finding the optimal value for a given hardware configuration and workload mix must be established by running multiple throughput tests using different values.

  6. Fine tuning the database:
    1. If you are not using DB2, refer to your the documentation for your database system for information about monitoring the performance, identifying and eliminating bottlenecks, and fine tuning the performance.
    2. If you are using DB2, check the db2diag.log file for your DB2 instance for messages like:
      2004-08-25-15.53.42.078000   Instance:DB2   Node:000
      PID:2352(db2syscs.exe)   TID:4360   Appid:*LOCAL.DB2.027785142343
      data management  sqldEscalateLocks Probe:4   Database:BPEDB
      
      ADM5503E  The escalation of "10" locks on table "GRAALFS .ACTIVITY_INSTANCE_T" 
      to lock intent "X" has failed.  The SQLCODE is "-911".
      
      Getting such messages indicates that the parallelism for business process applications has improved so much, that number of locks available is now too small. It is recommended to increase the LOCKLIST value to approximately (10 * P).
      For example, if you sized your process choreographer database, BPEDB, with a value of P=50, enter the following command:
      db2 UPDATE DB CFG FOR BPEDB USING LOCKLIST 500
    3. If you used the DB2 configuration advisor, your database should already have a good throughput.
      To improve the performance further:
      • Follow the best practices for database tuning that are described in the DB2 online documentation, books, and articles.
      • Consider using DB2 monitors and examining the db2diag.log file for more information on bottlenecks within the database.
      • Consider tuning the following DB2 parameters:
        • MINCOMMIT
        • AVG_APPLS
        • NUM_IO_CLEANERS
        • LOG_FILSIZ
        • SOFTMAX
        • DBHEAP
        • PACKCACHESIZ

Results

Your interruptible processes are running as fast as possible under the current environment and loading conditions.




Searchable topic ID:   t5tuneint
Last updated: Jun 21, 2007 8:07:48 PM CDT    WebSphere Business Integration Server Foundation, Version 5.0.2
http://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp?topic=/com.ibm.wasee.doc/info/ee/workflow/tasks/t5tuneint.html

Library | Support | Terms of Use | Feedback