Net.Data Programming Guide

Live Connection

Some servers can improve performance by using live connection to keep persistent database connections. Some Net.Data actions require a large start up time. For example, before a database query can be issued, the process must identify itself to the DBMS and connect to the database. This is often a significant portion of the processing time needed for Net.Data macros that access a database. Another example of an expensive start up is a Java virtual machine that is needed to run Java applications (but not Java applets). Because of the way CGI programs operate, these startup costs are paid on every request to the Web server.

Live connection can dramatically improve performance by eliminating this startup overhead. The savings come from continuously running one or more processes that perform the startup functions. These processes then wait to service requests. You can run live connection if you use Net.Data as a CGI program. If you use a Web server API, you must use live connection.

Live connection consists of a connection manager and cliettes. Cliettes are single-threaded processes that the connection manager starts and stay active while the server is running. Cliettes process data and communicate with Net.Data language environments in the initialization file with the key word CLIETTE. Each type of cliette is made to handle a specific backend function, such as the DB2 cliette, which connects to the database and sets up operations to perform SQL calls before any Net.Data macros are processed by Net.Data. The executable is named in the configuration file, dtwcm.cnf.

Live Connection Advantages

There are three main advantages to using live connection:

• Improved performance

  Reusing connections is more efficient than making new connections. You must determine how expensive connect time is compared to overall processing time. Generally, if you request small SQL statements (for example, simple queries on a database with fewer than 100,000 rows), or if your connect is difficult (for example, remote servers), then the connect time is significant.

• Multiple database access

  Live connection lets you have one Net.Data macro connect to multiple databases at the same time. This is possible because each database has unique cliettes, and therefore Net.Data simply communicates with multiple cliettes.

• Sequential display

  Connection manager has the ability to process SELECT statements and discard the first N rows. This gives the ability to use an application that gives next and previous options to navigate a large list. By setting the variable START_ROW_NUM to N you can break down large queries to show manageable chunks, letting application users click a NEXT button to retrieve the next set of rows.

Should I Use Live Connection?

You must use live connection if you choose to use an API (instead of CGI) to communicate with your database. You might benefit from live connection if your application has these requirements:

• Your application requires data from multiple databases.

• Live connection, used together with an API, improves performance for many systems, depending on their load and configuration, because new processes are not created when a request (from a URL) is sent to the Web server. However, using the connection manager requires additional system administration. You should experiment with your own system to determine the configuration that works best for you.

Many applications can improve performance without using live connection by using the ACTIVATE DATABASE or START DATABASE command to save time establishing database connections. See your database's documentation for details on the command your database uses. Also check your platform's documentation to see if there are additional steps to help improve performance.

Starting the Connection Manager

The connection manager is a separate executable from Net.Data and named dtwcm. You should start it when you start the Web server. When you start connection manager, it reads a configuration file and spawns a group of processes that process data. In each process the connection manager begins the execution of a particular cliette.

Once everything is configured and running (including the database, Web server, and connection manager) Net.Data processing typically involves these steps when live connection is enabled:

• The Web server is invoked, starting either a CGI process or an API thread to execute Net.Data.

• Net.Data starts processing the Net.Data macro.

• When Net.Data encounters a function call that uses live connection, it determines what type of cliette is needed from the initialization file. For DB2, the cliette type is often a name based on the DB2 database name. In this example, the cliette name is DTW_SQL:CELDIAL. Net.Data asks the connection manager for a cliette of that type. The connection manager looks for cliettes of that type that are available. If none are available, the connection manager puts the request on a queue and processes it when the right cliette type is available. When a cliette becomes available, the connection manager tells Net.Data how to communicate with the cliette.

• Net.Data asks the cliette to process the function.

• This process is repeated from step 3 until the Net.Data macro processing is complete.

• All cliettes are released.

If a cliette is specified in the initialization file, but the connection manager is not running, Net.Data loads the DLL and processes the macro. If you use an API, you are likely to receive errors and should start the connection manager.

Configuring Live Connection

Live connection uses a configuration file, dtwcm.cnf, to determine what cliettes need to be started. A sample configuration file is shown in this example, which contains these blocks:

• administrative information

• SQL cliette information for a DB2 connection

• Java application cliette information

The lines are numbered for reference:

 1  CONNECTION_MANAGER{
 2  MAIN_PORT=7100	
 3  ADMIN_PORT1=7101
 4  ADMIN_PORT2=7102
 5  }
 6 
 7  CLIETTE DTW_SQL:CELDIAL{
 8  MIN_PROCESS=1		
 9  MAX_PROCESS=5		
10  START_PRIVATE_PORT=7200		
11  START_PUBLIC_PORT=7210	
12  EXEC_NAME=./cltdb2
13  DATABASE=CELDIAL		
14  BINDFILE=/usr/... .../d2wsql.bnd
15  LOGIN=marshall		
16  PASSWORD=stlpwd		
17  }
18
19  CLIETTE DTW_APPLET{	
20  MIN_PROCESS=1	
21  MAX_PROCESS=5		
22  START_PRIVATE_PORT=7300	
23  START_PUBLIC_PORT=7310	
24  EXEC_NAME=./javaapp	
25  }

Lines 1 - 5 are required for the configuration file. Lines 7 - 12 and 17 are required for all SQL cliettes. You can include additional information, such as a user ID and password, if you connect to a DB2 database. These additional values are shown in lines 13 - 16. If you use Java applets, lines 19 - 25 are all required.

You can specify to use default values for the LOGIN and PASSWORD variables, which means that Net.Data uses the same user ID that started the connection manager to connect to the DB2 database. This lets you avoid placing this information is the configuration file. For example, replace lines 15 and 16 with these lines:

LOGIN=*USE_DEFAULT
PASSWORD=*USE_DEFAULT

You must determine which port numbers to use on your system. You also need to determine values for MIN_PROCESS and MAX_PROCESS. The number of processes specified by MIN_PROCESS are started when the connection manager is started. Afterwards, if simultaneous processes arrive, the connection manager starts more cliettes, adding one as needed, until the value specified for MAX is reached. The values you use can impact performance, but you can change them later.

To use live connection, you must include ENVIRONMENT statements for DTW_SQL (and possibly DTW_ODBC) in the Net.Data initialization file. See Net.Data Language Environment Guide for more information.

Before modifying your configuration file, be aware of these points:

• Cliette names are used by the connection manager to uniquely identify a set of cliettes.

• For database cliettes, you must have one named set of cliettes for each database you plan to access. For databases that are rarely accessed, you can set the MIN and MAX number of cliettes to 1. Alternatively, you can also set MIN to 0, which means processes are not spawned until a Net.Data request is made for the cliette.

• The NAME of the cliette must match the name in the initialization file. The name is at the end of the line with the DTW_SQL environment statement, which reads CLIETTE "xxx" where xxx is how the name is built in the Net.Data macro file. It can contain variables, and in the case of DB2 cliettes, it should include the variable $(DATABASE). The default is DTW_SQL:$(DATABASE). When an SQL section is encountered, $(DATABASE) is replaced with the current value of DATABASE. This is how you can access multiple databases. If you have three databases that you wanted to access in your Net.Data macro (for example, D1, D2, and D3), and your initialization file has the standard CLIETTE "DTW_SQL:$(DATABASE)" line, then you need three sections in the configuration file such as:

  CLIETTE DTW_SQL:D1{ ...} 
  CLIETTE DTW_SQL:D2{....} 
  CLIETTE DTW_SQL:D3{....}
  

• Processes are started but not stopped. If you set the MAX number of processes to M and at any time M processes are used simultaneously, they stay alive until you shut down the connection manager, so you do not want MAX to be so high that you use up all of your system resources starting processes that are rarely used. We suggest trying different values for MIN and MAX to see what works best for your system. If the connection manager receives more requests than the specified MAX, the last is put in a queue until a cliette finishes processing. When a cliette becomes available, the request is processed. This is transparent to the application user.

• Be careful to use port numbers that do not conflict. Each cliette uses 2 ports. When you specify a set of ports, you must specify which ports get used. The first two values are START_PUBLIC_PORT and START_PRIVATE_PORT. The other is MAX number of cliettes. The following example shows which ports get used.

  START_PUB_PORT=1000
  START_PRIV_PORT=1010
  MAX_NUM_PROC=5
  

  This example uses these ports:
  

  1000	1010
  1001	1011
  1002	1012
  1003	1013
  1004	1014

  A common error is to have two sets of cliettes overlap the ports they use. Check with your system administrator to ensure that the ports you plan to use are available. The readme file for your platform has general guidelines on what ports are valid for your operating system.

• You can use the same type of cliette for different named sections. For example, all DB2 database sections of the configuration file use the same cliette type. You cannot have two sections with the same name.

If you are using CGI, and want only some databases to use live connection, simply list the databases you want in the configuration file. When Net.Data is processing a Net.Data macro and encounters an SQL section, it asks the connection manager for a specific cliette. If the connection manager does not have that type of cliette it responds with a NO_CLIETTE_AVAIL message. Net.Data processes the request with a DLL version instead.