Data Movement Utilities Guide and Reference

AutoLoader Overview

The AutoLoader is a utility that can:

• Transfer data from one system (like MVS) to another system (like UNIX).
• Partition that data in parallel.
• Load the data simultaneously on the corresponding database partitions.

The AutoLoader can be run in one of four modes:

• SPLIT_AND_LOAD. Data is partitioned (perhaps in parallel) and loaded simultaneously on the corresponding database partitions.
• SPLIT_ONLY. Data is partitioned (perhaps in parallel) and the output is written to files in a specified location, or in the AutoLoader current working directory.
• LOAD_ONLY. Data is assumed to be already partitioned; the split process is skipped, and the data is loaded simultaneously on the corresponding database partitions.
• ANALYZE. An optimal partitioning map with even distribution across all database partitions is generated.

In a partitioned database, large amounts of data are located across many partitions. Partitioning keys are used to determine on which database partition each portion of the data resides. The data must be split before it can be loaded at the correct database partition. The AutoLoader utility can perform both operations (see Figure 7).

The AutoLoader utility uses a hashing algorithm to partition the data into as many output sockets as there are database partitions in the nodegroup in which the table was defined. It then loads from these output sockets concurrently across the set of database partitions in the nodegroup. A key feature of this utility is that it uses direct TCP/IP communication using sockets for all data transfer required during both split and load processes. It also allows the use of multiple database partitions for the splitting phase, thereby significantly improving performance.

Figure 7. AutoLoader Overview. In this example, source data is read by the AutoLoader, and half is sent to each of two partitioning agents, which partition the data and send it to one of three database partitions. The load utility at each partition loads the data.