Where a sub-division of processing space or large amounts of extra processing (see Identifing any Required Extra Processing) are required, it may be beneficial to divide the processing between multiple batch programs rather than trying to complete all the processing in a single batch program. In particular the development of multiple batch programs may significantly simplify the design and development effort for each program as well as allowing the processing and choice of processing unit be optimized for each distinct task. There are two distinct options for developing multiple batch programs:
- Independent Batch Programs
- The advantage of this approach is that given each process has a distinct chunker and stream implementation more specialized approaches can be taken to improve performance etc. It's also important to note that, given suitable hardware and scheduling software, independent batch programs can be run at the same time - this can offer substantially better utilization of the batch window provided the processing for each program is independent. Alternatively, it is possible to create a single batch program that runs several distinct chunker processes in sequence without creating multiple batch programs, see Running Multiple Chunker Processes in a Single Instance.
- Single Batch Program using parameters
- This is not directly supported by the batch streaming infrastructure, but can be achieved by adding a (mandatory) parameter to the chunker identifying the subset of data to be processed and scheduling a run of the chunker for each sub-division. It's important to note that, barring a specialized implementation (see Running Multiple Instances of the Same Batch Program) multiple instances of a single batch program can not be run at the same time - the batch streaming infrastructure uses a program key to allow the chunker to be restarted in the event of a crash. Two instances cannot be run concurrently with the same key.