The batch streaming infrastructure implements a number of features which are typically required for batch programs. Using the batch streaming infrastructure means that these features do not need to be directly implemented in your batch program. The following are the main features of the batch streaming infrastructure:
- Commit point processing
- The transaction is committed after each chunk is processed. This stops the transaction size getting too large which can cause performance and/or database locking issues.
- Skipping processing for records with errors
- Any record which causes an error when being processed is automatically skipped. The transaction for the chunk is rolled back, the record in error is added to a skip list and the processing of the chunk restarts.
- Generic batch logging/reporting
- Logging of progress of the batch process should assist in locating any issues that might be found. The generic elements of the reporting of the work done by the batch program - chunks processed, any chunks unprocessed and any records skipped - are built-in. Additional reporting can be added, see Providing Meaningful Information using the Batch Summary.
- Re-startability
- In the situation where the chunker and/or stream processes crash or are killed. They can be restarted and processing will continue from the point where processing was stopped. This typically means that any chunks being processed at the time the stream(s) exited will be re-started, however there is potential under limited circumstances that one or more chunks could remain unprocessed in the event of a restart.