Overview
IBM WebSphere Application Server V7 Feature Pack for Modern Batch Overview
This module provides an overview of the IBM WebSphere® Application Server V7 Feature Pack for Modern Batch.
Batch processing
Batch processing Comparison of online transaction processing and batch processing A request / receive model (Often but not necessarily synchronous) Duration of processing relatively short Often transactional in nature Application server runtime enforces timeouts for workload Online Transaction Processing Batch Processing (or Long Running) Submit Response Work on Record A submit / work / result set model (Asynchronous to the submitter) Duration of processing a function of work to be done; can be hours/days Often transactional in nature Often multi-step processes Request Response
Online transaction processing is a request/receive model where the duration of the processing is relatively short, and the tasks are typically transactional in nature. In this model, the application server runtime enforces timeouts for the workload. Batch processing is a submit/work/result set model where the duration of the processing is a function of the tasks to be completed. In some cases, the tasks can require hours or even days to complete. In this model, the work tasks are typically transactional in nature and typically involve multi-step processes.
Batch modernization
Batch modernization Modernization does not mean wholesale replacement of existing batch function that serves the business needs. Business imperatives drive technical solutions. Providing greater flexibility / control of batch window processing Moving to a continuously available batch model intermixed with OLTP Using goal-oriented workload management to enable job completion within deadlines SOA transformation - Running batch as a service Hosting batch processes within SOA-capable application server environments Optimally reusing services through efficient co-location Leveraging Java™ skills for all applications Utilizing Java for batch processing and OLTP Reducing cost of running batch on z/OS® Moving batch processing to Java to take advantage of zAAP specialty engines Utilizing processor resources available when OLTP workloads are light
Batch modernization involves these techniques: -Providing greater flexibility and control of batch window processing by moving to a continuously available batch model intermixed with online transaction processing and using goal-oriented workload management to enable job completion within deadlines. -Hosting batch processes within service-oriented architecture (SOA) capable WebSphere Application Server environments and optimally reusing services through efficient co-location. -Leveraging Java skills for batch and online transaction processing tasks. -On the z/OS platform, reducing costs by moving batch processing to Java to take advantage of System z® Application Assist Processor (zAAP) specialty engines and utilizing processor resources available when online transaction processing workloads are light.
Use JVM launcher
Use JVM launcher JCL JVM Initialized JVM Terminated Java Batch Run Java Program Java Virtual Machine This approach works, but has limitations: High overhead associated with repeated cycling of JVMs Application must provide most batch processing services Many attempt to use a JVM launcher to run Java batch programs.
Java Virtual Machine (JVM) launcher technologies are easy to use and free, and the programming interfaces are very useful; however, a new JVM is created for every step, and the resulting overhead is significant. In addition, JVM launcher technologies do not provide a transaction manager. Ultimately, the application developer must author a transaction manager within their application. On the z/OS platform, the Feature Pack for Modern Batch and WebSphere Compute Grid complement the Java Batch Toolkit for z/OS (JZOS). The JZOS launcher can be used when only a few Java batch steps must be executed. Once the size of the Java batch infrastructure grows to the point where the overhead of creating and terminating JVMs is an issue, the Feature Pack for Modern Batch or WebSphere Compute Grid can be used. The second component of JZOS, it’s API’s for accessing z/OS resources, can be used from Modern Batch or Compute Grid applications.
Use application server runtime
Use application server runtime A second approach uses the application server runtime and authors the batch application as a message driven bean or an asynchronous bean. In both cases, the application developer must author custom middleware to support the batch processing. WebSphere Application Server JVM Batch Processing Code 1 2 Servlet or Session Bean Message Driven Bean Asynchronous Bean When a message driven bean is used, the application must manage the timeout on the message driven bean input The preferred technique (used by the Feature Pack for Modern Batch and WebSphere Compute Grid) is to use an asynchronous bean
A second approach uses the application server runtime and authors the batch application as message driven bean or an asynchronous bean. When a message driven bean is used, the application must manage the timeout on the message driven bean input; therefore, using an asynchronous bean is the preferred technique. However, in both cases, the application developer must author custom middleware to support the batch processing, given that the solution lacks these features: -A batch programming model, job definition language, and supporting development tools -Interweaving of online transaction processing tasks and batch tasks -Checkpointing and restarting of jobs -Integration with enterprise schedulers -Graphical and command line interfaces for job monitoring and management -Job usage accounting and activity logging
Java batch complexity
Java batch complexity JVM launcher WebSphere Application Server asynchronous beans Requires custom code Requires job control interface into JVM Requires workload management, classification, and throttling based on priorities Single-step Java program that accesses UNIX data Asynchronous processing inside application server Checkpoint and restart processing Pause and resume jobs within JVM Manage resource usage to meet deadlines
Initial Java batch programming projects typically begin with a relatively simple Java Virtual Machine (JVM) launcher design, but grow more complex over time as additional management features are required. These projects frequently progress in this manner: -An organization uses the Java command line interface or the Java Batch Toolkit for z/OS (JZOS) to run a single-step Java program that accesses UNIX® data. -After realizing that transaction management capabilities are required, the organization moves the application to a Java Enterprise Edition (Java EE) application server. -As the organization begins to process large numbers of records as a single batch job, it finds that it is no longer practical to restart entire jobs when failures occur in the middle of a job. The organization then introduces checkpointing and restart capability. -A large number of jobs in the batch environment compete for the same resources, and administrators need a way to manage the jobs and handle resource conflicts. The organization then builds a job control interface into the JVM that processes the batch applications. -As batch loads increase even further, it becomes impossible to manually handle resource conflicts; as a result, the organization now requires a batch job management system that can optimally distribute workloads based on priorities and deadlines.
Custom middleware versus core business
Custom middleware versus core business Disadvantages of custom batch middleware solutions Removes focus from core business objectives Expensive to build Very expensive to support May limit future progress Easy to end up with isolated islands of custom batch solutions outside a cohesive managed framework Your code that addresses the business batch requirements Your code that provides custom middleware functionality Vendor supplied and supported middleware functionality Business executives prefer a focus on the core business Business executives prefer to avoid custom middleware
Organizations are frequently tempted to solve tactical issues with custom middleware code; however, the middleware code requires extensive time to develop and maintain, which ultimately decreases an organization’s ability to develop custom code that addresses core business objectives.
IBM Java batch offerings
IBM Java batch offerings Pacing/Throttling Job Submission Parallel Job Processing WLM Classification Resource Usage Reporting Batch Application Quiesce/Update Job Schedules Auto-submit based on calendar System Resource Management Job Repository Checkpoint Processing Job Pause and Resume Job Console into JVM Job Control Within JVM Record Oriented Stream Handling JVM remains instantiated Job Scheduler Web, WS, IIOP Basic Data Access Job Start Mechanism z/OS SYSOUT MVS Datasets and DD Cards Conditional Multi-Step Simple JVM launchers such as Java CLI JZOS enhancements to the BPXBATCH model Feature Pack for Modern Batch Common Batch Container WebSphere Compute Grid z/OS Only
Java batch tasks can be managed in several different ways: -Simple Java Virtual Machine (JVM) launchers can manage single step batch jobs that require basic data access. -On the z/OS platform, the Java Batch Toolkit for z/OS (JZOS) enhances the BPXBATCH model by supporting conditional multi-step batch jobs with access to MVS datasets and use of Data Definition (DD) cards. -The Feature Pack for Modern Batch provides multi-step job support, a managed container for execution of batch jobs, a job control interface, job checkpoint and restart capability, and a batch application development framework. -Finally, WebSphere Compute Grid adds support for job repository and schedules, workload management, job usage reporting, batch application quiesce and update, parallel job support, and pacing and throttling of jobs.
Java batch foundation
Java batch foundation WebSphere Application Server Multiple JVM model Web and EJB Containers JDBC, JCA, JMS, MQ Security Management Transaction Management Thread Management Integration with WLM OS Platform Batch Execution Platform Solution Modern Batch Feature Pack, WebSphere Compute Grid Your batch business function Key benefits Custom middleware code is no longer required, so developers can stay focused on core business imperatives Online Transaction Programming (OLTP) and batch loads can be mixed while maintaining respective service level agreements
The basic IBM Java batch foundation includes WebSphere Application Server and the Feature Pack for Modern Batch. Customers demanding advanced workload management, accounting, and job scheduler integration capabilities can use WebSphere Application Server and WebSphere Compute Grid. Both solutions provide these benefits: -Custom middleware code is no longer required, so developers can stay focused on core business imperatives. -Online Transaction Programming (OLTP) and batch loads can be mixed while maintaining the service level agreements for each type of traffic.
Batch Container
Batch Container Application Server Function that dispatches the job to the Batch Container WebSphere Application Server Platform Native Code Java Virtual Machine IBM Batch Container Classes and API’s Batch Application Job Steps Data Access Checkpoints Results Batch Container Batch Controller Bean Batch Controller Bean controls the batch application and processes the job definition from start to finish The Batch Container is the heart of the Feature Pack for Modern Batch and WebSphere Compute Grid products.
The Batch Container is the heart of the Feature Pack for Modern Batch and Compute Grid offerings. It runs a batch job under the control of an asynchronous bean, which can be thought of as a container-managed thread. The batch container ultimately processes a job definition and carries out the life cycle of a job. The Batch Container provides these services: -Checkpointing, which involves resuming batch work from a selected position. -Result processing, which involves intercepting and processing step and job return codes. -Batch data stream management, which involves reading, positioning, and repositioning data streams to files, relational databases, native z/OS datasets, and many other different types of input and output resources.
Batch components and workflow
Batch components and workflow Application Server WebSphere Application Server Platform Native Code Java Virtual Machine Batch Scheduler Collection of servlets, JSPs and EJBs xJCL 1 2 3 4 5 6 Job is submitted Job Control Definition is specified Scheduler analyzes the request Job is dispatched to batch endpoint Batch endpoint begins execution Batch application is invoked
-Batch jobs are submitted to the system using the Job Management Console or programmatically by way of Enterprise Java Beans (EJB), Java Message Service (JMS), or web services. -Each job is submitted in the form of an XML Job Control Language (xJCL) document. -The Job Dispatcher selects the best endpoint for job execution based on several different metrics. -The endpoint sets up the jobs in the Batch Container and executes the batch steps based on the definitions in the xJCL. -The Job Dispatcher aggregates job logs and provides life-cycle management functions like start, stop, cancel, and so on.
Batch programming model
Batch programming model Job Control (xJCL) Batch Controller Bean Part of the Batch Container code supplied by IBM Batch Data Streams Provides data input and output services for the job steps Checkpoint Algorithms Service to programmatically determine and handle checkpointing Results and Return Codes Services to determine, manipulate and act upon return codes, both at the application and system level Job Step Control Invoking and coordinating processing between steps Batch Container Batch App POJO Step 1 Step 2 Step n Development Libraries RAD or Eclipse WebSphere Application Server Runtime Interfaces JDBC, JCA, Security, Transaction, Logging, Deployment, etc., etc.
The batch programming model consists of four principal interfaces, two of which are essential to building a batch application, and two which are optional and intended for advanced scenarios: -The batch job step defines the interaction between the batch container and the batch application. -The batch data stream abstracts a particular input source or output destination for a batch application and defines the interaction between the batch container and a concrete BatchDataStream implementation. -A checkpoint policy algorithm defines the interaction between the batch container and a custom checkpoint policy implementation. A checkpoint policy is used to determine when the batch container will checkpoint a running batch job to enable a restart to occur after a planned or unplanned interruption. The Feature Pack for Modern Batch and WebSphere Compute Grid include two ready-to-use checkpoint policies. -A results algorithm defines the interaction between the batch container and a custom results algorithm. The purpose of the results algorithm is to provide the overall return code for a job. The algorithm has visibility to the return codes from each of the job steps. The Feature Pack for Modern Batch and WebSphere Compute Grid include one ready-to-use results algorithm.
Job control definition - xJCL
Job control definition - xJCL
batch_controller_bean_jndi package.class input_stream Roughly analogous to the JOB card A job step Like the EXEC PGM= statement in JCL Similar to DD statements
Jobs are expressed using an Extensible Markup Language (XML) dialect called XML Job Control Language, or xJCL. The xJCL definition is very similar to the traditional JCL. The xJCL definition of a job is not part of the batch application, but is constructed separately and submitted to the job scheduler to run. The job definition identifies which batch application to run, and it’s inputs and outputs. It also identifies which checkpoint algorithms and results algorithms to use. The Job Scheduler uses information in the xJCL to determine where and when the job should be run.
Batch environment review
Batch environment review WebSphere Application Server Runtime System Platform Platform exploitation below the open standard specification line Batch Container Your batch applications deployed into the batch- enabled server or cluster Eclipse or RAD Batch Data Stream Development Framework Classes Application Server JVM xJCL Job Definition File Job Scheduler Job Console Browser Web Services EJB IIOP Scheduler dispatches to end points based on knowledge of environment
The Job Scheduler provides the job management functions such as submit, cancel, restart, and so on. It maintains a history of all job activity, including waiting jobs, and running jobs, and completed jobs. The job scheduler is hosted in a WebSphere Application Server or a server cluster. Jobs are described using job control language called XML Job Control Language (xJCL), which identifies the batch application to run, it’s inputs and outputs. The Batch Container provides the execution environment for batch jobs. There can be multiple Batch Containers in a WebSphere cell. Batch applications are regular WebSphere Java Enterprise Edition (Java EE) applications, deployed as Enterprise Archive (EAR) files.
Feature pack installation and configuration
Feature pack installation and configuration Install WebSphere Application Server V7.0 Install WebSphere Application Server Version 7.0 Fix Pack 11 or later and the Java SDK 1.6.0 SR7 Cumulative Fix for WebSphere Application Server Version 7.0 Fix Pack 11 or later Install the Feature Pack for Modern Batch Distributed platforms Download IBM Installation Manager from the WebSphere Application Server support site If the system cannot access the remote IBM product repositories or an appropriate internal repository server, download the local feature pack and import repositories Install IBM Installation Manager Define the location of the local repositories in IBM Installation Manager (if applicable) Import the WebSphere Application Server installation into IBM Installation Manager Use IBM Installation Manager to install the feature pack z/OS Download the feature pack PTF from the WebSphere Application Server Support site Use SMP/E to install the feature pack Augment your existing profiles to support the feature pack or create new profiles that include support for the feature pack Create a JDBC provider, data sources, and a database to support the Job Scheduler Configure the Job Scheduler and the grid endpoints Install, configure, and run the included sample applications to validate the setup and explore the capabilities of the feature pack
To use the Feature Pack for Modern Batch as a base for your Java batch environment, complete these steps: 1. Install WebSphere Application Server V7.0. 2. Install WebSphere Application Server Version 7.0 Fix Pack 11 or later and the Java SDK 1.6.0 SR7 Cumulative Fix for WebSphere Application Server Version 7.0 Fix Pack 11 or later. You must install both fix packs at the same level. 3. Install the Feature Pack for Modern Batch -On distributed platforms, complete these steps: -Download the appropriate version of IBM Installation Manager from the WebSphere Application Server support site. -If the system cannot access the remote IBM repositories or an appropriate internal repository server, download the local feature pack and WebSphere Application Server import repositories. -Install IBM Installation Manager. -Define the location of the local repositories in IBM Installation Manager (if applicable). -Import the WebSphere Application Server installation into IBM Installation Manager. -Use IBM Installation Manager to install the feature pack. -On z/OS, complete these steps: -Download the feature pack PTF from the WebSphere Application Server Support site. -Use SMP/E to install the feature pack. 4. Augment your existing profiles to support the feature pack or create new profiles that include support for the feature pack. 5. Create a JDBC provider, data sources, and a database to support the Job Scheduler. The feature pack includes DDL files for creating the database. 6. Configure the Job Scheduler and the grid endpoints. 7. Install, configure, and run the included sample applications to validate the setup and explore the capabilities of the feature pack. Complete setup information is included in the Feature Pack for Modern Batch information center.
Deployment options
Deployment options Common batch container, development tools, and operational commands to manage batch job life cycle ¦ ¦ ¦ Container managed checkpoint/restart capabilities ¦ ¦ ¦ Job management console ¦ ¦ ¦ Application execution platform ¦ ¦ ¦ Basic scheduler/job dispatcher ¦ ¦ ¦ System-managed job logs ¦ ¦ ¦ High-availability and clustering of Job Scheduler/Dispatcher ¦ ¦ Multi-site disaster recovery for batch platform ¦ ¦ Integration with workload management on z/OS ¦ ¦ Interoperability between Java and COBOL on z/OS ¦ Non-disruptive batch application update/endpoint quiesce ¦ Job usage accounting, including SMF integration on z/OS ¦ Job classes and workload classification ¦ Integrated Parallel Job Manager for job parallelization across multiple JVM’s ¦ Enterprise scheduler connectors ¦ Enterprise monitoring capabilities ¦ Disaster recovery with operational state transfer ¦ Integration with WebSphere Virtual Enterprise for goal-oriented job placement ¦ WebSphere Application Server Base with Modern Batch WebSphere Application Server Network Deployment or z/OS with Modern Batch WebSphere Application Server Network Deployment with Compute Grid
IBM offers several different Java batch environments: -WebSphere Application Server Base Edition with the Feature Pack for Modern Batch -WebSphere Application Server Network Deployment Edition with the Feature Pack for Modern Batch -WebSphere Application Server for z/OS with the Feature Pack for Modern Batch -WebSphere Compute Grid When the Feature Pack for Modern Batch is used in conjunction with WebSphere Application Server Base Edition, it provides these features: -A managed batch container -A batch application framework -A job management console -A job dispatcher -System-managed job logs When the Feature Pack for Modern Batch is used in conjunction with WebSphere Application Server Network Deployment Edition or WebSphere Application Server for z/OS, it provides these additional features: -High-availability and clustering of the job dispatcher -Multi-site disaster recovery support -Integration with z/OS workload management WebSphere Compute Grid provides these features beyond those provided by the Feature Pack for Modern Batch: -Job classes and workload classification -Job parallelization across multiple Java Virtual Machines -Job usage accounting -Connectors for integration with enterprise schedulers -Enterprise monitoring capabilities -Interoperability between Java and COBOL on z/OS -Non-disruptive batch application update support -Integration with WebSphere Virtual Enterprise for goal oriented job placement
Feedback
Feedback Your feedback is valuable You can help improve the quality of IBM Education Assistant content to better meet your needs by providing feedback. Did you find this module useful? Did it help you solve a problem or answer a question? Do you have suggestions for improvements? Click to send email feedback: mailto:iea@us.ibm.com?subject=Feedback_about_overview.ppt This module is also available in PDF format at: ../overview.pdf
You can help improve the quality of IBM Education Assistant content by providing feedback.
Trademarks