An Automation and High Availability Solution
for WebSphere Application Server for OS/390 and z/OS
Based on System Automation for OS/390
Contents
1.1 WebSphere Application Server: a High-level Overview
1.2 System Automation for OS/390: a High-level Overview
1.3 WebSphere Application Automation and High Availability
Chapter 2. The Environment for WebSphere Application
Server for OS/390 and z/OS
2.2.1 How WebSphere Application Server Uses z/OS
Components
z/OS Unix Systems Services (USS)
Resource Recovery Services (RRS)
2.2.2 The Structure of WebSphere Application Server for
OS/390 and z/OS
Application Server Environment
2.3 WebSphere Application Server in a Sysplex Environment
2.4 Options for WebSphere Application Server for OS/390
and z/OS
2.4.2 Using the IBM HTTP Server for z/OS
2.4.3 Using an HTTP Server for Distributed Platforms
Chapter 3. High Availability for WebSphere Application
Server for OS/390 and z/OS
3.1 WebSphere Application Server for OS/390 and z/OS
Administration
3.2 WebSphere Application Server for OS/390 and z/OS
Operations
3.3 Changing Configurations for WebSphere Application
Server for OS/390 and z/OS
3.3.1 Starting, Stopping, Monitoring Address Spaces
Using Identifiers, not Jobname
3.3.2 Conversation Activation and System Automation
Disable Automation and let Activation Restart Servers
Stop and Restart Servers using System Automation for
OS/390
4.3 Base WebSphere Application Server for OS/390 and z/OS
Setup
4.3.2 HTTP Session Related Definitions
Chapter 5. Setup for System Automation for OS/390
5.5.3 The WebSphere Base Environment
5.8.1 Activation using the SM Scripting API
Figure
1: The WebSphere software Platform
Figure 2: The WebSphere Application Server
for OS/390 and z/OS Environment
Figure 3: WebSphere Application Server for
OS/390 and z/OS Environment
Figure 4: Components of a J2EE Server
Figure 5: WebSphere Application Server V4.0.1 for z/OS and OS/390 Plug-in Environment
Figure 6: HTTP Transport Handler
Figure 7: WebSphere in a Sysplex
Figure 8: Persistent Sessions with IBM HTTP
Server
Figure 9: Session Affinity with a
distributed HTTP Server
Figure 10: WebSphere on a Sysplex: Sample Configuration
Figure 11: Groups and Applications in a Sysplex Environment
Figure 12: BBO_PLEX WebSphere Base Environment
Figure 13: BBO_J2EE Group for J2EE Servers
Running WebSphere Application Server for OS/390 and z/OS on a z/OS single system or in a sysplex environment increases the complexity of operating such a system. This is because WebSphere has a number of prerequisite products that it interfaces with, and also uses a number of z/OS address spaces itself to deliver high performance Web services.
When you run
Internet applications on a system, its availability is exposed to the public.
It is therefore essential to provide a scalable and highly available setup for
operations providing Internet services.
This paper
presents a system automation and high availability solution. This provides a
sample of how to set up a fully automated operational environment for all
WebSphere Application Server V4.0.1 for z/OS and OS/390 components and
prerequisites and related products running on a z/OS or OS/390 environment
using System Automation for OS/390 V2.2 (SA OS/390). IBM supports this set-up
by
·
providing this white paper as a
guideline
·
supplying
a sample Policy Database (PDB) over the Internet that contains the SA OS/390
definitions used to set up the environment described in the white paper. You
can use this PDB as a sample to set up SA OS/390 for your own
WebSphere Application Server for OS/390 and z/OS installation.
·
enhancing the message table by adding
messages issued by WebSphere and related products during the start up and
termination processing through APAR OA02375
· shipping a common routine that can be used to clean up server address spaces that WebSphere was unable to terminate during its end processing through the same APAR
WebSphere is Internet infrastructure software -
known as middleware. It enables companies to develop, deploy and integrate
next-generation e-business applications.
Figure 1 shows the WebSphere software platform.
|
|
Figure 1:
The WebSphere software Platform
The foundation of the platform is the IBM WebSphere Application Server, that provides specialized configurations designed to meet your most critical business needs.
WebSphere Application Server provides a rich,
e-business application deployment environment with a complete set of
application services. These include capabilities for transaction management,
security, clustering, performance, availability, connectivity and scalability.
WebSphere Application Server is a Java-based
Web application server, built on open standards, that helps you deploy and
manage Web applications. It is J2EE-compliant and provides a portable Web
deployment platform for Java components, XML and Web services, that can
interact with databases and provide dynamic Web content.
It on workstation platforms, including Windows®, Linux, AIX®, UNIX and z/OS operating systems.
IBM WebSphere® Application Server V4.0.1 for
z/OS® and OS/390® extends the role of S/390 and zSeries as premier servers for
enterprise e-business -- offering superior qualities of service, scalability,
performance, security, and availability. It offers:
For more details on WebSphere please refer to http://www7b.boulder.ibm.com/wsdd/zones/newcomers/
System Automation for OS/390
plays an important role in building the end-to-end automation of the IBM
autonomic computing initiative. The unique functions of SA OS/390 V2.2 can help
customers with single z/OS or OS/390 systems and Parallel Sysplex clusters to
ease management, may reduce costs, and increase availability. SA OS/390 is
designed to automate I/O, processor, and system operations and includes
"canned" automation for IMS, CICS, IBM Tivoli Workload Scheduler, and
DB2.
Working examples are provided
for:
· SAP R/3 High-Availability Automation
· High Availability Solution for WebSphere Application Server for z/OS and
OS/390.
For Parallel Sysplex
application automation SA OS/390 V2 introduced extremely powerful new concepts,
including sysplex-wide grouping of resources, relationships, and goals that
make Parallel Sysplex automation a reality. You can manage any number of
stand-alone systems and clusters from a single point-of-control, even from an
easy-to-use Java-based GUI. In addition, you can move applications and related
timers inside a sysplex.
SA OS/390, a NetView
application, integrates with
SA OS/390 primarily deals
with starting and stopping applications in accordance with their
inter-relationships. These include relationships of applications to other
applications, or being a component application of an application complex. SA
OS/390 also supports the permanent availability of an application by moving it
to another system if there is an unrecoverable failure.
All applications and systems
that you may want to include in automation must be defined to SA OS/390 in an automation Policy Database. This database contains the
objects to be managed by SA OS/390, and the rules according to that automation
of these objects proceeds.
For more details on System Automation for OS/390 please refer to
http://www-1.ibm.com/servers/eserver/zseries/software/sa/
Running WebSphere Application Server for OS/390 and z/OS on a z/OS single system or in a sysplex environment increases the complexity of operating such a system. This is because WebSphere has a number of prerequisite products that it interfaces with, and also uses a number of z/OS address spaces itself to deliver high performance Web services.
When you run Internet applications on a system, its availability is exposed to the public. It is therefore essential to provide a highly available setup for operations of this type. The aspects of high availability in general include:
In a monoplex environment your options to react
to outages are limited because you do not have spare images that you could use
for back up in case of failures. But you can still use SA OS/390 to automate an
environment with WebSphere Application Server for OS/390 and z/OS focusing on:
In addition to the monoplex advantages, SA
OS/390 can increase availability in a sysplex environment by
It is currently necessary to shut down and
restart a J2EE server when you change its configuration. WebSphere Application
Server will automatically shut down and restart all running J2EE servers with a
changed configuration when the conversation containing these changes is
activated.
This prevents a set up that fully supports
continuous operations even if you run in a sysplex environment.
With SA OS/390 enabled for WebSphere Application Server you can minimize downtimes. It offers facilities to allow that maintenance for WebSphere Application Server for OS/390 and z/OS can safely be performed by terminating WebSphere Application servers and making them available again as quickly as possible. This is discussed in more detail in 2.1 “Conversation Activation and System Automation” on page 30.
In this chapter we give the software prerequisites for WebSphere Application Server for OS/390 and z/OS. We also give an overview of the dependencies between these z/OS components and products, and the WebSphere Application Server. We tell you how the WebSphere Application Server uses these components. The parts that make up the WebSphere Application Server for OS/390 and z/OS itself are shown, and their functions are explained.
Base
products for WebSphere
Application Server for OS/390 and z/OS automation are, of course, WebSphere Application Server for OS/390 and z/OS V4.0.1 and System
Automation for OS/390 V2.2. In section 2.1.1 Mandatory
Requisites, we list the products that are required to run the
base products. In 0 Functional
Requisites on page 6 we list products that are not strictly required but that
we used in the set-up we describe in this paper.
A mandatory requisite is defined as a product
that is required without exception; the presented solution will not install or
will not function unless these requisites are met.
A functional requisite is defined as a product
that is not required for the successful installation of this solution,
but is needed at run time for a specific function to work.
Note: Some PTFs
have to be applied to different software components. For details please check
the Program Directories of WebSphere Application Server V4.0.1 for z/OS and
OS/390 and Tivoli System Automation for OS/390 Version 2 Release 2.
The following products are required:
and their prerequisite
operating system:
or
In addition the following products or features
are required:
The list includes prerequisites for System Automation for OS/390 and for WebSphere Application Server for
OS/390 and z/OS.
Figure 2: The WebSphere Application Server for OS/390 and z/OS Environment” on page 13 shows the main software components required to run a WebSphere Application Server for OS/390 and z/OS environment.
It also shows the sequence in that these components parts should be started in a z/OS image.
The use of the following z/OS components by the
WebSphere Application Server provides a stable platform, with built-in recovery
and security:
· TCP/IP as the communications vehicle
· Unix Systems Services (USS) platform support with HFS for file support
· LDAP and DB2 for directory services and database support
· WLM for work load management
· RACF for security
· RRS for recovery
Note
that WebSphere
Application Server for OS/390 and z/OS consists itself of a base
environment that provides the infrastructure for the J2EE and MOFW servers that
control and run J2EE web applications or CORBA applications.
Figure 2: The WebSphere Application Server for OS/390 and z/OS Environment
This section provides some details of how the WebSphere Application Server for OS/390 and z/OS uses functions offered by z/OS components.
z/OS Unix
Systems Services provides services needed to run the WebSphere Application
Server environment and the Java applications that run within it. It also
provides the Hierarchical File System used by the WebSphere
Application Server.
HFS is used to store Java code, static files,
and the configuration files needed to run the application servers. In a sysplex
configuration all the systems that run WebSphere need shared access to the
WebSphere configuration HFS. The recommendation is to use shared HFS.
The Workload Manager functionality is provided
by the z/OS operating system. It gives increased flexibility to WebSphere when
running on the zSeries platform.
With workload management, you define
performance goals and assign a business importance to each goal. The system
decides how much resource, such as CPU and storage, should be allocated to meet
the goal. Workload Manager will constantly monitor the system and adapt
processing to meet the goals that you set.
WebSphere requires that WLM is running in goal
mode and that Application Environments are defined for its servers. To balance
work on the system WebSphere uses the following services of the Workload
Manager:
· Queuing and address space management: to
dispatch work requests from the WebSphere Application Server for z/OS Control
Region to one or more WebSphere Application Server for z/OS
· Prioritizing work to meet
performance goals: WebSphere delegates the responsibility for
starting and stopping Server Regions to the WLM.
In
addition we used WLM in connection with the Sysplex Distributor.
· Providing utilization information: to enable the Sysplex Distributor to route incoming service requests to the most suitable system.
WebSphere security is based on RACF (or any
other external security server implementing the SAF interface).
All entities or principals, internally or
externally, have to be authenticated implicitly or explicitly. This includes Control
and Server Regions running under WebSphere. Therefore a number of IDs have to
be defined in RACF and many authorizations in several RACF classes have to be
granted (such as association between started task procedures and user IDs via
the STARTED class, access to servers via the CBIND class, access to DB2 via the
DSNR class etc.).
For WebSphere running in a sysplex you should
ensure that the RACF database is shared between the systems in the sysplex,
because these permissions must be consistent across the sysplex.
The z/OS Lightweight Directory Access Protocol
(LDAP) server is part of the SecureWay Security
Server for z/OS.
LDAP directory services provide an easy way to store,
update, and retrieve directory information in a central location for the Java
Naming and Directory Interface (JNDI) and CORBA naming and interface repository
services. The contents of the directory are stored in a DB2 table.
LDAP is used by both the “system servers” -- Systems
Management Server (SMS) and Interface Repository (IR) -- and the J2EE servers,
but in different ways. SMS and IR regions include DLL code to directly access
naming information in the LDAP database. J2EE servers access the LDAP database
through the LDAP server with the Java Naming Directory Interface (JNDI).
WebSphere V4 requires the use of DB2 UDB for
OS/390 and z/OS V7.1 or later for the following purposes:
· WebSphere configuration and session control information is held in DB2 databases.
The WebSphere control information consists of about 60 tables in two databases:
BBOMDB01, BBOJDB01
· DB2 is the default database for the LDAP server
· DB2 is used to store persistent session data
In addition DB2 user data can be accessed by
Java applications through JDBC.
For WebSphere running in a sysplex you should use
a shared DB2 to make the data consistent across the sysplex.
Resource Recovery Services (RRS) consists of
the protocols and program interfaces that allow an application program to make
consistent changes to multiple protected resources.
For CORBA applications the Object Transaction
Service (OTS) manages transactions and provides the interface to z/OS RRS, that coordinates resource recovery across several
Resource Managers. OTS requires RRS.
The IMS, CICS and DB2 Resource Managers have
also implemented an interface to RRS so that, in the case of a WebSphere client
application, resource coordination between the various Resource Managers can be
driven by RRS.
TCP/IP is part of the z/OS Communications Server.
The TCP/IP protocol is used as the basis for
communication between the clients and the HTTP server. It is also used to
communicate between other portions of the WebSphere Application Server
environment, such the System Management Server and the System Management User
Interface.
We make use of the Virtual IP Addressing (VIPA)
and Sysplex Distributor functionality offered by TCP/IP to achieve high
availability and to utilize z/OS workload balancing.
With VIPA, you define a virtual IP address that
does not correspond to any physical attachment or interface. Communication
Server for z/OS IP then takes care that the VIPA address appears to the IP network
as if it was a separate sub-network. A client selecting the VIPA address to
contact its server will have packets routed to the VIPA via any one of the
available real host interfaces.
Communication Server extends the VIPA concept
to allow for the recovery of failed system images or entire TCP/IP stacks:
· The automatic VIPA takeover function allows you to define the same VIPA address
on multiple TCP/IP stacks in a sysplex. One stack is defined as the primary or
owning stack and the others are defined as secondary or backup stacks for the
VIPA. Only the primary one is made known to the IP network. If the owning stack
fails, then one of the secondary stacks takes its place and assumes ownership
of the VIPA In this case, applications associated with these DVIPAs are active on the backup systems, thereby providing
a hot standby for the services.
· Dynamic VIPA (for an application instance) allows an application to
register to the TCP/IP stack with its own VIPA address. This lets the
application server move around the sysplex images without affecting the clients.
In this way, the application can dynamically activate the VIPA on the system
image it wishes to host the application. Because the application instance is
only active on one image in the sysplex at a time, the other images provide a cold
standby of the service.
These VIPA enhancements are enabled by the use
of XCF to communicate between the TCP/IP stacks.
Sysplex Distributor is the strategic IBM
solution for connection workload balancing in the S/390 or zSeries sysplex.
The concept is that a cluster of server
instances is represented by a single IP address. Because
Sysplex Distributor is built on Dynamic VIPAs, this
is called a Distributed VIPA (DVIPA). Basically, it is defined on a primary
TCP/IP stack and on backup stacks as appropriate. In addition, the primary
stack contains a configuration statement that identifies the DVIPA as
Distributed, the ports that an application will use (between one and four ports
per Distributed DVIPA), and the target application hosting TCP/IP stacks in the
sysplex.
“All present and future stacks” may be
specified as the target application hosting stacks, or targets may be limited to
selected specific TCP/IP stacks. This information is distributed automatically
to all candidate target TCP/IP stacks, and, on all candidate target stacks, the
same IP address is activated as a “hidden” DVIPA. It is hidden in the sense
that the target-application hosting stacks do not advertise its presence via
their dynamic routing daemons, so its presence on those stacks is not known to
the outside world.
The Sysplex Distributor routing stack
advertises ownership of the DVIPA to the world, and waits for connection
requests to be sent to it.
The routing stack will not send a TCP
connection to a target application host until there is an actual application
ready to receive work. Because the application-hosting target stacks are aware
of the port numbers associated with a Distributed DVIPA, the target stack
notifies the routing stack when an application is bound to one of these ports,
and accepts connections to the Distributed DVIPA. The routing stack will send
future work (TCP connections) for that application (port) to that target stack.
If the server should close its listening socket, or end abnormally, the hosting
target stack is immediately aware of this, and notifies the routing stack.
Should a target stack suffer an outage,
connections to any server instances on that target stack are of course lost,
but other server instances are still available for the affected clients when
they attempt to reconnect to the server via the Distributed DVIPA. The routing
stack will immediately send a connection reset and the reconnection request
will go to one of the remaining functioning instances. New connection requests
will not be routed to
a failed target stack or
LPAR.
To cover the case of a failure of the routing
stack itself the Distributed DVIPA should be configured with backup routing
stacks. If the routing stack should suffer an outage, the backup routing stack
will take over global responsibility for the Distributed DVIPA, and each target
stack will be aware of the takeover. This is all handled by the collaborating Sysplex
Distributor TCP/IP stacks, and client and server applications themselves are entirely
unaware of the failure of the former routing stack.
The HTTP Server is primarily used to serve
static pages. The HTTP Server allows you to provide simple Web serving and an e-business
presence on the Internet, but the technology is limited.
The HTTP Server also provides an environment
for the WebSphere Application Server Plugin. This plugin environment expands
your ability to do business on the Internet. For details see the discussion of
the WebSphere Plugin environment on page 21.
In this section, we look first at WebSphere Application Server for OS/390 and z/OS in a monoplex environment. For sysplex considerations
see section 2.3 “WebSphere Application Server in a Sysplex
Environment”on page 23.
The WebSphere Application Server for
OS/390 and z/OS environment consists of a base environment that provides
support for application servers, and the application servers themselves.
Figure 3 shows the structure of a WebSphere
Application Server for OS/390 and z/OS environment:
Figure 3: WebSphere Application Server for OS/390 and z/OS Environment
Each of
the subcomponents in the above figure is a separate address space.
The base environment consists of the Daemon,
Systems Management (SMS), Naming and Interface Repository (IR) servers.
The Daemon brings up the WebSphere Application
Server environment. It exists as a control region only within a single z/OS
address space. It is started via a procedure. It starts the other control
regions of the base environment (Systems Management, Naming and IR).
The purpose of the SMS is to allow users to add
new applications and control the configuration of the application server environment.
This is accomplished by connecting a Systems Management Extended User Interface
(SMEUI) console to the SMS server via a TCP/IP connection.
The SMS consists of a Control Region and
multiple Server Regions, each in a separate address space that can be started
by WLM if needed.
The Naming server is used to locate Java
objects, such as EJBs, and works in conjunction with the IR server. This provides
the operational support for both the J2EE and CORBA (MOFW) servers. It allows applications,
EJBs, and CORBA Business Objects to locate and communicate with each other.
Like the SMS, the Naming Server consists of a
control region and multiple server regions that can be started by WLM if
needed.
The IR server manages the inventory of CORBA
business object interfaces. The IR server also consists of a control region and
multiple server regions. It is required to run on the bootstrap system in a sysplex
(the first system where you install WebSphere).
During the installation of the WebSphere
Application Server Environment you also install the SMEUI application on a PC. This
graphical user interface allows you to view and update the current
configuration of your application servers.
There are two types of application servers that
run in this environment: the J2EE application server and the MOFW (CORBA)
server.
J2EE application servers are based on the Java
language only and allow you to run J2EE components, such as servlets or EJBs. The
CORBA application server is used for serving CORBA Business Objects that can be
written in a number of different languages for use on different platforms. Both
of these technologies can talk to each other in a WebSphere
Application Server for OS/390 and z/OS environment.
The J2EE server of the WebSphere Application
Server for z/OS and OS/390 provides a highly available, secure, reliable, and
scalable run-time environment for Java 2 Enterprise Edition (J2EE)
applications.
A J2EE server consists of one Control Region
and one or more Server Regions. Additional Server Regions are started by WLM if
required by the J2EE application workload.
The J2EE Server supports both Enterprise
JavaBeans and Web components such as JSPs and Servlets that conform to the J2EE
specifications and packaging standards published by Sun Microsystems.
These two types of J2EE application components
run on a WebSphere for z/OS J2EE server, and can use
· the application programming interfaces (APIs) and services that the Java
2 Standard Edition (J2SE) Software Development Kit (SDK) V1.3 provides, and
· enterprise
services such as Java Database Connectivity (JDBC), Java Naming and Directory
Interface (JNDI), and the Java Transaction Service (JTS) and API (JTA).
The J2EE specifications dictate that APIs and
services each type of application component may use, and the environment in that
they must run. Although both Enterprise beans and Web applications may run in a
single WebSphere for z/OS J2EE server, each component actually runs in a separate
type of container within the J2EE server.
Figure 4: Components of a J2EE Server
WebSphere for z/OS also provides a Managed
Object Framework (MOFW) Server that provides a run-time environment for CORBA-compliant
components.
We did not implement an MOFW Server in our
installation and will therefore not discuss it further.
The WebSphere Plugin is used primarily to route
servlet and JSP requests to the WebSphere Application Server. It is shipped WebSphere Application Server V4.0.1
for z/OS and OS/390, but it is code that runs within the IBM HTTP Server.
Starting with WebSphere 4.0.1 you have the choice
of whether you want to run your servlets in the WebSphere Plugin (locally), or to
run them remotely in a Web container residing in the same address space as your
EJBs.
The ability to run servlets
locally provides you with a means to migrate to WebSphere 4.0.1 using your existing
environment with minimum impact. The ability to run servlets remotely in a Web
container allows you to minimize the overhead when talking to EJBs. You can
keep all the pieces of your application within a single z/OS address space.
This is the only option we consider in the rest of this document.
Figure
5 shows the setup involved with this configuration.
Figure 5: WebSphere Application Server V4.0.1 for z/OS and OS/390 Plug-in Environment
Based on the context root of an HTTP request coming from a browser the request is passed from the IBM HTTP server to the WebSphere Application Server V4 Plugin. The plugin routes the request to the J2EE server’s Web container where static or dynamic content can be composed and passed back in HTML format to the HTTP server via the plugin. If the request invokes a servlet that in turn invokes EJBs these are running in the J2EE Servers EJB container.
The HTTP Transport Handler runs as part of the J2EE Server’s Control region. This component can handle HTTP(S) requests from clients.
Figure 6: HTTP Transport Handler
The protocol handler can speed up HTTP processing significantly, although, because it is not designed to deal with browser clients, it cannot handle the differences between browsers. It assumes that there is an HTTP server in front of it. In our case we included the IBM HTTP Server for z/OS in our setup and therefore do not cover in the rest of this paper the case where the Transport Handler is used to communicate with client browsers directly or through non-z/OS HTTP servers.
In a sysplex environment some (but not necessarily all) z/OS images can contain WebSphere Application Server for OS/390 and z/OS components. On each individual image we speak of a server instance, a server is made up of all the server instances on the z/OS images. Thus we have a WebSphere Application Server on the sysplex made up of several WebSphere Application Server instances. The WebSphere Application Server consists of a Daemon Server with Daemon Server instances, a Naming Server with Naming Server instances, an IR server with IR Server instances. Similarly J2EE and MOFW servers are made up out of J2EE and MOFW server instances.
Figure 7: WebSphere
in a Sysplex
A
sysplex cluster is configured into the WebSphere for z/OS namespace as a host
and is represented by a single Daemon IP Name. Because of this, systems and
applications outside the sysplex treat the sysplex as a single host. Functions
in WebSphere for z/OS route work through the sysplex according to the
availability of server instances and to workload balancing rules, in
cooperation with subsystems in z/OS or OS/390, such as TCP/IP, the domain name
server (DNS), and workload management.
On each image on that a J2EE or MOFW server
instance runs an instance of the Daemon and of System Management is required.
Naming instances and IR instances are not strictly required on each image of a sysplex
where you want to run J2EE or MOFW servers, but it is recommended to have them
running on all images.
Of course multiple J2EE Servers or MOFW servers
can run under the control of one WebSphere Application Server on a sysplex.
There are various options for setting up your environment for WebSphere Application Server for OS/390 and z/OS. One of the first choices is whether you include an HTTP server in your z/OS setup, or use an external HTTP server (on Linux, Unix or Windows platforms). Because this choice impacts how you can implement HTTP sessions, we will discuss the options you have for HTTP sessions first.
.
When users dynamically collect data as they
move through a Web application, the application needs a mechanism to hold the
user's state information over a period of time. However, HTTP treats each user
request as a discrete, independent interaction.
The Java servlet specification provides a
mechanism, known as a session, for servlet applications to maintain state
information. This mechanism allows a Web application developer to maintain all
user state information at the host, while passing minimal information back to
the user via cookies or URL rewrite. The state information associated with a
series of client requests is represented as an HTTP session object and
identified by a session ID. The session manager module that is part of each
servlet engine is responsible for managing HTTP sessions, providing storage for
session data, allocating session IDs, and tracking the session ID associated
with each client request, through the use of cookies or URL rewriting techniques.
There are two ways that this can be
implemented: session affinity or persistent sessions.
With session affinity the session information is held just in the memory of the servlet engine that started the session.
Session affinity means that when an application starts a session the session information is stored in a session object that resides in the specific JVM where the session is started. All further requests belonging to this session also need to run in this specific JVM. You must ensure that all requests during this session reach this specific servlet engine.
With persistent sessions, the session
information is stored in a shared database, and each servlet engine has access
to it.
Besides storing session objects in memory when
you enable persistent session management
WebSphere places session objects in a predefined
database. All information stored in a persistent session database must be
serialized. As a result, all the objects held by a session must implement java.io.Serializable in this case.
When
you use the IBM HTTP server for z/OS, you also use the WebSphere
Application Server for z/OS plugin to route requests from the HTTP server to
the application server.
This has the advantage that you have the entire
infrastructure required to run your Web applications on z/OS where you can
control and automate it more easily. In addition the workload can be
distributed across all z/OS images using the information available to WLM to
choose the optimal system.
There is, however, one disadvantage: The WebSphere
Application Server for z/OS plugin does not currently support session affinity.
You therefore have to implement persistent sessions in this setup.
Figure 8 gives an overview of the flow of requests from
browser to Web application in such a setup.
Figure 8: Persistent Sessions with IBM HTTP Server
As you can see, subsequent requests can end up being executed in any server region belonging to the J2EE Server. Since session data is stored in a shared DB2 database, it is accessible from any J2EE server region. You can also see that the workload is distributed at a HTTP request and an Application Server instance level. In both cases we can make use of the information available to WLM about sysplex load and distribute accordingly.
If you use IBM HTTP Server (IHS) based on the
Apache HTTP server on distributed platforms, you can use the WebSphere Advanced
Edition Web server plugin available for the respective distributed platform. This
plugin can connect directly to the HTTP Transport Handler of an application server
on WebSphere Application Server for z/OS.
The main value of using the WebSphere Advanced
Edition Web server plugin is that it can honor session affinity to a specific
application server control region on z/OS. Within the correct control region
the WebSphere Application Server honors session affinity to route the request to the
right server region.
Figure
9 shows the request flow when you implement session
affinity.
Figure 9: Session Affinity with a distributed HTTP Server
Normally it is not sufficient to move just the
HTTP Server to a non-z/OS platform. To implement a high availability solution
and enable scalability and workload distribution, you will also need a product
such as the Network Dispatcher on the distributed platforms.
Note that the Network Dispatcher does not
directly have access to information about the workload on your sysplex. It can only
distribute to various HTTP Servers based only on HTTP workload. You should also
note that workload distribution with the sysplex can only be done for the first
request within an HTTP session. From then on the z/OS image with the sysplex
and the J2EE server region must remain bound to the session, and can not be
changed to better distribute system utilization.
In
this chapter we briefly discuss the basic high availability considerations for WebSphere Application Server for
OS/390 and z/OS and the role System
Automation for OS/390 can play to support it.
It is necessary to perform administration
actions for a WebSphere Application Server for OS/390 and
z/OS when you want to:
· add new application servers or resources
· change parameters for the existing
environment, that is manage environment variables
· install new
applications into an application server after the applications were assembled
and deployed.
WebSphere Application Server for
OS/390 and z/OS keeps the complete setup information of its
configuration in so called conversations.
After initial installation and customization,
WebSphere for z/OS manages environment data through the Administration
application and writes the environmental data into the system management
database. When you activate a conversation, the data is also written to HFS
files.
To install an application on WebSphere for
z/OS, you must also use the WebSphere for z/OS Administration application. During
the application installation process, deployment descriptors are customized for
the WebSphere for z/OS environment, and application components
are loaded into the WebSphere for z/OS application server.
There are two interfaces to administer a WebSphere Application Server for OS/390 and z/OS:
1. The
WebSphere for z/OS Administration application that runs on your Windows workstation
and communicates with the System Management server, that is the ‘System Management
Enhanced User Interface’ (SMEUI)
2. The SM Scripting API that provides the same functions as the SMEUI. REXX is currently the only script language that is supported by the SM Scripting API.
WebSphere Application Server for OS/390 and z/OS shuts down and restarts all J2EE servers with changed configuration when the conversation containing these changes is activated. The impact this has on availability and how this needs to be reflected with respect to System Automation for OS/390 is discussed in 3.3 “Changing Configurations for WebSphere Application Server for OS/390 and z/OS” on page 28.
Through WebSphere for z/OS operations you can
manage WebSphere for z/OS servers and server instances. This includes actions
to:
· display the status of all server instances
· stop application servers and server instances
· cancel application servers and server instances
· restart
servers and server instances.
WebSphere for z/OS operations can be performed
from:
· a z/OS or OS/390 MVS console
· the Systems Management Enhanced User Interface on Windows
· TSO or RRS panels (for some operations)
If
you use SA OS/390 to automate WebSphere, it will perform these tasks for you.
There is normally no need , for example, for you to
start, stop or restart servers. When, in exceptional cases, you have to
initiate any of these tasks manually, you should use SA OS/390 facilities to do
so. You may get confusing results if, for example, you use the SMEUI to stop or
start applications servers that are controlled by SA OS/390.
When you want to change parameters for a J2EE Server or install applications on a J2EE server, you have to do this by defining the changes in a new conversation through the SMEUI or the Administration API and, finally, activate the conversation. For your changes to be reflected in the running servers, they need to be stopped and restarted. This is done automatically during the activation step for a conversation. If a server cannot be stopped during activation, activation fails.
When System Automation for OS/390 controls WebSphere operations, it is, as a rule, responsible for monitoring servers and restarting them if they end normally or abnormally. This may lead to conflicts with the activation process.
There are two options for avoiding a conflict:
1. To let activation restart servers, perform the following steps
a) Switch off automation for all J2EE servers
b)
Activate the conversation; all
running servers will be stopped and restarted through activation
c) Switch on automation after all servers have restarted
2. To
use SA OS/390 to restart servers perform the following steps:
a)
Before you activate a conversation
stop all J2EE Servers through SA /390
b)
Activate the conversation; no
server restart will be triggered through activation since no servers are
running
c)
After activation is complete
restart all J2EE servers through SA OS/390
The advantages and disadvantages of each approach are discussed below in 3.3.2 “Conversation Activation and System Automation” on page 30. First we outline the operational alternatives and their consequences on the SA OS/390 definitions.
Four items play a role when you start, stop and
monitor WebSphere control and server regions in sysplex:
1. the member name --
this is the name of
the library member containing the catalogued procedure that is used to start
the server. This is a required parameter when you start a task.
2. the jobname --
this is the name that will be assigned to the job. This is an optional parameter when you start a task. If the source JCL in the member is a procedure and you omit the JOBNAME keyword, the member name will be assigned as the job name.
3. the identifier used in the start command --
this is optional; it is not used as a replacement for the jobname. You can not specify both jobname and identifier. If the identifier is specified, it is used in the DISPLAY, MODIFY, RESET, CANCEL, FORCE, and STOP commands for the started tasks
4. the server name --
this is a WebSphere Application Server for
OS/390 and z/OS specific parameter that tells the WebSphere
which server instance is to be started.
In the WebSphere manuals it is recommended that
you use the server name also as an identifier, and start the control regions
with commands in the form
S ControlRegionProcName.server-instance,SRVNAME=’ server-instance ’
For example to start
the daemon on the first system, issue
S BBODMN.DAEMON01,SRVNAME=’DAEMON01’
To start it on the second system, issue
S BBODMN.DAEMON02,SRVNAME=’DAEMON02’
To start an application server, issue
S BBOASR2.BBOASR2A,SRVNAME=’BBOASR2A'
Stop commands that correspond to these start
commands are
P ControlRegionProcName.server-instance or simply P
server-instance
Cancel commands are analogous. The jobname in
this case is the ControlRegionProcName.
Since SA OS/390 requires unique jobnames for the
resources that it manages, you cannot start two servers on the same system
using the same procedure but different server-instance names. You could, for
example, attempt to start two application servers (APPSRV1 and APPSRV2) using
the same procedure (J2EEPROC) within the same system by issuing
S J2EEPROC.APPSRV1,SRVNAME=’APPSRV1’ and
S J2EEPROC.APPSRV1,SRVNAME=’APPSRV1’
You would not, however, be able to automate
these two servers with SA OS/390.
WebSphere Application Server for
OS/390 and z/OS Administration uses this form of commands when it
starts and stops servers.
In the publication “MVS System Commands” IBM
recommends that you use the JOBNAME parameter rather than an identifier. If you
use the JOBNAME parameter, SMF records, messages, and automated programs can
reflect or react to job status; identifiers can only be viewed at a console.
With the JOBNAME parameter you start the
control regions with commands in the form
For example to start
the daemon on the first system, issue
To start it on the second system, issue
To start an application server, issue
Stop commands that correspond to these start
commands use the jobname (that we set to the name of the server-instance):
P
server-instance
Cancel command are analogous.The
jobname in this case is the name of the server-instance.
SA OS/390 can automatically create start, stop
and cancel commands from the information it has or gets when a server is
started or ends normally or abnormally.
As mentioned in 3.2 “WebSphere Application Server for OS/390 and z/OS Administration” on page 27, WebSphere Application Server for OS/390 and z/OS Administration shuts down and restarts all J2EE servers for which the configuration has been changed when the conversation containing these changes is activated.
When
you automate WebSphere operations one way of achieving continuous availability
is to automatically restart WebSphere components as fast as possible if they
end normally or abnormally. Stopping severs that are controlled by an automation
product will usually cause the automation product to immediately restart the
servers.
If
the automatic restart ends before the attempt by WebSphere Application Server
Administration to restart, then this may prevent WebSphere Administration from
considering activation to have successfully finished. (You may see the
message: "Activate failed because
server cannot be stopped." and the conversation state will not have
changed to activated.)
If, however, the restart issued by WebSphere Application Server Administration is successful before the one initiated by the automation product the automatic restart will fail. This causes automation to get confusing messages: The ones coming from Administration’s restart would say that the servers are up; the ones coming from automatic restart would say that restart failed. System automation would in such a case not set the actual state of the respective server to AVAILABLE and thus indicate that manual intervention is required.
To avoid this you can either
· temporarily disable automation and let activation restart servers or
· use SA
OS/390 to stop servers before activation and restart them after activation has
finished.
In both cases there is a maintenance interval period
during that not all WebSphere Application Server for
OS/390 and z/OS services are available.
Operations during such a maintenance interval
can be automated. System Automation for OS/390
allows you to define a service period. Thus it provides the ability to make stop
and start requests at specified points in time independently of human
intervention. An operator can temporarily modify a service period if manual
control is desired. The activities to be performed during this interval can be initiated
manually or you can use OPC Automation.
OPC is now called Tivoli Workload Scheduler
(TWS). OPC Automation is an extension of SA OS/390 that capitalizes on the
strengths of NetView, SA OS/390, and TWSC by providing the ability to greatly
expand job execution, scheduling, monitoring, and alert notification
capabilities.
Using OPC Automation you could during a service
period schedule a job to run that activates WebSphere Application Server for
OS/390 and z/OS conversations automatically. But you still need to
decide how you handle the process for stopping and restarting servers.
The steps to be performed with the maintenance
interval are
1. Make your changes to the configuration. This can be done outside the service window.
2. Disable automation for all J2EE servers
3.
Activate the conversation; all
running servers will be stopped and restarted through activation
4. Enable automation after all servers have restarted
This approach impacts application availability as little as possible: WebSphere Administration shuts down servers as late in the activation process as possible and restarts the immediately after shutdown. In addition, it leaves untouched servers for which nothing has changed.
WebSphere Administration always uses the start command with the identifier notation (and thus without the JOBNAME keyword). Therefore SA OS/390 has to monitor on procedure name as job name after the restart by WebSphere Administration. To be consistent SA OS/390 must use the same form of the start command itself for restarts and moves. It must also use the identifier rather than the job name in its stop and cancel commands for WebSphere. Because these are not the commands SA OS/390 generates by default, they need to be individually defined for each server. In cases were there are many servers and systems involved this makes SA OS/390 definitions much more complex.
During the maintenance interval automation for all servers is switched off. Should any recovery action be required during this time for servers that would otherwise be unaffected by activation this action will not be triggered by SA OS/390.
The
steps to be performed with the maintenance interval are
1. Make your changes to the configuration. This can be done outside the service window.
2.
Before you activate a conversation
stop all J2EE Servers through SA OS/390
3.
Activate the conversation; no
server restart will be triggered through activation since no servers are
running
4.
After activation is complete
restart all J2EE servers through SA OS/390
All J2EE servers are shut down before activation and can only be restarted when activation is complete. Therefore users cannot use any of the Web applications offered through these servers during this time.
WebSphere Administration does not restart servers in this case; SA OS/390 performs the restart. It can therefore use the start command with the JOBNAME parameter and the matching stop and cancel commands that are generated according to SA OS/390 standards. This makes setting up SA OS/390 for WebSphere much easier.
Our test environment consisted of a sysplex with three systems: KEYA, KEYB and KEYC. The infrastructure on each system was a shared HFS, a shared DB2, and TCP/IP and this was included in the automation setup.
The Sysplex Distributor was used for load balancing and port-availability monitoring.
The HTTP server was running under z/OS and was included in the automation set up.
We used two J2EE Servers: BBOASR2 and BBOBANK.
The defined availability target was:
· LDAP on two of the three systems
· IBM HTTP server on two of the three systems
· Two of three systems for BBOASR2
· Two of three systems for BBOBANK
We were using dynamic VIPA. The IP addresses on each of the three systems were defined for dynamic VIPA backup.
All relevant ports were defined to the Sysplex Distributor and distributed across all system images. This included the LDAP port, the HTTP Server port, the WebSphere Application Server’s base environment (Daemon, Naming) ports and the J2EE server ports for the HTTP Transport Handler. Thus it was irrelevant on which two systems LDAP or HTTP Servers or J2EE Servers ran.
We run the base environment for WebSphere Application Server (Daemon, Naming) on each system, considering it as part or the WebSphere infrastructure. In case of a failover of a J2EE server this setup makes moving it to a different system much faster.
Alternatively you could start the base environment only on those systems where a J2EE server was started. So if for example BBOASR2 ran on KEYA and KEYB and BBOBANK ran on KEYB and KEYC all three systems had a Daemon and Naming running. But if both BBOASR2 and BBOBANK were running on KEYA and KEYB only, daemon and Naming server would not be required on KEYC.
Both HTTP servers had identical service definitions for the WebSphere plug-in. This was necessary because the HTTP port is distributed using the Sysplex Distributor.
Both LDAP servers could be used by any of the WebSphere instances because they used shared DB2 data and because their VIPA address was used in the WebSphere definitions.
As a consequence of using the IBM HTTP Server for z/OS we could not use session affinity. We therefore used persistent sessions, storing session information in shared DB2 tables.
Figure 10 shows how the components could be distributed over our three systems.
Figure 10: WebSphere on a Sysplex: Sample Configuration
Note that
· there are two instances of the HTTP Server (KEYA and KEYC)
· there are two instances of the LDAP Server (KEYB and KEYC)
· there are two instances of each J2EE Server
- BBOASR2 runs on KEYA and KEYB
- BBOBANK runs on KEYB and KEYC
· the WebSphere Base, as part of the infrastructure runs on all systems
Also note that HTTP server, LDAP server and J2EE servers were able to run on any of the three systems and that System Automation for OS/390 would move them to any other system in case of a failure that prevent restarts on the failing system. In this sense Figure 10 shows only one possible run-time environment for our sysplex.
The system set up is still fully functional (possibly with degraded performance) as long as at least one LDAP, one HTTP server, and one of each of the J2EE servers is running on any one of the three sysplex images. With backup IP stacks provided by dynamic VIPA the IP stack was not a single point of failure either. The only exception to completely continuous operation comes through the fact that WebSphere Application Servers needs to restart servers to activate configuration changes (see the discussion in the section 3.3 “Changing Configurations for WebSphere Application Server for OS/390 and z/OS” on page 28).
The
following sections give details of our configured sample setup for WebSphere
system automation. Only the major components that required setup decisions are
listed and, even for these, only the most relevant parameters are mentioned.
Our sysplex consisted of three images: KEYA, KEYB and KEYC.
The set up covered a heterogeneous MVS environment:
· z/OS 01.04.0 on KEYA
· OS/390 02.10 on KEYB
· z/OS 01.02.0 on KEYC
Apart from this the systems were mostly symmetrical: all prerequisite products and all components of WebSphere Application Server for OS/390 and z/OS were installed and were able to run on any of the three systems.
Cross-system sharing was implemented where applicable: RACF definitions, DB2 and HFS were shared.
The TCP addresses and the corresponding DNS names were:
· KEYA: DNS-Name boekeya.boeblingen.de.ibm.com on 9.152.64.73
· KEYB: DNS-Name boekeyb.boeblingen.de.ibm.com on 9.152.64.74
· KEYC: DNS-Name boekeyc.boeblingen.de.ibm.com on 9.152.64.77
Three VIPAs were defined:
· boekpla.boeblingen.de.ibm.com on 9.152.84.225
providing ports 9000, 5555, 1389, and 2389. These were the ports used for WebSphere base services and LDAP
· boekplb.boeblingen.de.ibm.com on 9.152.84.226
providing ports 80, 8081, 8082, and 8083.
These were the ports used by the IBM HTTP server and the HTTP Transport
Handlers of the J2EE Servers (8083 is spare)
· boekplc.boeblingen.de.ibm.com on 9.152.84.227 providing ports 20, 21, 23, and 623 for FTP, TELNET, and
OTELNET
Dynamic XCF addresses were defined for KEYA,
KEYB and KEYC: 9.152.84.249, 9.152.84.250 and 9.152.84.251.
The VIPAs were
distributed over the dynamic XCF addresses. The tcp.profile
for each system contained the following statements:
VIPADISTRIBUTE DEFINE
DESTIP 9.152.84.249 9.152.84.250
9.152.84.251
VIPADISTRIBUTE DEFINE
DESTIP 9.152.84.249 9.152.84.250
9.152.84.251
VIPADISTRIBUTE DEFINE
DESTIP 9.152.84.249 9.152.84.250
9.152.84.251
Each VIPA served as backup address, the tcp.profile BACKUP contained the following statements:
VIPADYNAMIC
VIPAbackup 100
9.152.84.225
VIPAbackup 100
9.152.84.226
VIPAbackup 100
9.152.84.227
ENDVIPADYNAMIC
This setup ensured high availability of the IP
applications running on the sysplex cluster even if one physical network
interface should fail or an entire IP stack or z/OS LPAR should be lost.
We
used the defaults for WebSphere
Application Server whenever possible. Here we only indicate the
parameters that relate to the setup for automation and high availability.
Our bootstrap system was KEYA and we used its DNS
name boekeya.boeblingen.de.ibm.com for the DAEMON_IPNAME parameter. Later we
introduced VIPA. However, you cannot easily change the DAEMON_IPNAME at a later
stage. Therefore we could not use the DNS name that is used for the dynamic IP
address as DAEMON_IPNAME. This has the disadvantage that we had a single point
of failure for WebSphere Administration: you can connect with the SMEUI to any
system with a running daemon but connection will only be established if the
daemon on KEYA is running. Had we used the VIPA DNS name
boekpla.boeblingen.de.ibm.com that distributes the ports we used for System
Management (9000) and for the DAEMON (5555), then we would have avoided this
problem.
Other TCP related parameters were set as
follows:
DAEMON_PORT=5555
RESOLVE_IPNAME= boekpla.boeblingen.de.ibm.com
RESOLVE_PORT=9000
We defined the port for the HTTP Transport
Handlers for the J2EE Servers as:
for BBOASR2:
BBOC_HTTP_PORT=8081
for
BBOBANK: BBOC_HTTP_PORT=8082.
These ports were distributed through boekplb.boeblingen.de.ibm.com.
In the webcontainer.conf
files we had virtual host definitions as follows:
for BBOASR2A on KEYA:
host.default_host.alias=
boekeya.boeblingen.de.ibm.com:8081,boekeya.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8081,boekplb.boeblingen.de.ibm.com
for BBOASR2B on KEYB:
host.default_host.alias=
boekeyb.boeblingen.de.ibm.com:8081,boekeyc.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8081,boekplb.boeblingen.de.ibm.com
for BBOASR2C on KEYC:
host.default_host.alias=
boekeyc.boeblingen.de.ibm.com:8081,boekeyc.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8081,boekplb.boeblingen.de.ibm.com
for BBOBANKA on KEYA:
host.default_host.alias=
boekeya.boeblingen.de.ibm.com:8082,boekeya.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8082,boekplb.boeblingen.de.ibm.com
for BBOBANKB on KEYB:
host.default_host.alias=
boekeyb.boeblingen.de.ibm.com:8082,boekeyc.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8082,boekplb.boeblingen.de.ibm.com
for BBOBANKC on KEYC:
host.default_host.alias=
boekeyc.boeblingen.de.ibm.com:8082,boekeyc.boeblingen.de.ibm.com,
boekplb.boeblingen.de.ibm.com:8082,boekplb.boeblingen.de.ibm.com
This allowed us to contact a specific instance
on a specific system by using the DNS names mapped to the real IP address of
that system. If, on the other hand, we used boekplb.boeblingen.de.ibm.com we
reached the system chosen by the Sysplex Distributor.
On the Naming server, we defined the LDAP URL
as follows:
com.ibm.ws.naming.ldap.masterurl=ldap://boekpla.boeblingen.de.ibm.com:1389
Port 1389 is distributed through boekplba.boeblingen.de.ibm.com. Therefore an LDAP server can run on any of our systems and still be reached by WebSphere Naming.
We installed an application on BBOBANK that required HTTP session support. As discussed in 2.4.1 “HTTP Sessions” on page 24 we had to use persistent sessions. To enable persistent session support for BBOBANK we defined in the WebSphere Application Server conversation a J2EE resource named IBMHttpSession. Its resource type was DB2datasource, and the location name was KEYDNSA that defines the same shared DB2 we used for all WebSphere definitions and for all application data.
In
the webcontainer.conf file for all BBOBANK instances
we enabled persistent sessions as follows:
session.enable=true
session.urlrewriting.enable=false
session.cookies.enable=true
session.protocolswitchrewriting.enable=false
session.cookie.name=JSESSIONID
session.dbenable=true
session.dbtablename=CU05.SESSTAB
The HTTP server listened on port 80. We enabled the WebSphere Application Server V4.0.1 for z/OS and OS/390 plug-in for all our applications. The definitions in the httpd.con file were:
Port 80
# ===========================
# WebSphere Directives V
4.0.1
# ===========================
ServerInit /usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:init_exit /usr/lpp/WebSphere,/local/websrv/was.conf
ServerTerm /usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:term_exit
Service /PolicyIVP/*
/usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:service_exit
Service /PolicyTwo/*
/usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:service_exit
Service /webapp/examples/*
/usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:service_exit
Service /webapp/examplesTwo/*
/usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugin.so:service_exit
Service /TheBankWeb/*
/usr/lpp/WebSphere/WebServerPlugIn/bin/was400plugn.so:service_exit
The SERVER INIT statement pointed to /local/websrv/was.conf. This file contained no application definitions, since we did not run servlets in the plug-in directly.
In the httpd.envars files for all systems we added the following statement:
HTTRESOLVE_IPNAME=boekpla.boeblingen.de.ibm.com
RESOLVE_PORT=9000
In this chapter we describe the System Automation for OS/390 setup hat we implemented to automate WebSphere Application Server for OS/390 and z/OS. As outlined in 1.3 WebSphere Application Automation and High Availability on page 8, the main objective was to provide
· ease of operations support:
- start all prerequisites and components of WebSphere in the right order
- stop all components of WebSphere in the right order
- show the status of all prerequisites and components
· availability support reducing downtimes:
- monitor prerequisites and components
- automatically restart failing components.
These objectives apply to a monoplex environment. In a sysplex environment System Automation for OS/390 can additionally increase availability by
· ensuring that WebSphere Application Servers run on a predefined number of images in the sysplex
· moving all or parts of WebSphere Application Server for OS/390 and z/OS from a failing image to another image when restart of vital component fails
· cleaning up failed system.
For WebSphere Application Server for OS/390 and z/OS administration a separate service period can be defined.
All applications and systems that you want to include
in automation must be defined to System Automation for OS/390
in an automation policy database. This database contains the objects to
be managed by System Automation for OS/390, and
the rules according to that automation of these objects proceeds.
The objects that are defined in the policy
database are called policy objects or entries. Applications and
systems, for example, are policy objects. Every policy object belongs to an entry
type that is identified by a three letter code. The most common entry types
are APL (application), APG (application group), GRP (groups of systems, i.e. sysplexes) and SYS (system).
Using the System Automation for OS/390
customization dialog you provide information to System
Automation for OS/390 such as:
· which resources you
want to automate, monitor and control.
· how resources are to be associated (grouped) with each other for
automation
· the relationships
between resources and groups of resources.
· which automation, such as automatic
startup or shutdown, is to be applied to these resources and how
· under which conditions automated actions should occur, and what actions should
be performed
· when
automation should be active, and how automation handles certain tasks and events
during system operation.
As a starting point, the following table
contains a list
of the main applications defined for WebSphere Application Server automation. Included are the names used in the policy
database, a short description and the number of systems the application should
run on. Please note that all the applications that are listed were able to run
on any of our systems.
|
Name |
Description |
Active Instances |
JES2 |
JES2 |
Job Entry Subsystem 2 |
On all systems |
VTAM |
VTAM |
Virtual Telecommunication
Access Method |
On all systems |
TCP/IP |
TCPIP |
TCP/IP |
On all systems |
WTR |
BBOWTR |
WebSphere trace writer |
On all systems where the
Daemon runs |
RRS |
RRS |
Resource Recovery Services |
On all systems |
DB2 Server |
DB2_DBM1 |
DB2 Subsystem |
On all systems |
LDAP Server |
BBOLDAP |
Websphere
LDAP |
On 2 of 3 systems |
WebSphere Base |
BBODMN |
WebSphere Daemon |
On all systems |
|
BBOSMS |
WebSphere SMS control region |
On all systems |
|
BBONM |
WebSphere Naming-Server control region |
On all systems |
|
BBOIR |
WebSphere I/F-repository control region |
On all systems |
J2EE Server |
BBOASR2 |
WebSphere J2EE-server control region |
On 2 of 3 systems |
|
BBOBANK |
WebSphere J2EE-server control region |
On 2 of 3 systems |
HTTP Server |
WEBSRV |
IBM HTTP Server |
On 2 of 3 systems |
Additionally, for WebSphere MQSeries, we automated the MQSeries Queue Manager and the Channel Initiator, although this is not required for WebSphere.
Figure 11 gives an overview of the defined groups.
Figure 11: Groups and Applications in a Sysplex Environment
The main purpose of the sysplex group named KEYAPLEX group is to select the systems included in the sysplex and the application groups of the sysplex. It contains the following:
Short description |
sysplex with systems KEYA, KEYB, KEYC |
Group type |
SYSPLEX |
Linked Systems |
|
Entry Name |
Short Description |
KEYA |
System KEYA |
KEYB |
System KEYB |
KEYC |
System KEYC |
Linked ApplicationGroups |
|
Entry Name |
Short Description |
BBO_ASR2 |
WebSphere J2EE sysplex server group |
BBO_BANK |
WebSphere J2EE sysplex server group |
BBO_J2EE |
WebSphere sysplex basic group |
BBO_LDAP |
WebSphere LDAP sysplex server group |
BBO_MQPLEX |
WebSphere MQ Series sysplex basic group |
BBO_PLEX |
WebSphere sysplex server group |
SGA_PLEX |
DB2 - SGAn sysplex basic group |
WWW_WEBSRV |
IBM HTTP Server sysplex server group |
If you have more systems in your sysplex include them in Linked Systems. If you have additional J2EE or CORBA servers include them as Linked ApplicationGroups.
These system groups define the clone values to be used for AOCCLONE and AOCCLONE1 and all application groups for each system.
Short description |
System KEYA |
Clone value 0 |
1 |
Clone value 1 |
A |
Linked ApplicationGroups |
|
Entry Name |
Short Description |
BBO_DAEMON |
WebSphere system basic group |
BBO_MQ |
WebSphere MQ Series system group |
SGA_SYS |
DB2 - SGAn basic system group |
KEYB and KEYC are similar (except for the clone values):
Short description |
System KEYB |
Clone value 0 |
2 |
Clone value 1 |
B |
and
Short description |
System KEYC |
Clone value 0 |
3 |
Clone value 1 |
C |
The AM_KEY application group defines all Automation Managers, SYSVIEW SA OS/390 Agent Resources. The BASE_SYS group defines basic system components such as JES2,VLF,LLA. The groups BBO_MQPLEX and BBO_MQ are included for WebSphere MQ Series and are not used here. NETWORK defines networking resources. SGA_PLEX and SGA_SYS defines DB2. These groups are considered extensions of the base system. No details are given here.
The following application groups are directly related to our WebSphere set-up:
· BBO_PLEX
· BBO_ASR2
· BBO_BANK
· BBO_LDAP
Some details are given below. For a full description please refer to our sample PDB.
The
relationships between groups or applications are discussed in 5.5
Relationships on page 54.
This group defines the WebSphere base environment.
Figure 12: BBO_PLEX WebSphere Base Environment
Here are the definitions:
Short description |
WebSphere sysplex server group |
Long description |
group with BBO_DAEMON groups included |
Application Group Type |
SYSPLEX |
Nature |
Server |
Automation Name |
BBO_PLEX |
Behaviour |
ACTIVE |
Default Preference |
*DEF |
Auto-link APL Resources to APG |
YES |
Availability Target |
*ALL |
Linked Resources |
||||
Resource Name |
Entry Type |
System |
Entry Name |
Preference Value |
BBO_DAEMON/APG/KEYA |
APG |
KEYA |
BBO_DAEMON |
700 |
BBO_DAEMON/APG/KEYB |
APG |
KEYB |
BBO_DAEMON |
700 |
BBO_DAEMON/APG/KEYC |
APG |
KEYC |
BBO_DAEMON |
700 |
Since BBO_PLEX is included
in the Linked ApplicationGroups of our sysplex group KEYAPLEX all
systems defined there are by default included as Linked Resources.
If you have systems where you never want the WebSphere infrastructure to run you must exclude then from this list.
Note that we consider the WebSphere base as part of the infrastructure and therefore specified an Availability Target of 3. This means that it will automatically be started on all of our systems. The advantage is that System Automation for OS/390 can move a J2EE server from a failed system to a different system without first having to start the base WebSphere environment. You can change the availability target to a lower value. In that case it can happen that System Automation for OS/390 will first have to start the base environment before it can move a J2EE server that, of course takes additional time.
A J2EE server can only run on a system where the base WebSphere Application Server environment is running. You should therefore set the Availability Target at least as high as the maximum of the targets for all the J2EE Servers APGs.
This basic system group defines that the WebSphere infrastructure consists of the Daemon, System Management, Naming and Interface Repository.
Short
description |
WebSphere
system basic group |
Long
description |
Group
with Daemon, SMS, Naming and IR included |
Application
Group Type |
SYSTEM |
Nature |
Basic |
Automation
Name |
BBO_DAEMON |
Behaviour |
ACTIVE |
Default
Preference |
*DEF |
Auto-link
APL Resources to APG |
YES |
Linked Applications |
|
Entry Name |
Short Description |
BBODMN |
WebSphere Daemon |
BBOIR |
WebSphere I/F-repository control region |
BBONM |
WebSphere Naming-Server control region |
BBOSMS |
WebSphere SMS control region |
This basic sysplex group joins together the server groups for the two J2EE servers we use. It can be used to stop and restart all J2EE servers during WebSphere maintenance (see the discussion in 3.3 Changing Configurations for WebSphere Application Server for OS/390 and z/OS on page 28).
Figure 13: BBO_J2EE Group for J2EE Servers
Here is its description:
Short description |
WebSphere sysplex basic group |
Application Group Type |
SYSPLEX |
Nature |
Basic |
Automation Name |
BBO_J2EE |
Behaviour |
ACTIVE |
Default Preference |
*DEF |
Auto-link APL Resources
to APG |
YES |
Linked Resources |
|||
Resource Name |
Entry Type |
Entry Name |
Preference Value |
BBO_ASR2/APG |
APG |
BBO_ASR2 |
SELECTED |
BBO_BANK/APG |
APG |
BBO_BANK |
SELECTED |
If you have more J2EE servers you should include them here as Linked Resources.
This sysplex server group covers one of our two J2EE servers.
Short description |
WebSphere J2EE sysplex server group |
Application Group Type |
SYSPLEX |
Nature |
Server |
Automation Name |
BBO_ASR2 |
Behaviour |
ACTIVE |
Default Preference |
*DEF |
Auto-link APL Resources
to APG |
YES |
Availability Target |
2 |
Linked Resources |
||||
Resource Name |
Entry Type |
System |
Entry Name |
Preference Value |
BBOASR2/APL/KEYA |
APL |
KEYA |
BBOASR2 |
700 |
BBOASR2/APL/KEYB |
APL |
KEYB |
BBOASR2 |
700 |
BBOASR2/APL/KEYC |
APL |
KEYC |
BBOASR2 |
700 |
The main purpose of this group is to define that an instance of this J2EE server can run on any of our three systems (with no specific priority as to on that it should be started) and that two instances should be started under normal circumstances.
If you have more systems in your sysplex, all of that are included in the sysplex group, then all are listed in Linked Resources. You may exclude some systems explicitly to prevent them from being a candidate for running the J2EE server.
If you have preferred systems for this specific J2EE server, change the Preference Value of the systems.
Adapt the Availability Target for each J2EE server to your needs according to the expected load for the server.
Note: you have a single point of failure if you use a target of 1. The
applications installed on this J2EE server are then not available while System Automation for OS/390 restarts the J2EE server on another system in case of
a failover of the current system.
This group does exactly the same for our second J2EE server what BBO_ASR2 does for the first one.
This sysplex server group defines that LDAP servers can run on any of our three systems and that two LDAP servers should normally be started.
Short description |
WebSphere LDAP sysplex server group |
Application Group Type |
SYSPLEX |
Nature |
Server |
Automation Name |
BBO_LDAP |
Behaviour |
ACTIVE |
Default Preference |
*DEF |
Auto-link APL Resources to APG |
YES |
Availability Target |
2 |
Linked Resources |
||||
Resource Name |
Entry Type |
System |
Entry Name |
Preference Value |
BBOLDAP/APL/KEYA |
APL |
KEYA |
BBOLDAP |
700 |
BBOLDAP/APL/KEYB |
APL |
KEYB |
BBOLDAP |
700 |
BBOLDAP/APL/KEYC |
APL |
KEYC |
BBOLDAP |
700 |
If you use a set-up similar to the one we use you can have LDAP servers on as many systems as you need for covering the load (increase Availability Target if necessary). If you have preferred systems for your LDAP servers change the settings for Preference Value.
This sysplex server group defines that HTTP servers can run on any
of our three systems and that two HTTP servers should normally be started.
Short description |
IBM HTTP Server sysplex server group |
Application Group Type |
SYSPLEX |
Nature |
Server |
Automation Name |
WWW_WEBSRV |
Behaviour |
ACTIVE |
Default Preference |
*DEF |
Auto-link APL Resources to APG |
YES |
Availability Target |
2 |
Linked Resources |
||||
Resource Name |
Entry Type |
System |
Entry Name |
Preference Value |
WEBSRV/APL/KEYA |
APL |
KEYA |
WEBSRV |
700 |
WEBSRV/APL/KEYB |
APL |
KEYB |
WEBSRV |
700 |
WEBSRV/APL/KEYC |
APL |
KEYC |
WEBSRV |
700 |
If you use a set-up similar to the one we use you can have HTTP servers on as many systems as you need for covering the load (increase Availability Target if necessary). If you have preferred systems for your HTTP servers change the settings for Preference Value.
The following applications are specific to our WebSphere set-up and will be briefly discussed here:
· BBODMN
· BBOSMS
· BBONM
· BBOIR
· BBOWTR
· BBOASR2
· BBOBANK
· BBOLDAP
· WEBSRV
The relationships between groups or applications are discussed in 5.5 Relationships on page 54.
Also included in the PDB are the definitions for WebSphere MQSeries. Since MQSeries is not required for WebSphere Application Server the definitions are not discussed here.
BBO_CLASS represents a common policy to be
inherited the WebSphere daemon, the J2EE and the LDAP servers.
It decreases the effort for defining the
entries for each server individually and keeps the common definitions
consistent.
Short description |
WebSphere class with general definitions |
Object Type |
CLASS |
Application Type |
STANDARD |
Subsystem Name |
BBO_CLASS |
Application Messages
and User Data |
|
Message id |
Description |
SHUTFORCE |
Executed when force shutdown is invoked |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTIMMED |
Executed when immediate shutdown is invoked |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTNORM |
Executed when normal shutdown is invoked |
|
|
Linked Instances |
|
Instance Name |
Description |
BBOASR2 |
WebSphere J2EE-control region |
BBOBANK |
WebSphere J2EE-control region |
BBODMN |
WebSphere Daemon |
BBOLDAP |
WebSphere LDAP |
The common definitions for all servers cover the actions to be taken during shutdown. If you have additional J2EE servers you should link them to this class.
This is the definition for the WebSphere daemon.
Short description |
WebSphere Daemon |
Linked to Class |
BBO_CLASS |
Application Type |
STANDARD |
Subsystem Name |
BBODMN |
MVS job name |
DAEMON0&AOCCLONE. |
JCL Procedure Name |
BBODMN |
Subsystem Startup Parameters |
,SRVNAME=DAEMON0&AOCCLONE. |
Application Messages
and User Data |
|
Message id |
Description |
PRESTART |
Executed before startup is initiated |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTFINAL |
Executed after final termination message |
|
|
The WLM environments for System Management, Naming and Interface Repository may have been stopped by too many failing attempts to restart the component. Because the scope of WLM is sysplex-wide this may have happened on a different system from the one where the Daemon is to be started now. Therefore the WLM environments are activated again (RESUMEd) before the Daemon is started.
Normally, when the WebSphere daemon terminates SMS, Naming and IR end as well and the respective control region and server region address spaces are no longer available. If termination of the daemon fails to end all the dependent servers, inactive address spaces may be left in the system. To avoid problems System Automation issues cancel commands for the dependent control regions in the final termination phase of the daemon. The same cancel commands are issued before the daemon is started to protect against the case that the SHUTFINAL phase was not executed because no final termination message was received by System Automation.
Note that only control regions are known to System Automation. To terminate server regions you can use INGRCLUP. INGRCLUP calls a clean up routine that checks for and, if necessary, terminates address spaces. INGRCLUP is called after termination of the WebSphere daemon to terminate SMS, Naming and IR server regions if the daemon ends in such a way, that if cannot clean up on its own accord. INGCLUP is also executed before the daemon is started. This may be necessary if the WebSphere daemons abends in such a way that it even fails to notify System Automation of the abend.
For more details on the cleanup routine refer to 5.7
Clean-up
Program on page 58.
The WebSphere System Management Server is supervised by, but not started or stopped by System Automation.
Short description |
WebSphere SMS control region |
Application Type |
STANDARD |
Subsystem Name |
BBOSMS |
MVS job name |
BBOSMS |
External Startup |
ALWAYS |
External Shutdown |
ALWAYS |
Application Messages
and User Data |
|
Message id |
Description |
BBOU0199E |
WLM environment stopped |
|
|
The WebSphere System Management Server
is started and stopped by the WebSphere Daemon (External as far as System Automation is concerned, see 5.5.3 The WebSphere Base Environment on page 56).
Note: Only the SMS Control Region is known to System Automation. Server regions are not under System Automation control. However, the WLM environment that starts server regions is resumed if it has previously been stopped by WLM. This happens while the SMS control region is running and is triggered by the BBOU0199E message.
BBONM and BBOIR are treated analogously to
BBOSMS.
System Automation starts and stops the WebSphere trace writer as part of the Base system.
Short description |
WebSphere trace writer |
Application Type |
STANDARD |
Subsystem Name |
BBOWTR |
MVS job name |
BBOWTR |
Application Messages
and User Data |
|
Message id |
Description |
SHUTFORCE |
Executed when force shutdown is invoked |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTIMMED |
Executed when immediate shutdown is invoked |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTINIT |
Executed when shutdown is initiated |
|
|
Application Messages
and User Data |
|
Message id |
Description |
SHUTNORM |
Executed when normal shutdown is invoked |
|
|
Application Messages
and User Data |
|
Message id |
Description |
STARTUP |
Executed to initiate the startup |
|
|
If you want to start trace writers for other components you can model System Automation definitions for those after the definitions for the WebSphere trace writer shown here.
This is the first J2EE server definition:
Short description |
WebSphere J2EE-control region |
Linked to Class |
BBO_CLASS |
Application Type |
STANDARD |
Subsystem Name |
BBOASR2 |
MVS job name |
BBOASR2&AOCCLONE1. |
JCL Procedure Name |
BBOASR2 |
Subsystem Startup Parameters |
,SRVNAME=BBOASR2&AOCCLONE1. |
Application Messages and User Data |
|
Message id |
Description |
BBOU0199E |
WLM environment stopped |
|
|
Application Messages and User Data |
|
Message id |
Description |
PRESTART |
Executed before startup is initiated |
|
|
Application Messages and User Data |
|
Message id |
Description |
SHUTFINAL |
Executed after final termination message |
|
|
Note: Only the Control Region is known to System Automation. Server Regions are not under System Automation control.
However, the WLM environment that starts server regions is resumed if it has previously been stopped by WLM. This happens once while the J2EE control region is running triggered by the BBOU0199E message. It also happens before the J2EE server is started.
Normally, when the J2EE control region terminates all
its server regions end as well and their address spaces are no longer
available. If this process fails inactive address spaces may be left in the
system. INGRCLUP is therefore called after termination of the J2EE server
control ends to terminate server regions if the control region ends in such a
way, that if cannot clean up on its own accord. INGRCLUP is also executed
before the control region is started. This may be necessary if the J2EE control
region abends in such a way that it even fails to
notify System Automation of the abend.
For more details on the cleanup routine refer to 5.7
Clean-up
Program on page 58.
The definitions for the second J2EE server are analogous to the
definitions for the first J2EE server.
No special considerations apply to the LDAP server and the HTTP
server.
Figure 14 completes Figure 11 “Groups and Applications in a Sysplex Environment” on page 41 by including the relationships defined between groups and applications.
Figure 14: Relationships
Some details follow.
Each LDAP server needs TCP/IP and DB2.
Relationships |
|||
Relationship Type |
Supporting Resource |
Description |
Satisfy condition |
FORCEDOWN |
SGA_SYS/APG/= |
stop LDAP if DB2 fails |
WhenObservedDown |
FORCEDOWN |
TCPIP/APL/= |
stop LDAP if TCPIP fails |
WhenObservedDown |
HASPARENT |
SGA_SYS/APG/= |
LDAP is dependent on DB2 |
|
HASPARENT |
TCPIP/APL/= |
LDAP is dependent on TCPIP |
|
The HTTP Server needs TCP/IP.
Relationships |
|||
Relationship Type |
Supporting Resource |
Description |
Satisfy condition |
FORCEDOWN |
TCPIP/APL/= |
stop HTTP server if TCPIP fails |
WhenObservedDown |
HASPARENT |
TCPIP/APL/= |
HTTP server is dependent on TCPIP |
|
No webserver should be started if not needed, i.e. if no J2EE server runs. Therefore the WWW_WEBSRV group depends on the J2EE server group.
Relationships |
|||||
Relationship Type |
Supporting Resource |
Description |
Automation |
Chaining |
Satisfy condition |
MAKEAVAILABLE |
BBO_J2EE/APG |
start if at least 1 J2EE server active |
ACTIVE |
WEAK |
WhenRunning |
The applications BBOIR, BBONM and BBOSMS have a HASPARENT relationship to the application BBODMN.
Relationships |
|||
Relationship Type |
Supporting Resource |
Description |
Satisfy condition |
HASPARENT |
BBODMN/APL/= |
IR is started by WAS daemon |
StartsMeAndStopsMe |
BBOIR, BBONM and BBOSMS cannot be started until
the BBODMN is available. They cannot be stopped before the daemon is
unavailable.
Actually the Satisfy
condition StartsMeAndStopsMe takes care of the fact that BBOIR, BBONM and BBOSMS are
started by the daemon.
The daemon, BBOIR, BBONM and BBOSMS are the
applications in the BBO_DAEMON group that represent the WebSphere base environment on each system. This group has the
following
relationships:
Relationships |
|||||
Relationship Type |
Supporting Resource |
Description |
Automation |
Chaining |
Satisfy condition |
FORCEDOWN |
BBO_LDAP/APG |
stop WAS if LDAP fails |
|
|
WhenObservedHardDown |
FORCEDOWN |
SGA_SYS/APG/= |
stop WAS if DB2 fails |
|
|
WhenObservedDown |
FORCEDOWN |
TCPIP/APL/= |
stop WAS if TCPIP fails |
|
|
WhenObservedDown |
HASPARENT |
SGA_SYS/APG/= |
start/stop WAS dependent on DB2 |
|
|
|
HASPARENT |
TCPIP/APL/= |
start/stop WAS dependent on TCPIP |
|
|
|
MAKEAVAILABLE |
BBO_LDAP/APG |
start WAS if at least 1 LDAP available |
ACTIVE |
WEAK |
WhenRunning |
This
means that the WebSphere base environment can only be started when the local DB2
is running and has to be stopped if the local DB2 fails.
The WebSphere base environment can only be started if the local TCP
is running and has to be stopped if the local TCP fails.
It should be started when the LDAP group is
running, that means that at least one of the LDAP servers runs and has to be
stopped when the group is hard down, that means that no
LDAP server is running.
BBOASR2 and BBOBANK have a relationship of HASPARENT to the application group BBO_DAEMON.
Relationships |
||
Relationship Type |
Supporting Resource |
Description |
HASPARENT |
BBO_DAEMON/APG/= |
J2EE server dependent on daemon |
If you
have additional J2EE servers in your installation you should define an
equivalent relationship for each of them.
You
should also check if you have additional dependencies between the applications
you run on your J2EE servers and other resources such as MQSeries or CICS or
IMS and so on. If you identified any other dependencies you should add
additional relationships here.
SA OS/390 provides an enhanced message table including messages needed to automate WebSphere Application Server for OS/390 and z/OS through APAR OA02375.
Messages issued during start and termination processing as well as messages that indicate error conditions on that System Automation can react are included for the following components:
· WebSphere
- the base environment with daemon, System Management Services, Naming and Interface Repository
- J2EE Server control region
· Related components
- Trace writer
- LDAP Server
- IBM HTTP Server
- TCP/IP subcomponents
Also included in the enhanced message table are WebSphere MQ Series messages.
The clean-up program is a safeguard against the case that address spaces are left over in the system after termination of a component that would normally also terminate dependent address spaces.
For WebSphere Application Server for OS/390 and z/OS could happen for
· System Management Services, Naming and Interface Repository control regions that are started by the daemon
· Server regions started by those control regions or by J2EE control regions when needed to satisfy the WLM defined service goals.
Since
the control regions will normally have unique jobnames simple MVS cancel or
stop commands can be issued by System
Automation during shut-down or pre-start processing as shown in 5.4.2
BBODMN on page 49.
The server regions for a control region will however all have the same jobname. To cancel those you have to issue individual cancel commands including jobname and address space ID. This is done by the cleanup program. You call the cleanup program with the command
INGRCLUP jobname
To
reduce the risk of inadvertently destroying address spaces, we allow only fully
qualified jobnames and only jobnames that do not address a resource defined in
the automation policy.
This section addresses what you can do to automate the activation of WebSphere configuration changes and minimize the impact this has on availability due to the fact that WebSphere Application Server for OS/390 and z/OS Administration shuts down and restarts all running J2EE servers with changed configuration when the conversation containing these changes is activated.
You can either use basic System Automation for OS/390 or extend it by OPC Automation.
· System Automation for OS/390
allows you to define service period. In this way it provides the possibility to
make stop and start requests at specified points in time independently of human
intervention. An operator can temporarily modify a service period if manual
control is desired. The activities to be performed during this interval could
be initiated manually or started by System Automation.
· OPC itself refers to the Tivoli
Workload Scheduler. OPC Automation is an extension of SA OS/390 to collaborate
with Tivoli Workload Scheduler. This expands your job execution, scheduling,
monitoring, and alert notification capabilities. Using OPC Automation you could,
during a service period, use Tivoli Workload Scheduler to stop through System Automation, the WebSphere J2EE
servers, then run a job that activates WebSphere Application Server for
OS/390 and z/OS conversations and finally restart the WebSphere J2EE servers through System Automation.
The steps to be
performed with the maintenance interval are
1. Make
your changes to the configuration. This can be done at any time outside the maintenance
window. You should use the WebSphere
Application Server for OS/390 and z/OS SMEUI and go through all the steps except the last
one that is conversation activation.
2. Before
you activate a conversation stop all J2EE Servers through SA OS/390 or OPC
Automation.
3. Activate
the conversation; no server restart will be triggered through activation since
no servers are running. You can either do this manually through the SMEUI or
write little REXX execs to perform the conversation activation using the WebSphere Application Server for
OS/390 and z/OS SM Scripting API. For this moment the only language
that is supported by the SM Scripting API is REXX. When you use this interface
you can invoke the execs through a job that you can automate through OPC.
Details are described below. Samples can be found in the download zone.
4. After activation is complete restart all J2EE servers through SA OS/390 or OPC Automation.
To use the SM
Scripting API you have to write a REXX script that you should run under OMVS.
A typical SM Scripting
API script consists of three parts:
· One part to generate the input
file
· One part to call the function
· One part to process the result
The following functions,
supplied by the API, were used in the sample EXECs:
· CB390CFG(...) for calling an
administrations function
· XMLGEN(...) for generating the
input file
To use one of the
administration functions, the default input XML files must be accessible. Each
default XML file lists all valid attributes for the corresponding administrative
function.
To use the API
additions to the LIBPATH, CLASSPATH and PATH plus one new environment variable
are needed.
You must run under an
OMVS userid registered as a WebSphere for z/OS
Systems Management Administrator such as CBADMIN.
Note: The scripts cannot be run from another userid that has done a switch user to become CBADMIN.
As a sample we provide two jobs, scripts and REXX EXECs.
· Each job sets the userid to CBADMIN and calls a script to be executed under OMVS.
· Each OMVS script set up the required environment variables and calls the respective REXX EXEC.
· The REXX EXECs call the administration function and set the return code.
- WASMNTR1 scans all conversations to find if there are any conversations ready to be deployed.
o If it finds no conversation it set the return code to 4.
o If it finds exactly one conversation, it generates an XML file that can be used as input by WASMNTR2 to activate the conversation. The return code is set to 0.
o If it finds more than one conversation it still generates the XML file, but set the return code to 8.
- WASMNTR2 commits (activates) the conversation named in the XML input file. If more than one conversation is contained in the input file, it activates the most recent one.
Note that WebSphere Application Server for OS/390 and z/OS allows you to have more than one conversation that is in a state where it could be activated. But normally this should be avoided: After one of them has been activated WebSphere will not allow you to active the others. This is the reason for setting a return code of 8 by WASMNTR1 in this case.
For more details, look at the source of the samples.
Using these or similar jobs you could perform the following steps:
Run the first job
1. if return code is 4: no further action is needed, there is nothing to be activated.
2. if return code is 0: there is exactly one conversation to be
activated. You should
· stop all J2EE servers either through System Automation commands or through
OPC Automation,
· run the second job to activate the conversation,
- if return code is
0, the new conversation has successfully been actived.
Restart all stopped J2EE servers either through System Automation
commands or through OPC Automation.
- else restart all
stopped J2EE servers, offline check what the problem was, schedule a new
maintenance period when the problem is resolved.
3. if return code is 8
· offline check the input file for the second step. Through SMEUI delete all conversations you do not want to activate and schedule a new maintenance period.
4. if return code is greater than 8
- offline check the output and error files and fix the problem,
- when the problem is fixed, schedule a new maintenance period.
Possible error causes could be:
· set-up problems, e.g.: wrong CLASSPATH, PATH, LIBPATH environment variables
· system problems, e.g.: WebSphere SMS on bootstrap system not running
· authorization problems, e.g.: no access to USS files, no WebSphere Administrator userid
· conversation problems, e.g.: conversation found ready for activation is not based on currently active conversation.