PQ84216: AFTER APPLYING UQ84418 (W502001) SERVANT REGIONS FAIL TO BEFORE INITIALIZATION COMPLETE

 A fix is available

Obtain the fix for this APAR



APAR status
Closed as program error.

Error description
After applying UQ84418 (W502001) servant regions (Deployment
Manager SR or app server SRs) may not stay up when the
controller regions are started.  There problem lies in the
communication between the servant region and its controller
region for activating internal runtime MBeans.
.
This failure only occurs in a multi-homed environment.
.
The symptoms are as follows:
.
In the servant region (SR):
BBOO0222I SECJ0244I: Security service failed to start
   successfully
.
and in the error log for the SR and/or exception tracing:
.
BBOO0220E SECJ0206E: Error creating or registering
   SecurityAdmin mBean. The exception is
   com.ibm.websphere.management.exception.AdminException:
   ADMN0005E: Unable to activate MBean: type SecurityAdmin...
.at com.ibm.ws.management.MBeanFactoryImpl.activateMBean
.at com.ibm.ws.management.MBeanFactoryImpl.activateMBean
.at com.ibm.ws.security.core.SecurityComponentImpl.initialize
.at com.ibm.ws.security.core.SecurityComponentImpl.start
.at com.ibm.ws.runtime.component.ContainerImpl.startComponents
.at com.ibm.ws.runtime.component.ContainerImpl.start
.at com.ibm.ws.runtime.component.ServerImpl.start
.at com.ibm.ws.runtime.WsServer.start
.at com.ibm.ws.runtime.WsServer.main
.at com.ibm.ws390.orb.CommonBridge.WsServerStartup
.at com.ibm.ws390.orb.ServerRegionBridge.initCSI
.at com.ibm.ws390.orb.ORBEJSBridge.initCSI
---- Begin backtrace for nested exception.
java.rmi.MarshalException: CORBA MARSHAL 1229123584 Maybe;
  nested exception is:.
.org.omg.CORBA.MARSHAL:   minor code: 4942F000 completed: Maybe.
org.omg.CORBA.MARSHAL:   minor code: 4942F000 completed: Maybe.
.at java.lang.Class.newInstance0
.at java.lang.Class.newInstance
.at com.ibm.rmi.util.Utility.readSystemException
.at com.ibm.ws390.orb.ClientDelegate.invoke
.at org.omg.CORBA.portable.ObjectImpl._invoke
.at com.ibm.ws.management._ControlAdminService_Stub.
                                            activateProxyMBean
.at com.ibm.ws.management.MBeanFactoryImpl.activateMBean
.at com.ibm.ws.management.MBeanFactoryImpl.activateMBean
.at com.ibm.ws.security.core.SecurityComponentImpl.initialize
...
.
BBOO0220E SECJ0007E: Error during security initialization. The
   exception is com.ibm.websphere.management.exception..
   AdminException: ADMN0005E: Unable to activate MBean:
   type SecurityAdmin...
   <<< same stack trace with marshal exception above >>>
...
BBOO0220E WSVR0003E: Server xxxxxx failed to start
java.lang.RuntimeException: ADMN0005E: Unable to activate
    MBean: type SecurityAdmin...
   <<< same stack trace with marshal exception above >>>
...
BBOO0220E WSVR0009E: Error occurred during startup
.
In the controller region (CR) numerous lines of:
java.lang.OutOfMemoryError
.
The servant region never gets the usual messages before it
ends:
BBOO0222I WSVR0001I: Server SERVANT PROCESS xxxx open for
e-business
and
BBOO0020I INITIALIZATION COMPLETE FOR WEBSPHERE FOR Z/OS SERVANT
PROCESS xxxxxx.
The absence of these messages combined with the numerous
marshal exceptions and out of memory exceptions is a match
for this apar.
.
With W502001, it is ok to see ONE set of the marshal exception
combined with an out of memory exception during startup.  If
the servant region remains up and gets open for business, the
presence of the messages is ok.
.
Even with a fix for PQ84216, you will still see ONE pair of
the java.lang.OutOfMemoryError (in CR) and org.omg.CORBA.MARSHAL
(in SR) exceptions during SR initialization.  The
java.lang.OutOfMemoryError is recoverable and has no relevance
to the java heap size of the server.  The java heap size of
the server should not be adjusted when this exception is seen
while the SR is initializing.
(3/16/2004 update - apars 
PQ85550 (MD19208) and 
PQ86116
  will remove the remaining expected java.lang.OutOfMemoryError
  and java.rmi.MarshalException exceptions.  With these two
  apars we will no longer issue these exceptions in expected
  conditions.)
Local fix
A ++APAR is now available as AQ84216 on top of service level
W502001.  PQ84216 will ship in service level W502002.
Contact L2 WebSphere to obtain the ++apar AQ84216.
.
There is a non-code work around available if it is a viable
alternative for customers.  This may not be a viable alternative
for all customers.  Since this problem is only exposed in
multi-home configurations, the environment variable
protocol_iiop_listenIPAddress can be set to a specific ip
address.  This env var can be set for any and all servers
experiencing the problem reported in this apar.  This will
effectively make the configuration no longer multi-homed.  We
understand this is not a permanent fix and may not be acceptable
to all customers but is an option as a temporary workaround
until a code temp fix is available.
.
The environment variable protocol_iiop_listenIPAddress can be
temporarily set in was.env of the server experiencing the
problem and restart the server to see if this solves the
problem.
.
To make the value for protocol_iiop_listenIPAddress permanent,
use the admin console and set the following fields (depending
on whether the configuration is base app server or ND):
.
App server (Base appserver or ND):
Servers -> Application Servers -> <yourServerName> ->
   End Points -> ORB_LISTENER_ADDRESS
.
Deployment Manager (ND):
System Administration -> Deployment Manager -> dmgr ->
   End Points -> ORB_LISTENER_ADDRESS
.
When the ++apar or PTF is available, you can reset these values
to what you had before.
Problem summary
****************************************************************
* USERS AFFECTED: All users of WebSphere Application Server    *
*                 V5.0 for z/OS                                *
****************************************************************
* PROBLEM DESCRIPTION: On Servant Region bring-up in a         *
*                      multi-home environment, initial         *
*                      Discovery between Controller Region and *
*                      Servant Region succeeds, followed by    *
*                      MARSHAL exceptions on all subsequent    *
*                      activateProxyMBean method calls, with   *
*                      the Servant Region terminating after    *
*                      the Security service fails to start     *
*                      successfully.                           *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Servant Region initialization in a multi-home environment
(especially where a VIPA is present), exposed a path where a
partial object key was used to drive the activateMBeanProxy
method from the Controller Region to the Servant Region. Use of
this partial key caused execution of a code path that prevented
correct interpretation of partner version in the Controller
Region, resulting in a series of CORBA::MARSHAL exceptions,
java.lang.OutOfMemory exceptions, and the inability of several
services required by the Servant Region to correctly initialize.
Problem conclusion
Guards were put in place to ensure correct handling and
propagation of partner version contexts, whenever present.

APAR PQ84216 is associated with SERVICE LEVEL W502002 of
WebSphere Application Server V5.0 for z/OS.
Temporary fix Comments
APAR information
APAR number PQ84216
Reported component name WEBSPHERE FOR Z
Reported component ID 5655I3500
Reported release 500
Status CLOSED PER
PE YesPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-02-04
Closed date 2004-02-12
Last modified date 2004-03-16

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
BBOUBINF          

Publications Referenced

Fix information
Fixed component name WEBSPHERE FOR Z
Fixed component ID 5655I3500

Applicable component levels
R500 PSY UQ85128    UP04/02/20 P F402 

  Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.


Document Information


Current web document: swg1PQ84216.html
Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PQ84216
IBM Group: Software Group
Modified date: Mar 16, 2004