PQ84216: AFTER APPLYING UQ84418 (W502001) SERVANT REGIONS FAIL TO BEFORE INITIALIZATION COMPLETE | |||||||||||||||||||||||||||||||||||||||||||
![]() |
|||||||||||||||||||||||||||||||||||||||||||
![]() APAR status Closed as program error. Error description After applying UQ84418 (W502001) servant regions (Deployment Manager SR or app server SRs) may not stay up when the controller regions are started. There problem lies in the communication between the servant region and its controller region for activating internal runtime MBeans. . This failure only occurs in a multi-homed environment. . The symptoms are as follows: . In the servant region (SR): BBOO0222I SECJ0244I: Security service failed to start successfully . and in the error log for the SR and/or exception tracing: . BBOO0220E SECJ0206E: Error creating or registering SecurityAdmin mBean. The exception is com.ibm.websphere.management.exception.AdminException: ADMN0005E: Unable to activate MBean: type SecurityAdmin... .at com.ibm.ws.management.MBeanFactoryImpl.activateMBean .at com.ibm.ws.management.MBeanFactoryImpl.activateMBean .at com.ibm.ws.security.core.SecurityComponentImpl.initialize .at com.ibm.ws.security.core.SecurityComponentImpl.start .at com.ibm.ws.runtime.component.ContainerImpl.startComponents .at com.ibm.ws.runtime.component.ContainerImpl.start .at com.ibm.ws.runtime.component.ServerImpl.start .at com.ibm.ws.runtime.WsServer.start .at com.ibm.ws.runtime.WsServer.main .at com.ibm.ws390.orb.CommonBridge.WsServerStartup .at com.ibm.ws390.orb.ServerRegionBridge.initCSI .at com.ibm.ws390.orb.ORBEJSBridge.initCSI ---- Begin backtrace for nested exception. java.rmi.MarshalException: CORBA MARSHAL 1229123584 Maybe; nested exception is:. .org.omg.CORBA.MARSHAL: minor code: 4942F000 completed: Maybe. org.omg.CORBA.MARSHAL: minor code: 4942F000 completed: Maybe. .at java.lang.Class.newInstance0 .at java.lang.Class.newInstance .at com.ibm.rmi.util.Utility.readSystemException .at com.ibm.ws390.orb.ClientDelegate.invoke .at org.omg.CORBA.portable.ObjectImpl._invoke .at com.ibm.ws.management._ControlAdminService_Stub. activateProxyMBean .at com.ibm.ws.management.MBeanFactoryImpl.activateMBean .at com.ibm.ws.management.MBeanFactoryImpl.activateMBean .at com.ibm.ws.security.core.SecurityComponentImpl.initialize ... . BBOO0220E SECJ0007E: Error during security initialization. The exception is com.ibm.websphere.management.exception.. AdminException: ADMN0005E: Unable to activate MBean: type SecurityAdmin... <<< same stack trace with marshal exception above >>> ... BBOO0220E WSVR0003E: Server xxxxxx failed to start java.lang.RuntimeException: ADMN0005E: Unable to activate MBean: type SecurityAdmin... <<< same stack trace with marshal exception above >>> ... BBOO0220E WSVR0009E: Error occurred during startup . In the controller region (CR) numerous lines of: java.lang.OutOfMemoryError . The servant region never gets the usual messages before it ends: BBOO0222I WSVR0001I: Server SERVANT PROCESS xxxx open for e-business and BBOO0020I INITIALIZATION COMPLETE FOR WEBSPHERE FOR Z/OS SERVANT PROCESS xxxxxx. The absence of these messages combined with the numerous marshal exceptions and out of memory exceptions is a match for this apar. . With W502001, it is ok to see ONE set of the marshal exception combined with an out of memory exception during startup. If the servant region remains up and gets open for business, the presence of the messages is ok. . Even with a fix for PQ84216, you will still see ONE pair of the java.lang.OutOfMemoryError (in CR) and org.omg.CORBA.MARSHAL (in SR) exceptions during SR initialization. The java.lang.OutOfMemoryError is recoverable and has no relevance to the java heap size of the server. The java heap size of the server should not be adjusted when this exception is seen while the SR is initializing. (3/16/2004 update - apars PQ85550 (MD19208) and PQ86116 will remove the remaining expected java.lang.OutOfMemoryError and java.rmi.MarshalException exceptions. With these two apars we will no longer issue these exceptions in expected conditions.)Local fix A ++APAR is now available as AQ84216 on top of service level W502001. PQ84216 will ship in service level W502002. Contact L2 WebSphere to obtain the ++apar AQ84216. . There is a non-code work around available if it is a viable alternative for customers. This may not be a viable alternative for all customers. Since this problem is only exposed in multi-home configurations, the environment variable protocol_iiop_listenIPAddress can be set to a specific ip address. This env var can be set for any and all servers experiencing the problem reported in this apar. This will effectively make the configuration no longer multi-homed. We understand this is not a permanent fix and may not be acceptable to all customers but is an option as a temporary workaround until a code temp fix is available. . The environment variable protocol_iiop_listenIPAddress can be temporarily set in was.env of the server experiencing the problem and restart the server to see if this solves the problem. . To make the value for protocol_iiop_listenIPAddress permanent, use the admin console and set the following fields (depending on whether the configuration is base app server or ND): . App server (Base appserver or ND): Servers -> Application Servers -> <yourServerName> -> End Points -> ORB_LISTENER_ADDRESS . Deployment Manager (ND): System Administration -> Deployment Manager -> dmgr -> End Points -> ORB_LISTENER_ADDRESS . When the ++apar or PTF is available, you can reset these values to what you had before.Problem summary **************************************************************** * USERS AFFECTED: All users of WebSphere Application Server * * V5.0 for z/OS * **************************************************************** * PROBLEM DESCRIPTION: On Servant Region bring-up in a * * multi-home environment, initial * * Discovery between Controller Region and * * Servant Region succeeds, followed by * * MARSHAL exceptions on all subsequent * * activateProxyMBean method calls, with * * the Servant Region terminating after * * the Security service fails to start * * successfully. * **************************************************************** * RECOMMENDATION: * **************************************************************** Servant Region initialization in a multi-home environment (especially where a VIPA is present), exposed a path where a partial object key was used to drive the activateMBeanProxy method from the Controller Region to the Servant Region. Use of this partial key caused execution of a code path that prevented correct interpretation of partner version in the Controller Region, resulting in a series of CORBA::MARSHAL exceptions, java.lang.OutOfMemory exceptions, and the inability of several services required by the Servant Region to correctly initialize.Problem conclusion Guards were put in place to ensure correct handling and propagation of partner version contexts, whenever present. APAR PQ84216 is associated with SERVICE LEVEL W502002 of WebSphere Application Server V5.0 for z/OS.Temporary fix Comments
APAR is sysrouted FROM one or more of the following: APAR is sysrouted TO one or more of the following: Modules/Macros
Publications Referenced
|
Document Information |
Current web document: swg1PQ84216.html
Product categories: Software > Application Servers >
Distributed Application & Web Servers > WebSphere Application
Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PQ84216
IBM Group: Software Group
Modified date: Mar 16, 2004
(C) Copyright IBM Corporation 2000, 2009. All Rights Reserved.