PQ93382: Hang situation between Control and Servant regions while startup | |||||||||||||||||||||||||||||||||||||||||||
![]() |
|||||||||||||||||||||||||||||||||||||||||||
![]() APAR status Closed as program error. Error description The server control and the servant get into a hang situation when a new servant has just been started the servant invokes ControlAdminService.startupServantJVM to alert JMX within the controller that it has another servant to manage. ControlAdminServiceImpl (within the controller) verifies that it can get a stub for the servant The controller then attempts to push a remote listener down to the servant.It appears that the servant is not accepting the request from the controller. In any event, the controller thread is blocked until the servant will accept this request. The server process may also fail to start or hang. The stack of the control region TCB bloceked looks like this.. ORB_Request::comm_outbound_ctl_local_request() ORB_Request::comm_outbound_request() CORBA::Request::invoke() ORBEJSBridge::invoke_request(JNIEnv_*,bboojorb*,char*,unsigned c ORBEJSBridge::build_and_invoke_request(JNIEnv_*,bboojorb*,char*, Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest(I[BIZI)[B com/ibm/ws390/orb/ClientDelegate.java com/ibm/ws390/orb/ClientDelegate.invoke org/omg/CORBA/portable/ObjectImpl._invoke com/ibm/ws390/management/connector/corba/_CorbaConnectorStub.add com/ibm/ws390/management/connector/corba/CorbaConnectorClient.ad com/ibm/ws390/management/InterProcessAdminService.addNotificatio com/ibm/ws/management/event/ZServantNotificationManager.addRemot com/ibm/ws/management/event/ZServantNotificationManager.createDS com/ibm/ws/management/event/ZServantNotificationManager.addDowns com/ibm/ws/management/event/ZServantListener.handleNotification com/tivoli/jmx/ListenerRepository$ListenerProxy.handleNotificati javax/management/NotificationBroadcasterSupport.sendNotification com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster.sendNotific com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport.send javax/management/modelmbean/RequiredModelMBean.sendNotification com/ibm/websphere/management/RuntimeCollaborator.sendNotificatio com/ibm/ws/management/ControlAdminServiceImpl.sendNotification com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke com/ibm/ws390/orb/CommonBridge.CORBAinvoke com/ibm/ws390/orb/ORBEJSBridge.CORBAinvokeLocal fix Problem summary **************************************************************** * USERS AFFECTED: All users of WebSphere Application Server * * V5.0 for z/OS * **************************************************************** * PROBLEM DESCRIPTION: Deadlock between controller and new * * servant. * **************************************************************** * RECOMMENDATION: * **************************************************************** A newly started servant invokes the startupServantJVM method on the ControlAdminServiceImpl object in the associated controller. The controller, before returning to the servant from the startupServantJVM method, issues a notification that a new servant has started. A couple of the listeners attempt to send a request back to the servant (while the MBean server is synchronized on the notifications listeners). In some cases all of the threads in the servant can be busy when this callback is made and each of these threads can invoke the controller to register an MBean proxy before becoming available for new work. Each of these servant threads will become blocked until the corresponding controller thread completes the MBean proxy registration, which cannot complete until the lock on the notification listeners is released. This results in a deadlock between the controller and servant. The stacktrace for the controller thread that is attempting the callback to the servant while synchronized on the notification listeners can be in two forms, depending on which notification listener is initiating the callback. In the case that ZServantListener is attempting to establish a notification listener in the servant the following stack trace will appear: ORB_Request::comm_outbound_ctl_local_request ORB_Request::comm_outbound_request CORBA::Request::invoke ORBEJSBridge::invoke_request ORBEJSBridge::build_and_invoke_request Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest com/ibm/ws390/orb/ClientDelegate.java com/ibm/ws390/orb/ClientDelegate.invoke org/omg/CORBA/portable/ObjectImpl._invoke com/ibm/ws390/management/connector/corba/_CorbaConnectorStub. addRMINotificationListener com/ibm/ws390/management/connector/corba/CorbaConnectorClient. addNotificationListener com/ibm/ws390/management/InterProcessAdminService. addNotificationListener com/ibm/ws/management/event/ZServantNotificationManager. addRemoteListener com/ibm/ws/management/event/ZServantNotificationManager. createDSMEntry com/ibm/ws/management/event/ZServantNotificationManager. addDownstreamProcess com/ibm/ws/management/event/ZServantListener.handleNotification com/tivoli/jmx/ListenerRepository$ListenerProxy. handleNotification javax/management/NotificationBroadcasterSupport.sendNotification com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster. sendNotification com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport. sendNotification javax/management/modelmbean/RequiredModelMBean.sendNotification com/ibm/websphere/management/RuntimeCollaborator. sendNotification com/ibm/ws/management/ControlAdminServiceImpl.sendNotification com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke com/ibm/ws390/orb/CommonBridge.CORBAinvoke com/ibm/ws390/orb/ORBEJSBridge.CORBAinvoke In the case that AppServerStatusCache is attempting to get attribute information from an MBean in the servant, the following stack trace will appear: ORB_Request::comm_outbound_ctl_local_request ORB_Request::comm_outbound_request CORBA::Request::invoke ORBEJSBridge::invoke_request ORBEJSBridge::build_and_invoke_request Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest com/ibm/ws390/orb/ClientDelegate.invoke org/omg/CORBA/portable/ObjectImpl._invoke com/ibm/ws390/management/connector/corba/_CorbaConnectorStub. invokeTypeName com/ibm/ws390/management/ServantMBeanInvoker.invokeNext com/ibm/ejs/jms/listener/ListenerPortMBeanProxy.isStarted INVOKDMY EXECJAVA mmipExecuteJava xeRunJvmMethod JVM_InvokeMethod Java_java_lang_reflect_Method_invoke java/lang/reflect/Method.invoke com/tivoli/jmx/modelmbean/MMBInvoker.invoke com/tivoli/jmx/modelmbean/MMBInvoker.getAttribute com/tivoli/jmx/modelmbean/DynamicModelMBeanSupport.getAttribute javax/management/modelmbean/RequiredModelMBean.getAttribute com/tivoli/jmx/GenericMBeanSupport.getAttribute com/tivoli/jmx/MBeanAccess.getAttribute com/tivoli/jmx/MBeanServerImpl.getAttribute com/ibm/ws/management/AdminServiceImpl.getAttribute mmipSelectInvokeSynchronizedJavaMethod com/ibm/ws/management/ status/AppServerStatusCache.refreshCache com/ibm/ws/management/status/AppServerStatusCache. handleNotification com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster. sendNotification com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport. sendNotification javax/management/modelmbean/RequiredModelMBean.sendNotification com/ibm/websphere/management/RuntimeCollaborator. sendNotification com/ibm/ws/management/ControlAdminServiceImpl.sendNotification com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke com/ibm/ws390/orb/CommonBridge.CORBAinvoke com/ibm/ws390/orb/ORBEJSBridge.CORBAinvokeProblem conclusion The notification listeners have been updated to not make callbacks to the servant while the MBean server is synchronized on the notification listeners. APAR PQ93382 is associated with SERVICE LEVEL W502018 of WebSphere Application Server V5.0 for z/OS.Temporary fix Comments
APAR is sysrouted FROM one or more of the following: PQ92409 APAR is sysrouted TO one or more of the following: PQ93391 Modules/Macros
Publications Referenced
|
Document Information |
Current web document: swg1PQ93382.html
Product categories: Software > Application Servers >
Distributed Application & Web Servers > WebSphere Application
Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PQ93382
IBM Group: Software Group
Modified date: Jan 21, 2005
(C) Copyright IBM Corporation 2000, 2009. All Rights Reserved.