PQ93382: Hang situation between Control and Servant regions while startup

 A fix is available

6.0.1: WebSphere Application Server Version 6.0 Refresh Pack 1



APAR status
Closed as program error.

Error description
The server control and the servant get into a hang situation
when a new servant has just been started the servant invokes
ControlAdminService.startupServantJVM to alert JMX within the
controller that it has another servant to manage.
ControlAdminServiceImpl (within the controller) verifies that it
can get a stub for the servant The controller then attempts to
push a remote listener down to the servant.It appears that the
servant is not accepting the request from the controller. In any
event, the controller thread is blocked until the servant will
accept this request. The server process may also fail to start
or hang.
The stack of the control region TCB bloceked looks like this..
ORB_Request::comm_outbound_ctl_local_request()
ORB_Request::comm_outbound_request()
CORBA::Request::invoke()
ORBEJSBridge::invoke_request(JNIEnv_*,bboojorb*,char*,unsigned c
ORBEJSBridge::build_and_invoke_request(JNIEnv_*,bboojorb*,char*,
Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest
com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest(I[BIZI)[B
com/ibm/ws390/orb/ClientDelegate.java
com/ibm/ws390/orb/ClientDelegate.invoke
org/omg/CORBA/portable/ObjectImpl._invoke
com/ibm/ws390/management/connector/corba/_CorbaConnectorStub.add
com/ibm/ws390/management/connector/corba/CorbaConnectorClient.ad
com/ibm/ws390/management/InterProcessAdminService.addNotificatio
com/ibm/ws/management/event/ZServantNotificationManager.addRemot
com/ibm/ws/management/event/ZServantNotificationManager.createDS
com/ibm/ws/management/event/ZServantNotificationManager.addDowns
com/ibm/ws/management/event/ZServantListener.handleNotification
com/tivoli/jmx/ListenerRepository$ListenerProxy.handleNotificati
javax/management/NotificationBroadcasterSupport.sendNotification
com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster.sendNotific
com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport.send
javax/management/modelmbean/RequiredModelMBean.sendNotification
com/ibm/websphere/management/RuntimeCollaborator.sendNotificatio
com/ibm/ws/management/ControlAdminServiceImpl.sendNotification
com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM
com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke
com/ibm/ws390/orb/CommonBridge.CORBAinvoke
com/ibm/ws390/orb/ORBEJSBridge.CORBAinvoke
Local fix Problem summary
****************************************************************
* USERS AFFECTED: All users of WebSphere Application Server    *
*                 V5.0 for z/OS                                *
****************************************************************
* PROBLEM DESCRIPTION: Deadlock between controller and new     *
*                      servant.                                *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A newly started servant invokes the startupServantJVM method on
the ControlAdminServiceImpl object in the associated controller.
The controller, before returning to the servant from the
startupServantJVM method, issues a notification that a new
servant has started. A couple of the listeners attempt to send
a request back to the servant (while the MBean server is
synchronized on the notifications listeners). In some cases all
of the threads in the servant can be busy when this callback is
made and each of these threads can invoke the controller to
register an MBean proxy before becoming available for new work.
Each of these servant threads will become blocked until the
corresponding controller thread completes the MBean proxy
registration, which cannot complete until the lock on the
notification listeners is released. This results in a deadlock
between the controller and servant.

The stacktrace for the controller thread that is attempting the
callback to the servant while synchronized on the notification
listeners can be in two forms, depending on which notification
listener is initiating the callback. In the case that
ZServantListener is attempting to establish a notification
listener in the servant the following stack trace will appear:

ORB_Request::comm_outbound_ctl_local_request
ORB_Request::comm_outbound_request
CORBA::Request::invoke
ORBEJSBridge::invoke_request
ORBEJSBridge::build_and_invoke_request
Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest
com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest
com/ibm/ws390/orb/ClientDelegate.java
com/ibm/ws390/orb/ClientDelegate.invoke
org/omg/CORBA/portable/ObjectImpl._invoke
com/ibm/ws390/management/connector/corba/_CorbaConnectorStub.
    addRMINotificationListener
com/ibm/ws390/management/connector/corba/CorbaConnectorClient.
    addNotificationListener
com/ibm/ws390/management/InterProcessAdminService.
    addNotificationListener
com/ibm/ws/management/event/ZServantNotificationManager.
    addRemoteListener
com/ibm/ws/management/event/ZServantNotificationManager.
    createDSMEntry
com/ibm/ws/management/event/ZServantNotificationManager.
    addDownstreamProcess
com/ibm/ws/management/event/ZServantListener.handleNotification
com/tivoli/jmx/ListenerRepository$ListenerProxy.
    handleNotification
javax/management/NotificationBroadcasterSupport.sendNotification
com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster.
    sendNotification
com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport.
    sendNotification
javax/management/modelmbean/RequiredModelMBean.sendNotification
com/ibm/websphere/management/RuntimeCollaborator.
    sendNotification
com/ibm/ws/management/ControlAdminServiceImpl.sendNotification
com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM
com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke
com/ibm/ws390/orb/CommonBridge.CORBAinvoke
com/ibm/ws390/orb/ORBEJSBridge.CORBAinvoke

In the case that AppServerStatusCache is attempting to get
attribute information from an MBean in the servant, the
following stack trace will appear:

ORB_Request::comm_outbound_ctl_local_request
ORB_Request::comm_outbound_request
CORBA::Request::invoke
ORBEJSBridge::invoke_request
ORBEJSBridge::build_and_invoke_request
Java_com_ibm_ws390_orb_ClientDelegate_jorbInvokeRequest
com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest
com/ibm/ws390/orb/ClientDelegate.invoke
org/omg/CORBA/portable/ObjectImpl._invoke
com/ibm/ws390/management/connector/corba/_CorbaConnectorStub.
    invokeTypeName
com/ibm/ws390/management/ServantMBeanInvoker.invokeNext
com/ibm/ejs/jms/listener/ListenerPortMBeanProxy.isStarted
INVOKDMY
EXECJAVA
mmipExecuteJava
xeRunJvmMethod
JVM_InvokeMethod
Java_java_lang_reflect_Method_invoke
java/lang/reflect/Method.invoke
com/tivoli/jmx/modelmbean/MMBInvoker.invoke
com/tivoli/jmx/modelmbean/MMBInvoker.getAttribute
com/tivoli/jmx/modelmbean/DynamicModelMBeanSupport.getAttribute
javax/management/modelmbean/RequiredModelMBean.getAttribute
com/tivoli/jmx/GenericMBeanSupport.getAttribute
com/tivoli/jmx/MBeanAccess.getAttribute
com/tivoli/jmx/MBeanServerImpl.getAttribute
com/ibm/ws/management/AdminServiceImpl.getAttribute
mmipSelectInvokeSynchronizedJavaMethod com/ibm/ws/management/
    status/AppServerStatusCache.refreshCache
com/ibm/ws/management/status/AppServerStatusCache.
    handleNotification
com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster.
    sendNotification
com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport.
    sendNotification
javax/management/modelmbean/RequiredModelMBean.sendNotification
com/ibm/websphere/management/RuntimeCollaborator.
    sendNotification
com/ibm/ws/management/ControlAdminServiceImpl.sendNotification
com/ibm/ws/management/ControlAdminServiceImpl.startupServantJVM
com/ibm/ws/management/_ControlAdminServiceImpl_Tie._invoke
com/ibm/ws390/orb/CommonBridge.CORBAinvoke
com/ibm/ws390/orb/ORBEJSBridge.CORBAinvoke
Problem conclusion
The notification listeners have been updated to not make
callbacks to the servant while the MBean server is synchronized
on the notification listeners.

APAR PQ93382 is associated with SERVICE LEVEL W502018 of
WebSphere Application Server V5.0 for z/OS.
Temporary fix Comments
APAR information
APAR number PQ93382
Reported component name WEBSPHERE FOR Z
Reported component ID 5655I3500
Reported release 500
Status CLOSED PER
PE NoPE
HIPER YesHIPER
Special Attention NoSpecatt
Submitted date 2004-08-26
Closed date 2004-11-12
Last modified date 2005-01-21

APAR is sysrouted FROM one or more of the following:
PQ92409

APAR is sysrouted TO one or more of the following:
PQ93391

Modules/Macros
BBOUBINF          

Publications Referenced

Fix information
Fixed component name WEBSPHERE FOR Z
Fixed component ID 5655I3500

Applicable component levels
R500 PSY UQ95030    UP04/11/18 P F411

  Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.


Document Information


Current web document: swg1PQ93382.html
Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PQ93382
IBM Group: Software Group
Modified date: Jan 21, 2005