APAR status
Closed as program error.
Error description
The root of the deadlock is in the following method:
com.ibm.ejs.jms.JMSManagedConnection.onException
One can identify the deadlock in the native_stdout.log, where
two stacks showing exceptions from:
com.ibm.mq.jms.MQQueueAgentThread.run
flowing up through the onException method and attempting to
cleanup will indicate locks on objects that result in a classic
AB BA deadlock condition. Please see the defect for an example
of the two stacks.
The effect this can have on the AppServer could be an ouright
hang. It will never recover from this condition.
Local fix
The problem only arises with MDBs that have more than one
Session consuming messages, i.e. the "Maximum sessions" property
of the listener port is set to greater than 1 (the default).
Therefore, reducing the maximum sessions to 1 will prevent the
deadlock.
Problem summary
****************************************************************
* USERS AFFECTED: This problem affects users of MDBs that *
* have set "Maximum sessions" on the listener *
* port to greater than 1. It is a timing *
* issue, so not all such users will be *
* affected. *
****************************************************************
* PROBLEM DESCRIPTION: In response to an exception, the *
* Sessions used by the MDB throw an *
* exception to the JMSManagedConnection *
* which in turn tires to cleanup. In *
* doing so locks on two different *
* objects are required in a manner that *
* can cause deadlock if more than one *
* thread executes the onException() *
* method, i.e. if more than 1 Session *
* is being used to consume messages. *
* *
* The following stacks demonstrate the *
* problem: *
* *
* at com.ibm.ejs.jms. *
* JMSManagedConnection.handleClosed() *
* - waiting to lock <0xcb92aff8> *
* at com.ibm.ejs.jms. *
* JMSConnectionHandle.close() *
* - locked <0xcb9022b8> *
* at com.ibm.ejs.jms.listener. *
* MDBListenerImpl.internalStop() *
* at com.ibm.ejs.jms.listener. *
* JMSExceptionListener.onException() *
* at com.ibm.ejs.jms. *
* JMSManagedConnection.onException() *
* at com.ibm.mq.jms. *
* MQConnection.deliverException() *
* at com.ibm.mq.jms. *
* MQQueueAgent.deliverException() *
* at com.ibm.mq.jms. *
* MQQueueAgentThread.run() *
* at java.lang.Thread.run() *
* *
* *
* at com.ibm.ejs.jms. *
* JMSConnectionHandle.invalidate() *
* - waiting to lock <0xcb9022b8> *
* at com.ibm.ejs.jms. *
* JMSManagedConnection.cleanup() *
* - locked <0xcb92aff8> *
* at com.ibm.ejs.j2c. *
* MCWrapper.cleanup() *
* at com.ibm.ejs.j2c.poolmanager. *
* FreePool.cleanupAndDestroyMCWrapper() *
* at com.ibm.ejs.j2c.poolmanager. *
* FreePool.returnToFreePool() *
* .... *
* at com.ibm.ejs.jms. *
* JMSManagedConnection.connectionErr.. *
* - locked <0xcb92afc8> *
* at com.ibm.ejs.jms. *
* JMSManagedConnection.onException() *
* at com.ibm.mq.jms. *
* MQConnection.deliverException() *
* at com.ibm.mq.jms. *
* MQQueueAgent.deliverException() *
* at com.ibm.mq.jms. *
* MQQueueAgentThread.run() *
* at java.lang.Thread.run() *
****************************************************************
* RECOMMENDATION: *
****************************************************************
When this deadlock occurs, message consumption will hang.
Normally, the listener port restarts in 60 seconds, but this
will not happen. This problem can have follow on effects, which
may eventually lead to an outright hang of the Application
Server, which may seem to be problems in the J2C code of other
connectors such as JDBC drivers.
The stacks showing this problem are often dumped into the
native_stdout.log.
Problem conclusion
The fix uses a semaphore to ensure only one thread can execute
the code of JMSManagedConnection.onException() at a time.
Additionally, it is realized that there is no need to process
more than 1 Exception from the Sessions belonging to the MDBs
Connection because processing 1 or more results in the same
actions being taken. Therefore, now we only process one to
save CPU cycles.
The fix is targetted for cumulative fixes:
- V5.1.1.4
- V5.0.2.10
- V6.0.0.2
Please refer to the Recommended Updates page for delivery dates:
http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP
&uid=swg27004980
Please refer to the Recommended Updates page for delivery
dates.
Temporary fix Comments
APAR information |
APAR number |
PQ99224 |
Reported component name |
WAS NETWRK DEPL |
Reported component ID |
5630A3601 |
Reported release |
00S |
Status |
CLOSED PER |
PE |
NoPE |
HIPER |
NoHIPER |
Special Attention |
NoSpecatt |
Submitted date |
2005-01-09 |
Closed date |
2005-01-13 |
Last modified date |
2005-01-13 |
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Modules/Macros
Publications Referenced
Applicable component levels |
R003 PSY |
UP |
R00A PSY |
UP |
R00H PSY |
UP |
R00I PSY |
UP |
R00P PSY |
UP |
R00S PSY |
UP |
R00W PSY |
UP |
R103 PSY |
UP |
R10A PSY |
UP |
R10H PSY |
UP |
R10I PSY |
UP |
R10P PSY |
UP |
R10S PSY |
UP |
R10W PSY |
UP |
|