PQ99224: Deadlock occurs during exception handling for MDB

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for AIX platforms
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for HP-UX platforms
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for OS/400 platforms
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.1.1.6: WebSphere Application Server Version 5.1.1 Cumulative Fix 6
5.1.1.7: WebSphere Application Server Version 5.1.1 Cumulative Fix 7
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.4: WebSphere Application Server Version 5.1.1 Cumulative Fix 4
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Solaris platforms
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
6.0.0.3: WebSphere Application Server V6.0.0 Fix Pack 3
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
PK09325; 5.1.1.4: Deadlock conditions after server sessions are returned to pool
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Windows platforms
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Linux platforms
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Linux
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
6.0.1.2: WebSphere Application Server Version 6.0.1 Fix Pack 2
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
PK02976; 5.1.1.3: JAVA Message service listener port deadlock
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
PK09325; 5.1.1.4: Deadlock conditions after server sessions are returned to pool
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
6.0.1: WebSphere Application Server Version 6.0 Refresh Pack 1
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
The root of the deadlock is in the following method:

com.ibm.ejs.jms.JMSManagedConnection.onException

One can identify the deadlock in the native_stdout.log, where
two stacks showing exceptions from:

com.ibm.mq.jms.MQQueueAgentThread.run

flowing up through the onException method and attempting to
cleanup will indicate locks on objects that result in a classic
AB BA deadlock condition.  Please see the defect for an example
of the two stacks.

The effect this can have on the AppServer could be an ouright
hang.  It will never recover from this condition.
Local fix
The problem only arises with MDBs that have more than one
Session consuming messages, i.e. the "Maximum sessions" property
of the listener port is set to greater than 1 (the default).
Therefore, reducing the maximum sessions to 1 will prevent the
deadlock.
Problem summary
****************************************************************
* USERS AFFECTED: This problem affects users of MDBs that      *
*                 have set "Maximum sessions" on the listener  *
*                 port to greater than 1.  It is a timing      *
*                 issue, so not all such users will be         *
*                 affected.                                    *
****************************************************************
* PROBLEM DESCRIPTION: In response to an exception, the        *
*                      Sessions used by the MDB throw an       *
*                      exception to the JMSManagedConnection   *
*                      which in turn tires to cleanup.  In     *
*                      doing so locks on two different         *
*                      objects are required in a manner that   *
*                      can cause deadlock if more than one     *
*                      thread executes the onException()       *
*                      method, i.e. if more than 1 Session     *
*                      is being used to consume messages.      *
*                                                              *
*                      The following stacks demonstrate the    *
*                      problem:                                *
*                                                              *
*                      at com.ibm.ejs.jms.                     *
*                      JMSManagedConnection.handleClosed()     *
*                      - waiting to lock <0xcb92aff8>          *
*                      at com.ibm.ejs.jms.                     *
*                      JMSConnectionHandle.close()             *
*                      - locked <0xcb9022b8>                   *
*                      at com.ibm.ejs.jms.listener.            *
*                      MDBListenerImpl.internalStop()          *
*                      at com.ibm.ejs.jms.listener.            *
*                      JMSExceptionListener.onException()      *
*                      at com.ibm.ejs.jms.                     *
*                      JMSManagedConnection.onException()      *
*                      at com.ibm.mq.jms.                      *
*                      MQConnection.deliverException()         *
*                      at com.ibm.mq.jms.                      *
*                      MQQueueAgent.deliverException()         *
*                      at com.ibm.mq.jms.                      *
*                      MQQueueAgentThread.run()                *
*                      at java.lang.Thread.run()               *
*                                                              *
*                                                              *
*                      at com.ibm.ejs.jms.                     *
*                      JMSConnectionHandle.invalidate()        *
*                      - waiting to lock <0xcb9022b8>          *
*                      at com.ibm.ejs.jms.                     *
*                      JMSManagedConnection.cleanup()          *
*                      - locked <0xcb92aff8>                   *
*                      at com.ibm.ejs.j2c.                     *
*                      MCWrapper.cleanup()                     *
*                      at com.ibm.ejs.j2c.poolmanager.         *
*                      FreePool.cleanupAndDestroyMCWrapper()   *
*                      at com.ibm.ejs.j2c.poolmanager.         *
*                      FreePool.returnToFreePool()             *
*                      ....                                    *
*                      at com.ibm.ejs.jms.                     *
*                      JMSManagedConnection.connectionErr..    *
*                      - locked <0xcb92afc8>                   *
*                      at com.ibm.ejs.jms.                     *
*                      JMSManagedConnection.onException()      *
*                      at com.ibm.mq.jms.                      *
*                      MQConnection.deliverException()         *
*                      at com.ibm.mq.jms.                      *
*                      MQQueueAgent.deliverException()         *
*                      at com.ibm.mq.jms.                      *
*                      MQQueueAgentThread.run()                *
*                      at java.lang.Thread.run()               *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
When this deadlock occurs, message consumption will hang.
Normally, the listener port restarts in 60 seconds, but this
will not happen.  This problem can have follow on effects, which
may eventually lead to an outright hang of the Application
Server, which may seem to be problems in the J2C code of other
connectors such as JDBC drivers.

The stacks showing this problem are often dumped into the
native_stdout.log.
Problem conclusion
The fix uses a semaphore to ensure only one thread can execute
the code of JMSManagedConnection.onException() at a time.
Additionally, it is realized that there is no need to process
more than 1 Exception from the Sessions belonging to the MDBs
Connection because processing 1 or more results in the same
actions being taken.  Therefore, now we only process one to
save CPU cycles.

The fix is targetted for cumulative fixes:

- V5.1.1.4
- V5.0.2.10
- V6.0.0.2

Please refer to the Recommended Updates page for delivery dates:
 
http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP
&uid=swg27004980

Please refer to the Recommended Updates page for delivery
dates.
Temporary fix Comments
APAR information
APAR number PQ99224
Reported component name WAS NETWRK DEPL
Reported component ID 5630A3601
Reported release 00S
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2005-01-09
Closed date 2005-01-13
Last modified date 2005-01-13

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
MSGING          

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00S
Software edition:
Reference #: PQ99224
IBM Group: Software Group
Modified date: Jan 13, 2005