PK13727: DEADLOCK IN NOTIFICATIONSERVICE THREADS

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
All the  threads in the NotificationDispatcher ThreadPool are
blocked waiting on a lock that another thread is holding and
hence cannot free up a slot in the BoundedBuffer.


The  javax.management.NotificationBroadcastSupport.
sendNotification call should not be holding onto this lock while
invoking listener.handleNotification.
Local fix Problem summary
****************************************************************
* USERS AFFECTED: Websphere Application Server users of        *
*                 version 5.1 and 5.02.                        *
****************************************************************
* PROBLEM DESCRIPTION: The server process hangs periodically   *
*                      due to a deadlock in Tivoli JMX code    *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A deadlock was detected in JMX notification service thread :

NotificationService Disptacher : 6289 (TID:0x3019D190) is
blocked on not being able to add to the ThreadPool
BoundedBuffer since it is already full. it is blocked
waiting for an empty slot. It is also however holding
a lock in
javax.management.NotificationBroadcastSupport.
sendNotification().

3XMTHREADINFO      "NotificationService dispatcher : 6289"
(TID:0x3019D190, sys_thread_t:0x47C3C2A0, state:CW, native
ID:0x6DE9)    prio=5
4XESTACKTRACE  at java.lang.Object.wait(Native Method)
4XESTACKTRACE          at java.lang.Object.wait
(Object.java(CompiledCode))
4XESTACKTRACE          at
com.ibm.ws.util.BoundedBuffer.put(BoundedBuffer.java
(Compiled Code))
4XESTACKTRACE          at
com.ibm.ws.util.ThreadPool.execute(ThreadPool.java
(Compiled Code))
4XESTACKTRACE          at
com.ibm.ws.util.ThreadPool.execute(ThreadPool.java
(Compiled Code))
4XESTACKTRACE          at
com.ibm.ws.management.event.
NotificationDispatcher$DispatchANotification
ToAllListeners.executeDispatchers
(NotificationDispatcher.java(Compiled
Code))
4XESTACKTRACE          at
com.ibm.ws.management.event.
NotificationDispatcher$DispatchANotification
ToAllListeners.
dispatch(NotificationDispatcher.java(Compiled Code))
4XESTACKTRACE          at
com.ibm.ws.management.event.
NotificationDispatcher.handleNotification(No
tificationDispatcher.java(Compiled Code))

4XESTACKTRACE          at
com.ibm.ws.management.event.
NotificationService.handleNotification(Notif
icationService.java(Compiled Code))

4XESTACKTRACE          at

com.ibm.ws.management.event.WsNotifDelegator.
handleNotification(WsNotifD
elegator.java(Compiled Code))
4XESTACKTRACE          at

com.ibm.ws.management.event.MBeanServerDelegateListener.
handleNotificati
on(MBeanServerDelegateListener.java(Compiled
4XESTACKTRACE          at

com.tivoli.jmx.ListenerRepository$ListenerProxy.
handleNotification(Liste
nerRepository.java(Compiled Code))

4XESTACKTRACE          at
javax.management.NotificationBroadcasterSupport.
sendNotification(NotificationBroadcasterSupport.java
(Compiled Code))

All the  threads in the NotificationDispatcher ThreadPool
are blocked waiting on the lock that 6289 is holding
and hence cannot free up a slot in the BoundedBuffer.



3XHSTACKLINE         at 0xD004FE70 in pthread_mutex_lock

3XHSTACKLINE         at 0xD09466DC in cooperative_mutex_lock

3XHSTACKLINE         at 0xD094632C in sysMonitorEnter

3XHSTACKLINE         at 0xD0CBD6B0 in lkMonitorEnter

3XHSTACKLINE         at 0xD3178018 in JITSigSegvHandler

The sendNotification call should not hold on to this lock
while it calls
listener.handleNotification(TID:0x3019D110,
sys_thread_t:0x47C03520,
state:MW, native ID:0x4EEA) prio=5

4XESTACKTRACE          at

javax.management.NotificationBroadcasterSupport.
sendNotification(NotificationBroadcasterSupport.java
(Compiled Code))
4XESTACKTRACE          at
com.tivoli.jmx.modelmbean.MMBNotificationBroadcaster.
sendNotification(MM
BNotificationBroadcaster.java(Compiled Code))

The javax.management.NotificationBroadcastSupport.
sendNotification call should not be holding onto this
lock while invoking listener.handleNotification.
Problem conclusion
The deadlock condition was avoided by altering the
synchronized block in Tivoli JMX code

The fix for this APAR is currently targeted for inclusion in
fixpack 5.0.2.17 and 5.1.1.11
 Please refer to the recommended updates page for delivery
information:

http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix Comments
APAR information
APAR number PK13727
Reported component name WAS NETWRK DEPL
Reported component ID 5630A3601
Reported release 10A
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2005-10-19
Closed date 2006-03-10
Last modified date 2006-03-10

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
ADMIN JMX        

Publications Referenced

Fix information
Fixed component name WAS NETWRK DEPL
Fixed component ID 5630A3601

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 10A
Software edition:
Reference #: PK13727
IBM Group: Software Group
Modified date: Mar 10, 2006