PQ90889: Prevent deadlock when multiple transaction errors occur on connections in the same connection pool and trace is enabled.

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.8: WebSphere Application Server V5.0.2 Cumulative Fix 8
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
If trace is enabled and there are multiple transaction errors on
connections at the same time, then there is the possibility of a
deadlock occurring.
.
The java stack trace for the threads involved in the deadlock
will look like this:
This output is from a formatted Windows user.dmp file
(ip)=77F82870  at  ntdll.dll  ZwWaitForSingleObject + 000B
    unwinding (old_esp=3C32EC70 old_eip=77F82870)
    parms_on_stack=( 00000EFC, 00000000, 00000000 )
(ip)=7C573B28  at  KERNEL32.dll  WaitForSingleObjectEx + 005A
    unwinding (old_esp=3C32EC80 old_eip=7C573B28)
    unwinding (new_ebp=00000001) at 7C573B3D
    parms_on_stack=( 00000EFC, FFFFFFFF, 00000000 )
(ip)=7C573B50  at  KERNEL32.dll  WaitForSingleObject + 000F
    unwinding (old_esp=3C32ECA8 old_eip=7C573B50)
    parms_on_stack=( 00000EFC, FFFFFFFF )
(ip)=00262C46  at  hpi.dll  _sysMonitorWait + 00A7
    unwinding (old_esp=3C32ECB4 old_eip=00262C46)
    unwinding (new_ebp=00770000) at 00262CB6
(ip)=10081EDC  at  jvm.dll  _lkMonitorEnter + 01FC
    unwinding (old_esp=3C32ECC8 old_eip=10081EDC)
    unwinding (new_ebp=109B5A40) at 10081F49
(cc)=282A0EAF  at
com/ibm/ejs/j2c/poolmanager/PoolManager.toString
()Ljava/lang/String;  MMI2JIT
(cc)=2829E823  at
com/ibm/ejs/j2c/poolmanager/PoolManager.toString
()Ljava/lang/String;  MMI2JIT
    unwinding (old_esp=3C32ECFC old_eip=2829E823)
old_mb=2472A270
    unwinding (new_ebp=00000000) at 2829EA35
    unwinding (new_ebp=3C32ED70) at 2829EB60
(cc)=25A36DE4  at  com/ibm/ejs/ras/TraceEvent.convertParameters
(Ljava/lang/Object;)V
    unwinding (old_esp=3C32ED54 old_eip=25A36DE4)
old_mb=220B1600
    unwinding (new_ebp=105A7C18) at 25A3707B
(cc)=2EBCAF6B  at
com/ibm/ejs/j2c/poolmanager/PoolManager.release
(Lcom/ibm/ws/j2c/MCWrapper;Ljava/lang/Object;)V
    unwinding (old_esp=3C32ED94 old_eip=2EBCAF6B)
old_mb=24729B68
.
The key calls to look for in the java stack are
-- com/ibm/ejs/j2c/poolmanager/PoolManager.toString
-- com/ibm/ejs/ras/TraceEvent.convertParameters
-- com/ibm/ejs/j2c/poolmanager/PoolManager.release
.
Local fix
As it happens, the problem cannot occur unless debug tracing is
enabled in the com.ibm.ejs.j2c.poolmanager.PoolManager class.
So one workaround is to disable tracing for the WAS.j2c
component.
Alternatively, you could leave j2c tracing enabled, but
disable only the debug tracing in the Pool Manager class, by
adding the following to the Trace Specification AFTER the
j2c element in the tracespec:
com.ibm.ejs.j2c.poolmanager.PoolManager=debug=disabled

.
Due to an issue with the tracing infrastructure, the Component
style j2c tracespec should be used instead of the Group stylebu
j2c tracespec.  The combination of the two parameters would
look like:
com.ibm.ejs.j2c.*=all=enabled:
com.ibm.ejs.j2c.poolmanager.PoolManager=debug=disabled
Problem summary
****************************************************************
* USERS AFFECTED: WebSphere Application Server users of        *
*                 JDBC, JCA and JMS Connections.               *
****************************************************************
* PROBLEM DESCRIPTION: If transaction errors occur on more     *
*                      than one Connection at the same time    *
*                      the connection pool can deadlock        *
*                      while trying to print trace             *
*                      information.                            *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
If trace is enabled for WAS.j2c and connection errors occur on
more than one sharable connection at the same time, we can get
a deadlock while printing the pool contents.

The error only occurs when trace is enabled and there are a
lot of transaction errors (i.e. timeouts).
Problem conclusion
Provider heavier lock around the error case for after
completion. This allows only one thread to reach the toString
method at a time.
Temporary fix Comments
APAR information
APAR number PQ90889
Reported component name WAS BASE 5.0
Reported component ID 5630A3600
Reported release 00W
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-07-01
Closed date 2004-07-27
Last modified date 2004-09-14

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
J2C          

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00W
Software edition:
Reference #: PQ90889
IBM Group: Software Group
Modified date: Sep 14, 2004