PQ97206: DEADLOCK IN PMI WHEN SETINSTRUMENTATIONLEVEL AND AN EJB METHOD ARE INVOKED AT THE SAME TIME.

APAR status
Closed as program error.

Error description
Problem appears as a deadlock between a
Servlet.Engine.Transports thread and an ORB.thread.pool thread.

Something similar to the following will appear in the thread
dump:

FOUND A JAVA LEVEL DEADLOCK:
----------------------------
"Servlet.Engine.Transports:190":
  waiting to lock monitor 0x1d1058 (object 0xa69f3e98, a
com.ibm.ws.pmi.server.modules.BeanModule$BeanMethodsModule),
  which is locked by "ORB.thread.pool:2"
"ORB.thread.pool:2":
  waiting to lock monitor 0x1974a0 (object 0xf7a27aa8, a
java.lang.Class),
  which is locked by "Servlet.Engine.Transports:190"

The deadlock is intermittent and not readily reproducible.


Technical description of why it occurs:
When the level is not "max", PMI does not create submodule for
an EJB method under "methods". When the level is set to "max"
(via setInstrumentationLevel), PMI will set the level and in
addition create the PMI modules for all the EJB methods that are
invoked previously when the level is not "max". As shown in the
thread dump, the
BeanModule$BeanMethodsModule.setInstrumentationLevel method
tries to create the PMI module for an EJB method, which
eventually waits to lock at PmiRegistry.registerModule. At the
same time, if an EJB method in the same EJB is invoked and no
PMI module is created for this method yet, the code path in PMI
may lock PmiRegistry.registerModule and wait on
BeanModule$BeanMethodsModule.add method. This results in a
deadlock.


Technical details of the fix:
Do not create PMI modules for EJB methods when
setInstrumentationLevel is called. The PMI module for an EJB
method will be created when the method is called after "max"
level is set. This will avoid the deadlock. Although some PMI
modules may be delayed to create, users will not lose any PMI
data. Because PMI data for each method will not be updated until
the method is called after "max" level is set.  The related PMI
module will be created and related data will be updated at the
calling time.
Local fix Problem summary
****************************************************************
* USERS AFFECTED: Resource analyzer users who set level to MAX *
*                 and at the same time another user calls EJB  *
*                 methods. Depending on a certain order, a     *
*                 deadlock may occur.                          *
****************************************************************
* PROBLEM DESCRIPTION: When the level is not "max", PMI does   *
*                      not create submodule for an EJB method  *
*                      under "methods". When the level is set  *
*                      to "max" (via setInstrumentationLevel)  *
*                      PMI will set the level and in addition  *
*                      create the PMI modules for all the EJB  *
*                      methods that are invoked previously     *
*                      when the level is not "max". The        *
*                      BeanModule$BeanMethodsModule            *
*                      setInstrumentationLevel method          *
*                      tries to create the PMI module for      *
*                      an EJB method, which eventually waits   *
*                      to lock at PmiRegistry.registerModule.  *
*                      At the same time, if an EJB method in   *
*                      the same EJB is invoked and no PMI      *
*                      module is created for this method yet,  *
*                      the code path in PMI may lock           *
*                      PmiRegistry.registerModule and wait on  *
*                      BeanModule$BeanMethodsModule.add        *
*                      method.                                 *
*                      This results in a deadlock.             *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Deadlock may occur when multiple users set level to MAX and call
EJB methods at the same time.
Problem conclusion
Do not create PMI modules for EJB methods when
setInstrumentationLevel is called. The PMI module for an EJB
method will be created when the method is called after "max"
level is set. This will avoid the deadlock. Although some PMI
modules may be delayed to create, users will not lose any PMI
data. Because PMI data for each method will not be updated
until the method is called after "max" level is set. The related
PMI module will be created and related data will be updated at
the calling time.

The fix for this APAR is available via iFix PQ97206
Temporary fix Comments
APAR information
APAR number PQ97206
Reported component name WEBSPHERE AE SO
Reported component ID 5630A2202
Reported release 400
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Submitted date 2004-11-15
Closed date 2004-11-30
Last modified date 2004-11-30

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros

Fix information

Applicable component levels
R400 PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 400
Software edition:
Reference #: PQ97206
IBM Group: Software Group
Modified date: Nov 30, 2004