PQ85848: A recursive call in the WLM code causes an out of memory or java segmentation fault in a workload managed environment.

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.7: WebSphere Application Server Express 5.0.2 Cumulative Fix 7
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.1.1.6: WebSphere Application Server Version 5.1.1 Cumulative Fix 6
5.1.1.7: WebSphere Application Server Version 5.1.1 Cumulative Fix 7
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.4: WebSphere Application Server Version 5.1.1 Cumulative Fix 4
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.8: WebSphere Application Server V5.0.2 Cumulative Fix 8
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.1.1.1: WebSphere Application Server Express 5.1.1 Cumulative Fix 1
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Linux
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.1: WebSphere Application Server Version 5.1.1 Cumulative Fix 1
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
PROBLEM:
Threads in a workload management environment can make many calls
that can cause recursion. The stack of the problem can be seen
below:

com.ibm.ws.wlm.client.selection.NoUsableTargetException
com.ibm.ws.wlm.client.selection.MasterProxy.getNextClone 262
Exception =
com.ibm.ws.wlm.client.selection.NoUsableTargetExceptionSource =
com.ibm.ws.wlm.client.selection.MasterProxy.getNextCloneprobeid
= 262 Stack Dump =
com.ibm.ws.wlm.client.selection.NoUsableTargetException
 at
com.ibm.ws.wlm.client.selection.WeightedOutstandingRequestTarget
Selector.getNextClone(WeightedOutstandingRequestTargetSelector.j
ava(Compiled Code))
 at
com.ibm.ws.wlm.client.selection.MasterProxy.getNextClone(MasterP
roxy.java(Compiled Code))
 at
com.ibm.ws.wlm.client.selection.SelectionManager.getTarget(Selec
tionManager.java(Compiled Code))
 at
com.ibm.ws.wlm.client.WLMClient.getNextTarget(WLMClient.java(Com
piled Code))
 at
com.ibm.ws.wlm.client.WLMClient.handleRemoteException(WLMClient.
java(Compiled Code))
 at
com.ibm.rmi.corba.ClientDelegate._createRequest_WLM(ClientDelega
te.java(Compiled Code))
 at
com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.ja
va(Compiled Code))
 at
com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.ja
va(Inlined Compiled Code))
 at
com.ibm.rmi.corba.ClientDelegate.request(ClientDelegate.java(Com
piled Code))
 at
org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java(Inlin
ed Compiled Code))
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
 at
com.ibm.WsnOptimizedNaming._NamingContextStub.get_context_id_str
ing(Unknown Source)(Compiled Code)
Local fix
Defect 168004 has been created that limits the number of
requests that a thread can make before it quits in a WLM
environment to prevent this recursion.
Problem summary
****************************************************************
* USERS AFFECTED: WebSphere Application Server users of WLM    *
*                 or Workload Managment.                       *
****************************************************************
* PROBLEM DESCRIPTION: A recursive call in the WLM code        *
*                      causes an out of memory or java         *
*                      segmentation fault in a workload        *
*                      managed environment.                    *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A recursive call in the WLM code causes an out of memory or
java segmentation fault in a workload managed environment.
Threads in a workload management environment can make calls
that can cause deep recursion resulting in the out of
memory error.

The stack below is an example of a sympton of the problem
that can be found in an ffdc log.  In this case though,
the out of memory error did not occur since the
NoUSableTargetException was eventually thrown.  The problem
occurs when the exception is not thrown.

com.ibm.ws.wlm.client.selection.MasterProxy.getNextClone 262
Exception =
com.ibm.ws.wlm.client.selection.NoUsableTargetException
Source =
com.ibm.ws.wlm.client.selection.MasterProxy.getNextClone
probeid = 262
Stack Dump =
com.ibm.ws.wlm.client.selection.NoUsableTargetException

at com.ibm.ws.wlm.client.selection
.WeightedOutstandingRequestTargetSelector
.getNextClone(WeightedOutstandingRequestTargetSelector
.java(Compiled Code))

at com.ibm.ws.wlm.client.selection.MasterProxy
.getNextClone(MasterProxy.java(Compiled Code))

at com.ibm.ws.wlm.client.selection.SelectionManager
.getTarget(SelectionManager.java(Compiled Code))

at com.ibm.ws.wlm.client.WLMClient
.getNextTarget(WLMClient.java(Compiled Code))

at com.ibm.ws.wlm.client.WLMClient
.handleRemoteException(WLMClient.java(Compiled Code))

at com.ibm.rmi.corba.ClientDelegate
._createRequest_WLM(ClientDelegate.java(Compiled Code))

at com.ibm.rmi.corba.ClientDelegate
.createRequest(ClientDelegate.java(Compiled Code))

at com.ibm.rmi.corba.ClientDelegate
.createRequest(ClientDelegate.java(Inlined Compiled Code))

at com.ibm.rmi.corba.ClientDelegate
.request(ClientDelegate.java(Compiled Code))

at org.omg.CORBA.portable.ObjectImpl
._request(ObjectImpl.java(Inlined Compiled Code))

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)

at com.ibm.WsnOptimizedNaming._NamingContextStub
.get_context_id_string(Unknown Source)(Compiled Code)
...
...
Problem conclusion
A check needed to be made to limit the depth of the calls
Temporary fix
When recycling the application servers and node agents,
stop the deployment manager last.  Then restart the
deployment manager first before restarting any of the node
agents or application servers.
Comments
APAR information
APAR number PQ85848
Reported component name WAS NETWRK DEPL
Reported component ID 5630A3601
Reported release 00A
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-03-10
Closed date 2004-07-21
Last modified date 2004-07-21

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
WLM          

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00A
Software edition:
Reference #: PQ85848
IBM Group: Software Group
Modified date: Jul 21, 2004