PQ97392: WHEN RESTARTING APP IN CLUSTER,OTHER CLUSTER-MEMBERS CREATE HEAPDUMPS, BECAUSE OF OOM WHEN PERFORMING EXTERNAL CACHE WRITES

 Fixes are available

PQ97392; 5.0.2.5: Heapdumps occur when restarting application in a cluster
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.1.1.6: WebSphere Application Server Version 5.1.1 Cumulative Fix 6
5.1.1.7: WebSphere Application Server Version 5.1.1 Cumulative Fix 7
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.4: WebSphere Application Server Version 5.1.1 Cumulative Fix 4
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Linux
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
After changing the sharing-policy to shared-push (with Dyna
Cache enabled) When restarting an app in a cluster, the other
cluster-members under load,create javacores and heapdumps.
It may be a replication problem.

According to javacores, the current thread at time the
oom occurs is performing external cache writes,

Java Stack
"DRSThreadPool : 923" (TID:0xB2388DB8, sys_thread_t:0x396D14B0,
state:R,
native ID:0x6B0D) prio=5
    at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
    at java.io.ObjectOutputStream.writeCanonical
    at java.io.ObjectOutputStream.writeInternal
    at java.io.ObjectOutputStream.write
    at java.io.ObjectOutputStream.write
    at com.ibm.ws.cache.CacheEntry.writeExternal
    at java.io.ObjectOutputStream.outputObject
    at java.io.ObjectOutputStream.writeObject
    at java.io.ObjectOutputStream.outputArray
    at
java.io.ObjectOutputStream.checkSubstitutableSpecialClasses
    at java.io.ObjectOutputStream.writeObject
    at java.io.ObjectOutputStream.outputClassFields
    at java.io.ObjectOutputStream.defaultWriteObject
    at java.io.ObjectOutputStream.outputObject
    at java.io.ObjectOutputStream.writeObject
    at com.ibm.ws.drs.DRSCacheMsg.writeExternal
    at java.io.ObjectOutputStream.outputObject
    at java.io.ObjectOutputStream.writeObject
    at com.ibm.disthub.impl.client.MessageImpl.setObjectBody
    at com.ibm.disthub.impl.jms.ObjectMessageImpl.setObject
    at com.ibm.ws.drs.DRSJMS.jmsPubUpd
    at com.ibm.ws.drs.DRSCacheApp.jmsPubUpd
    at com.ibm.ws.cache.drs.DRSNotificationService.bootstrap
    at com.ibm.ws.drs.DRSJMS.handleReceivedMessage
    at com.ibm.ws.drs.DRSReceiveMessageQueue$1.run
    at com.ibm.ws.util.ThreadPool$Worker.run
Local fix
No workaround
Problem summary
****************************************************************
* USERS AFFECTED: All WebSphere Application Server v502 & v511 *
*                 Dynamic Cache users                          *
****************************************************************
* PROBLEM DESCRIPTION: When restarting an application server   *
*                      in a clustered environment with         *
*                      Dynamic Cache enabled, other cluster    *
*                      members may fail due to an out of       *
*                      memory exception.                       *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
The following stack trace will be seen.  The key
element in this trace is DRSNotificationService.bootstrap.

Java Stack
"DRSThreadPool : 923" (TID:0xB2388DB8, sys_thread_t:0x396D14B0,
state:R,
native ID:0x6B0D) prio=5
at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
at java.io.ObjectOutputStream.writeCanonical
at java.io.ObjectOutputStream.writeInternal
at java.io.ObjectOutputStream.write
at java.io.ObjectOutputStream.write
at com.ibm.ws.cache.CacheEntry.writeExternal
at java.io.ObjectOutputStream.outputObject
at java.io.ObjectOutputStream.writeObject
at java.io.ObjectOutputStream.outputArray
at
java.io.ObjectOutputStream.checkSubstitutableSpecialClasses
at java.io.ObjectOutputStream.writeObject
at java.io.ObjectOutputStream.outputClassFields
at java.io.ObjectOutputStream.defaultWriteObject
at java.io.ObjectOutputStream.outputObject
at java.io.ObjectOutputStream.writeObject
at com.ibm.ws.drs.DRSCacheMsg.writeExternal
at java.io.ObjectOutputStream.outputObject
at java.io.ObjectOutputStream.writeObject
at com.ibm.disthub.impl.client.MessageImpl.setObjectBody
at com.ibm.disthub.impl.jms.ObjectMessageImpl.setObject
at com.ibm.ws.drs.DRSJMS.jmsPubUpd
at com.ibm.ws.drs.DRSCacheApp.jmsPubUpd
at com.ibm.ws.cache.drs.DRSNotificationService.bootstrap
at com.ibm.ws.drs.DRSJMS.handleReceivedMessage
at com.ibm.ws.drs.DRSReceiveMessageQueue$1.run
at com.ibm.ws.util.ThreadPool$Worker.run
Problem conclusion
This APAR will send cache content in blocks of either 50 CEs or
2% of CEs, whichever is greater ( 50 transmissions max ) during
the bootstrap process.

The fix for this problem is currently targetted for inclusion
in 5.0.2.10 and 5.1.1.3.
Please refer to the Recommended Updates page for delivery dates:
 
http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP
&uid=swg27004980
Temporary fix Comments
APAR information
APAR number PQ97392
Reported component name WAS BASE 5.0
Reported component ID 5630A3600
Reported release 00A
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-11-18
Closed date 2004-12-20
Last modified date 2004-12-20

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
DYNACACH          

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00A
Software edition:
Reference #: PQ97392
IBM Group: Software Group
Modified date: Dec 20, 2004