PQ87654: INTERMITTENT CONCURRENTMODIFICATIONEXCEPTION UNDER HEAVY LOAD

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.7: WebSphere Application Server Express 5.0.2 Cumulative Fix 7
5.1.0.5: WebSphere Application Server V5.1.0 Cumulative Fix 5
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.1.1.6: WebSphere Application Server Version 5.1.1 Cumulative Fix 6
5.1.1.7: WebSphere Application Server Version 5.1.1 Cumulative Fix 7
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.4: WebSphere Application Server Version 5.1.1 Cumulative Fix 4
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.8: WebSphere Application Server V5.0.2 Cumulative Fix 8
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.1.1.1: WebSphere Application Server Express 5.1.1 Cumulative Fix 1
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.1.1: WebSphere Application Server Version 5.1 Fix Pack 1 (Version 5.1.1)
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Linux
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.1: WebSphere Application Server Version 5.1.1 Cumulative Fix 1
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
When the application server is under heavy load on WebSphere 5.1
with memory-to-memory session clustering(client/server mode),
java.util.ConcurrentModificationException  appears
intermittently in the SystemOut.log file on the DRS servers.

The installation's SystemOut.log had a couple of different
exceptions but they contained the following stack trace snippet:

DRSCacheApp   E DRSW0008E: Exception is: java.util.
                           ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java)
at java.util.HashMap$EntryIterator.next(HashMap.java)
Local fix Problem summary
****************************************************************
* USERS AFFECTED: WebSphere Application Servers users using    *
*                 memory to memory replication                 *
****************************************************************
* PROBLEM DESCRIPTION: Exception is: java.util.                *
*                       ConcurrentModificationException        *
*                                                              *
*                      at java.util.HashMap$HashIterator.      *
*                       nextEntry(HashMap.java(Compiled Co     *
*                      at java.util.HashMap$EntryIterator.     *
*                       next(HashMap.java(Compiled Code))      *
*                      at java.util.HashMap.writeObject        *
*                       (HashMap.java(Compiled Code))          *
*                                                              *
*                      at sun.reflect.GeneratedMethod          *
*                       Accessor15.invoke(Unknown Source)      *
*                                                              *
*                      at sun.reflect.DelegatingMethod         *
*                                                              *
*                      AccessorImpl.invoke(DelegatingMethodA   *
*                                                              *
*                      at java.lang.reflect.Method.invoke      *
*                       (Method.java(Compiled Code))           *
*                      at java.io.ObjectStreamClass.           *
*                       invokeWriteObject                      *
*                       (ObjectStreamClass.jav                 *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeSerialData(ObjectOutputStream.jav  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeOrdinaryObject(ObjectOutputStream  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeObject0(ObjectOutputStream.java(C  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeObject(ObjectOutputStream.java(Co  *
*                                                              *
*                      at com.ibm.ws.drs.DRSCacheMsg.          *
*                       writeExternal(DRSCacheMsg.java:177)    *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeExternalData(ObjectOutputStream.j  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeOrdinaryObject(ObjectOutputStream  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeObject0(ObjectOutputStream.java(C  *
*                                                              *
*                      at java.io.ObjectOutputStream.          *
*                                                              *
*                      writeObject(ObjectOutputStream.java(Co  *
*                                                              *
*                      at com.ibm.disthub.impl.client.         *
*                                                              *
*                      MessageImpl.setObjectBody(MessageImpl   *
*                                                              *
*                      at com.ibm.disthub.impl.jms.            *
*                       ObjectMessageImpl.setObject            *
*                       (ObjectMessag                          *
*                      at com.ibm.ws.drs.DRSJMS.jmsAckGet      *
*                       (DRSJMS.java:981)                      *
*                      at com.ibm.ws.drs.DRSCacheApp.          *
*                       jmsAckGet(DRSCacheApp.java:3043)       *
*                                                              *
*                      at com.ibm.ws.webcontainer.httpsession. *
*                       DRSHttpSessCache.getEntry(DRS          *
*                      at com.ibm.ws.drs.DRSJMS.onMessage      *
*                       (DRSJMS.java(Compiled Code))           *
*                      at com.ibm.ws.drs.DRSCacheApp.          *
*                       onMessage(DRSCacheApp.java(Compiled    *
*                      C                                       *
*                      at com.ibm.disthub.impl.jms.            *
*                       MessageConsumerImpl.dispatch           *
*                       (MessageCons                           *
*                      at com.ibm.disthub.impl.jms.            *
*                      MessageImpl.dispatch(MessageImpl.java   *
*                       (Co                                    *
*                      at com.ibm.disthub.impl.jms.            *
*                       SessionDispatcher.run                  *
*                      (SessionDispatcher.                     *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Replicated data should be cloned before sending over to
another node.  This should address the concurrent modification
problem.
Problem conclusion
The code is fixed to address this problem.
Temporary fix
Fix uploaded to 
PQ99999.
Comments
APAR information
APAR number PQ87654
Reported component name WAS BASE 5.0
Reported component ID 5630A3600
Reported release 10A
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-04-16
Closed date 2004-05-17
Last modified date 2004-05-17

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
session          

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 10A
Software edition:
Reference #: PQ87654
IBM Group: Software Group
Modified date: May 17, 2004