PQ99966: CONCURRENTMODIFICATIONEXCEPTION IN NODEAGENT SYSTEMOUT.LOG FILE.

 Fixes are available

PQ99966; 5.1.1.3, 5.1.1.4: Abstract: CONCURRENTMODIFICATIONEXCEPTION
Required WebSphere interim fixes for WebSphere Portal version 5.1.0.3
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Windows
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.1.1.6: WebSphere Application Server Version 5.1.1 Cumulative Fix 6
5.1.1.7: WebSphere Application Server Version 5.1.1 Cumulative Fix 7
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.1.1.8: WebSphere Application Server 5.1.1 Cumulative Fix 8 for Linux
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
Exception in nodeagent SystemOut.log file.

java.util.ConcurrentModificationException
 at java.util.HashMap$HashIterator.next(HashMap.java:731)
 at com.ibm.ws.management.event.DownstreamServerManager.
    updateServerListeners(DownstreamServerManager.java:223)
 at
com.ibm.ws.management.event.DownstreamServerManager.setFilter
    (DownstreamServerManager.java:201)
 at com.ibm.ws.management.event.ClientNotificationService.

addClientListenerInternal(ClientNotificationService.java:132)
 at com.ibm.ws.management.event.ClientNotificationService.

addNotificationListenerExtended(ClientNotificationService.java:1
12)
 at com.ibm.ws.management.AdminServiceImpl.
    addNotificationListenerExtended(AdminServiceImpl.java:847)
 at com.ibm.ws.management.nodeagent.PidWaiter.
    listenForJ2EEStateStoppingNotifications(PidWaiter.java:108)
 at
com.ibm.ws.management.nodeagent.PidWaiter.<init>(PidWaiter.java:
93)
 at com.ibm.ws.management.nodeagent.NodeAgent.monitorProcess
    (NodeAgent.java:951)
 at com.ibm.ws.management.nodeagent.NodeAgent.handleNotification
    (NodeAgent.java:862)
 at com.ibm.ws.management.event.ListenerInfo.handleNotification
    (ListenerInfo.java:67)
 at com.ibm.ws.management.event.
    NotificationDispatcher$DispatchANotificationToAListener.run
    (NotificationDispatcher.java:334)
 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:672)

Exception shows a concurrency issue in notification.  More
specifically, as each
running application server is discovered by the node agent, it
is added to a list of downstream servers maintained by
notification logic.  A
ConcurrentModificationException would occur when multiple thread
tries to update this list at the same time.  This problem
probably is not
always reproducible as it is dependent on the timing.  In
earlier releases of 5.0.2.x, where it took more time to process
notifications,
this problem would not likely to happen.  In the later 5.0.2.x
releases, where notifications are processed faster, this problem
is more likely to show up.
 -
When this exception is seen there is a delay stopping
applications. When the stop is issued from the admin
console there will be a 3 minute delay between when the
first app server stops the application and when the
next app server stops it.
Local fix
Not Available
Problem summary
****************************************************************
* USERS AFFECTED: Websphere application server version 5.0.2   *
*                 and 5.1.X users who try to stop and start    *
*                 the clusters and individual cluster members. *
*                 Users also face the problem when they try    *
*                 to stop application installed on clusters.   *
*                 Users might also face this problem also      *
*                 when there are concurrent administrators.    *
****************************************************************
* PROBLEM DESCRIPTION: 1.ConcurrentModificationException in    *
*                      nodeagent systemout.log                 *
*                      2.stop application(j2ee.state.stopped)  *
*                      notification is missed because of non   *
*                      availability of the listener.           *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Customers sees ConcurrentModification exception in their
SystemOut.log.

1.Problem: Client upgraded system from 5.0.2.2 to 5.0.2.9 and
now they see the nodeagents throw the following error due to
concurrent administrators

Customer sees the delay in there j2ee.stop.notification.while
trying to stop the applications. 2. this is nodeA running
dmgr,na and two cluster members, nodeB running na
and two more cluster members. Stopping an eApp running in the
members will cause a considerable delay (about 6 min) with one
eApp stopped in mem1, then mem2, then mem3 and so on.
Problem conclusion
1.Synchronized the block which was causing the error.

The fix will be included in 5.0.2.11 and 5.1.1.5
Please refer to the recommended updates page for delivery
information:

http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix Comments
APAR information
APAR number PQ99966
Reported component name WAS NETWRK DEPL
Reported component ID 5630A3601
Reported release 00S
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2005-01-25
Closed date 2005-03-24
Last modified date 2005-04-13

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:
PK04172

Modules/Macros
ADMINJMX          

Publications Referenced

Fix information
Fixed component name WAS NETWRK DEPL
Fixed component ID 5630A3601

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00S
Software edition:
Reference #: PQ99966
IBM Group: Software Group
Modified date: Apr 13, 2005