PQ51338: THE ADMIN CONSOLE HANGS.


APAR

APAR status
Closed as program error.

Error description
Problem:***The Admin Console hangs.
Recap:Problem:***The Admin Console hangs.
***AIX/WAS 3.5.4 ***Client/Admin server/JVM local. ***Restarting the adminconsole fully recovers the problem. ***The heap(s) of the other JVM's (nanny, appserver) did not grow. ***The JVM (of the consle) grew slightly (by +3mB over a 20 hour test period). ***The system wide allocate virtual memory didn't increase. ***There were no more messages seen in the message window. ***The (servlet) engine continued to perform the auto reload. ***Attempting to change an object through the console rulted in an entirely grey foreground/background (at this point I got the java.lang.OutOfMemoryError exceptions thrown). ***No errors in the system log. ***The adminclient debug trace file gives no clues. It shows the log messages to the client up until the console hangs, then only Act waiting for notification and BackgroundLru going to sleep and waking up. ***Thread dumps of the client before and after the hang are exactly the same (the same running threads) except the SeriousEventMessagePanelUpdater thread is present before the hang. Description:Recap:***AIX/WAS 3.5.4***Client/Admin server/JVM local.***Restarting the adminconsole fully recovers the problem.***The heap(s) of the other JVM's (nanny, appserver) didnot grow.***The JVM (of the consle) grew slightly (by +3mB over a20 hour test period).***The system wide allocate virtual memory didn't increase.***There were no more messages seen in the message window.***The (servlet) engine continued to perform the auto reload.***Attempting to change an object through the console rultedin an entirely grey foreground/background (at this pointI got the java.lang.OutOfMemoryError exceptions thrown).***No errors in the system log.***The adminclient debug trace file gives no clues. It showsthe log messages to the client up until the console hangs,then only Act waiting for notification and BackgroundLrugoing to sleep and waking up.***Thread dumps of the client before and after the hang areexactly the same (the same running threads) except theSeriousEventMessagePanelUpdater thread is present beforethe hang.
Run the following script in AIX - while true do touch SnoopServlet.class echo touch to SnoopServlet.class ls -al SnoopServlet.class sleep 60 done This will modify the time stamp on the snoop class. From the console set the auto reload for the default_app to TRUE, interval 60 sec., and for the snoop servlet set load at startup to true. Start the app server (Default Server), the auto reload will generate messages that are written to the 'Console Messages' window in the console. After time, in the test about 5 hours, the messages will stop appearing in the window and the console will hang. This problem was reproduced by an IBM Japan team and by the WebSphere team in Raleigh.
Description:Run the following script in AIX -while truedotouch SnoopServlet.classecho touch to SnoopServlet.classls -al SnoopServlet.classsleep 60doneThis will modify the time stamp on the snoop class. From theconsole set the auto reload for the default_app to TRUE,interval 60 sec., and for the snoop servlet set load atstartup to true. Start the app server (Default Server),the auto reload will generate messages that are written tothe 'Console Messages' window in the console. After time,in the test about 5 hours, the messages will stop appearingin the window and the console will hang. This problem wasreproduced by an IBM Japan team and by the WebSphere teamin Raleigh.
Local fix
Restart the Admin Console
Problem summary
****************************************************************
* USERS AFFECTED: All WebSphere Application Server 3.5         *
*                 Advanced Edition users of the Administrative *
*                 Console                                      *
****************************************************************
* PROBLEM DESCRIPTION: Administrative Console can be           *
*                      overwhelmed by a large volume of        *
*                      Serious Event messages, becoming        *
*                      unusable                                *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
After the RAS component logs warnings and errors in the
activity log, it makes them available to the server (either
Administrative server or Application server) to be logged in
the Serious Event table of the admin database -- this is done
by the SeriousEventListener in each server. The Administrative
Console reads from this table to display messages in the bottom
portion of the the display.  Errant applications may result in
a large number of errors or warnings being created in a short
time -- these are all written to the database and the
Administrative Console attempts to display them all.  If the
arrival rate is too fast, the console can't keep up, and it may
become unusable.
Problem conclusion
The code has been changed to create a governor in the
SeriousEventListener.  As the listener is notified of serious
events, it checks to see if the arrival rate exceeds certain
parameters -- it shuts off writing the events to the data base
if this is the case and resumes again when the arrival rate
drops off.  The events continue to be logged to the activity
log, so the information contained within is not lost.  There
are several new properties that can be used to regulate the
operation of the serious event logging governor:

com.ibm.ejs.sm.server.maxnorm:operation of the serious event logging governor:
(double) this is the arrival rate threshhold above which SeriousEvent logging is turned off. If it is set to 4 and we get messages at a rate of 5 per second, we will stop logging SeriousEvents to the database. They continue to be recorded in the activity log. Default value is 10. com.ibm.ejs.sm.server.minclog:com.ibm.ejs.sm.server.maxnorm:(double) this is the, arrival rate threshhold above whichSeriousEvent logging is turned off. If it is set to 4 and weget messages at a rate of 5 per second, we will stop loggingSeriousEvents to the database. They continue to be recordedin the activity log. Default value is 10.
(double) this is the arrival rate threshhold below which SeriousEvent logging is resumed if it had previously been turned off. If it is set to 4 and the message arrival rate drops to 3.5 per second, we will resume logging. Default value is 5. com.ibm.ejs.sm.server.restarttrigger (double) this is the time interval in milliseconds that can result in resuming logging if it has been turned off. If it is set to 1000 and 2000 milliseconds elapse between message arrivals, we will resume logging. Default value is 2000. com.ibm.ejs.sm.server.clumpsize (integer) this is the number of messages grouped together in a "clump" for purposes determining arrival rates. Default value is 20.
com.ibm.ejs.sm.server.minclog:(double) this is the arrival rate threshhold below whichSeriousEvent logging is resumed if it had previously beenturned off. If it is set to 4 and the message arrival ratedrops to 3.5 per second, we will resume logging. Defaultvalue is 5.com.ibm.ejs.sm.server.restarttrigger(double) this is the time interval in milliseconds that canresult in resuming logging if it has been turned off. If itis set to 1000 and 2000 milliseconds elapse between messagearrivals, we will resume logging. Default value is 2000.com.ibm.ejs.sm.server.clumpsize(integer) this is the number of messages grouped together ina "clump" for purposes determining arrival rates. Defaultvalue is 20.
Temporary fix
Comments
APAR information
APAR numberPQ51338
Reported component nameWAS ADVANCED AI
Reported component ID5648C8400
Reported release350
StatusCLOSED PER
PENoPE
HIPERNoHIPER
Submitted date2001-08-13
Closed date2001-09-28
Last modified date2001-10-04

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:APAR is sysrouted FROM one or more of the following:

PQ53132

Modules/Macros

Fix information
Fixed component nameWAS ADVANCED AI
Fixed component ID5648C8400
APAR is sysrouted TO one or more of the following:PQ53132Modules/Macros

Applicable component levels
R350 PSYUP











Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, General
Software version: 350
Reference #: PQ51338
IBM Group: Software Group
Modified date: 2001-10-04