APAR status |
Closed as program error.
| Error description
Problem:***The Admin Console hangs.
Recap:Problem:***The Admin Console hangs.
***AIX/WAS 3.5.4
***Client/Admin server/JVM local.
***Restarting the adminconsole fully recovers the problem.
***The heap(s) of the other JVM's (nanny, appserver) did
not grow.
***The JVM (of the consle) grew slightly (by +3mB over a
20 hour test period).
***The system wide allocate virtual memory didn't increase.
***There were no more messages seen in the message window.
***The (servlet) engine continued to perform the auto reload.
***Attempting to change an object through the console rulted
in an entirely grey foreground/background (at this point
I got the java.lang.OutOfMemoryError exceptions thrown).
***No errors in the system log.
***The adminclient debug trace file gives no clues. It shows
the log messages to the client up until the console hangs,
then only Act waiting for notification and BackgroundLru
going to sleep and waking up.
***Thread dumps of the client before and after the hang are
exactly the same (the same running threads) except the
SeriousEventMessagePanelUpdater thread is present before
the hang.
Description:Recap:***AIX/WAS 3.5.4***Client/Admin server/JVM local.***Restarting the adminconsole fully recovers the problem.***The heap(s) of the other JVM's (nanny, appserver) didnot grow.***The JVM (of the consle) grew slightly (by +3mB over a20 hour test period).***The system wide allocate virtual memory didn't increase.***There were no more messages seen in the message window.***The (servlet) engine continued to perform the auto reload.***Attempting to change an object through the console rultedin an entirely grey foreground/background (at this pointI got the java.lang.OutOfMemoryError exceptions thrown).***No errors in the system log.***The adminclient debug trace file gives no clues. It showsthe log messages to the client up until the console hangs,then only Act waiting for notification and BackgroundLrugoing to sleep and waking up.***Thread dumps of the client before and after the hang areexactly the same (the same running threads) except theSeriousEventMessagePanelUpdater thread is present beforethe hang.
Run the following script in AIX -
while true
do
touch SnoopServlet.class
echo touch to SnoopServlet.class
ls -al SnoopServlet.class
sleep 60
done
This will modify the time stamp on the snoop class. From the
console set the auto reload for the default_app to TRUE,
interval 60 sec., and for the snoop servlet set load at
startup to true. Start the app server (Default Server),
the auto reload will generate messages that are written to
the 'Console Messages' window in the console. After time,
in the test about 5 hours, the messages will stop appearing
in the window and the console will hang. This problem was
reproduced by an IBM Japan team and by the WebSphere team
in Raleigh. Description:Run the following script in AIX -while truedotouch SnoopServlet.classecho touch to SnoopServlet.classls -al SnoopServlet.classsleep 60doneThis will modify the time stamp on the snoop class. From theconsole set the auto reload for the default_app to TRUE,interval 60 sec., and for the snoop servlet set load atstartup to true. Start the app server (Default Server),the auto reload will generate messages that are written tothe 'Console Messages' window in the console. After time,in the test about 5 hours, the messages will stop appearingin the window and the console will hang. This problem wasreproduced by an IBM Japan team and by the WebSphere teamin Raleigh. Local fix
Restart the Admin Console Problem summary
****************************************************************
* USERS AFFECTED: All WebSphere Application Server 3.5 *
* Advanced Edition users of the Administrative *
* Console *
****************************************************************
* PROBLEM DESCRIPTION: Administrative Console can be *
* overwhelmed by a large volume of *
* Serious Event messages, becoming *
* unusable *
****************************************************************
* RECOMMENDATION: *
****************************************************************
After the RAS component logs warnings and errors in the
activity log, it makes them available to the server (either
Administrative server or Application server) to be logged in
the Serious Event table of the admin database -- this is done
by the SeriousEventListener in each server. The Administrative
Console reads from this table to display messages in the bottom
portion of the the display. Errant applications may result in
a large number of errors or warnings being created in a short
time -- these are all written to the database and the
Administrative Console attempts to display them all. If the
arrival rate is too fast, the console can't keep up, and it may
become unusable. Problem conclusion
The code has been changed to create a governor in the
SeriousEventListener. As the listener is notified of serious
events, it checks to see if the arrival rate exceeds certain
parameters -- it shuts off writing the events to the data base
if this is the case and resumes again when the arrival rate
drops off. The events continue to be logged to the activity
log, so the information contained within is not lost. There
are several new properties that can be used to regulate the
operation of the serious event logging governor:
com.ibm.ejs.sm.server.maxnorm:operation of the serious event logging governor:
(double) this is the arrival rate threshhold above which
SeriousEvent logging is turned off. If it is set to 4 and we
get messages at a rate of 5 per second, we will stop logging
SeriousEvents to the database. They continue to be recorded
in the activity log. Default value is 10.
com.ibm.ejs.sm.server.minclog:com.ibm.ejs.sm.server.maxnorm:(double) this is the, arrival rate threshhold above whichSeriousEvent logging is turned off. If it is set to 4 and weget messages at a rate of 5 per second, we will stop loggingSeriousEvents to the database. They continue to be recordedin the activity log. Default value is 10.
(double) this is the arrival rate threshhold below which
SeriousEvent logging is resumed if it had previously been
turned off. If it is set to 4 and the message arrival rate
drops to 3.5 per second, we will resume logging. Default
value is 5.
com.ibm.ejs.sm.server.restarttrigger
(double) this is the time interval in milliseconds that can
result in resuming logging if it has been turned off. If it
is set to 1000 and 2000 milliseconds elapse between message
arrivals, we will resume logging. Default value is 2000.
com.ibm.ejs.sm.server.clumpsize
(integer) this is the number of messages grouped together in
a "clump" for purposes determining arrival rates. Default
value is 20. com.ibm.ejs.sm.server.minclog:(double) this is the arrival rate threshhold below whichSeriousEvent logging is resumed if it had previously beenturned off. If it is set to 4 and the message arrival ratedrops to 3.5 per second, we will resume logging. Defaultvalue is 5.com.ibm.ejs.sm.server.restarttrigger(double) this is the time interval in milliseconds that canresult in resuming logging if it has been turned off. If itis set to 1000 and 2000 milliseconds elapse between messagearrivals, we will resume logging. Default value is 2000.com.ibm.ejs.sm.server.clumpsize(integer) this is the number of messages grouped together ina "clump" for purposes determining arrival rates. Defaultvalue is 20. Temporary fixComments
APAR information | APAR number | PQ51338 | Reported component name | WAS ADVANCED AI | Reported component ID | 5648C8400 | Reported release | 350 | Status | CLOSED PER | PE | NoPE | HIPER | NoHIPER | Submitted date | 2001-08-13 | Closed date | 2001-09-28 | Last modified date | 2001-10-04 |
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:APAR is sysrouted FROM one or more of the following:
PQ53132
Modules/Macros
|
Fix information |
Fixed component name | WAS ADVANCED AI | Fixed component ID | 5648C8400 | APAR is sysrouted TO one or more of the following:PQ53132Modules/Macros
Applicable component levels | R350 PSY | UP |
|