APAR status
Closed as program error.
Error description
Customer environment: WP 5.0.2, WAS 5.0.2 with the APAR
PQ81144. It is also running DB2 8.1 and IBM Directory
Server 5.1 FP2. The box is on AIX 5.2 ML1.
There are 3 issues
1)100% CPU cylical
goes close to 100% for a minute, drops back down to
30%, goes back to 100%, drops back to 30% this only happened
when the heap grew large enough
before hand it was around 30% for 48+ hours after about 48 hours
it hit 100%.. then we saw increased latency
2)Outofmemory(OOM's)
Ct says they never even saw OOM's on this particular run
The problem seems to be happening due to heap exhaustion
3)No core file produced
Ct has following settings
User Limits (in bytes except for NOFILE and NPROC) -
-----------
RLIMIT_FSIZE : 1073741312
RLIMIT_DATA : 2147483645
RLIMIT_STACK : 33554432
RLIMIT_CORE : 1073741312
RLIMIT_NOFILE : 32000
NPROC(max) : 262144
<AF[15928]: Allocation Failure. need 1581072 bytes, 2620 ms
since last
AF>
<AF[15928]: managing allocation failure, action=2
(43507432/536803840)>
<GC(33308): Concurrent HALTED (state=64). Target=509269837
Traced=175847466 (134465010+41382456) Free=43507432>
<GC(33308): No Dirty Cards cleaned (Factor 0.147)>
<GC(33308): GC cycle started Tue Mar 30 10:26:08 2004
<GC(33308): freed 57695264 bytes, 18% free
(101202696/536803840), in
10090 ms>
<GC(33308): mark: 1574 ms, sweep: 71 ms, compact: 8445 ms>
<GC(33308): In mark: Final dirty Cards scan: 294 ms 26980
cards
(total:5731909 ms)
<GC(33308): refs: soft 1 (age >= 32), weak 0, final 944,
phantom 0>
<GC(33308): moved 6046724 objects, 301934528 bytes, reason=1,
used
2904 more bytes>
<AF[15928]: managing allocation failure, action=3
(101202696/536803840)>
<AF[15928]: managing allocation failure, action=4
(101202696/536803840)>
<AF[15928]: clearing all remaining soft refs>
<GC(33309): Concurrent ABORTED. Target=514388469 Traced=1259872
(0+1259872) Free=101202696>
<GC(33309): GC cycle started Tue Mar 30 10:26:10 2004
<GC(33309): freed 1997528 bytes, 19% free (103200224/536803840),
in 2121
ms>
<GC(33309): mark: 2053 ms, sweep: 68 ms, compact: 0 ms>
<GC(33309): refs: soft 392 (age >= 32), weak 0, final 157,
phantom 0>
<GC(33310): Concurrent ABORTED. Target=517270692 Traced=1259872
(0+1259872) Free=103200224>
<GC(33310): GC cycle started Tue Mar 30 10:26:19 2004
<GC(33310): freed 292744 bytes, 19% free (103492968/536803840),
in 8912
ms>
<GC(33310): mark: 2091 ms, sweep: 68 ms, compact: 6753 ms>
<GC(33310): refs: soft 0 (age >= 32), weak 0, final 0, phantom
0>
<GC(33310): moved 2185684 objects, 113816280 bytes, reason=1,
used 16
more bytes>
<AF[15928]: managing allocation failure, action=6
(103492968/536803840)>
<AF[15928]: totally out of heap space>
Writing Heap dump ....
Written Heap dump to
/usr/WebSphere/AppServer/heapdump81348.1080663979.txt
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to
/usr/WebSphere/AppServer/javacore81348.1080664082.txt
<AF[15928]: completed in 290409 ms>
total of 433 MB of heap com/ibm/ws/security/web/WebAc
cessPermission contributing to total of 169MB,which
intern references java/util/Hashtable$Entry and
java/lang/ref/Finalizer objects
Local fix Problem summary
****************************************************************
* USERS AFFECTED: WebSphere Application Server users who have *
* enabled security and are accessing protected *
* URIs. *
****************************************************************
* PROBLEM DESCRIPTION: Servicing excesively large numbers of *
* protecgted URIs will eventually consume *
* all memory available in the heap. *
****************************************************************
* RECOMMENDATION: *
****************************************************************
Servicing excesively large numbers of protecgted URIs will
eventually consume all memory available in the heap. The
reason for this is some security information is cached by
the URIand the cache size was not limited.
Portal Server is one application which has been reported
to have this issue.
Problem conclusion
The cache size is now limited to 600 by default. The
following jvm properties can be used to change the default
value.
com.ibm.websphere.security.core.phhashsize=<value>
com.ibm.websphere.security.web.hashsize=<value>
Temporary fix
ZE Fix Error
PQ88445 04/05/07
Comments
APAR information |
APAR number |
PQ87017 |
Reported component name |
WAS BASE 5.0 |
Reported component ID |
5630A3600 |
Reported release |
00A |
Status |
CLOSED PER |
PE |
NoPE |
HIPER |
NoHIPER |
Special Attention |
NoSpecatt |
Submitted date |
2004-04-01 |
Closed date |
2004-04-19 |
Last modified date |
2005-05-05 |
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
PK05321
Modules/Macros
Publications Referenced
Applicable component levels |
R003 PSY |
UP |
R00A PSY |
UP |
R00H PSY |
UP |
R00I PSY |
UP |
R00P PSY |
UP |
R00S PSY |
UP |
R00W PSY |
UP |
R103 PSY |
UP |
R10A PSY |
UP |
R10H PSY |
UP |
R10I PSY |
UP |
R10P PSY |
UP |
R10S PSY |
UP |
R10W PSY |
UP |
|