PQ87017: Memory leak with WebSpherePortal 5.0.2 on WebSphere App Server 5.0.2

 Fixes are available

5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for AIX
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for HP-UX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Linux
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for AIX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for HP-UX
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Solaris
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Windows
5.1.1.18: WebSphere Application Server V5.1.1 Cumulative Fix 18 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Linux
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Solaris
5.1.1.17: WebSphere Application Server V5.1.1 Cumulative Fix 17 for Windows
5.0.2.7: WebSphere Application Server Express 5.0.2 Cumulative Fix 7
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for AIX platforms
5.1.0.5: WebSphere Application Server V5.1.0 Cumulative Fix 5
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Solaris
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Windows
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Solaris
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Windows
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for AIX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for AIX
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for Windows
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Windows
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Solaris
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for AIX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for Linux
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for HP-UX platforms
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for OS/400 platforms
5.0.2.12: WebSphere Application Server 5.0.2 Cumulative Fix 12
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for HP-UX
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for AIX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Windows
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for HP-UX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for AIX
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for AIX
5.0.2.17: WebSphere Application Server 5.0.2 Cumulative Fix 17 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for HP-UX
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Solaris platforms
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Linux
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for HP-UX
5.1.1.9: WebSphere Application Server V5.1.1 Cumulative Fix 9 for Linux
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Solaris
5.0.2.8: WebSphere Application Server V5.0.2 Cumulative Fix 8
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Windows
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for AIX
PQ88445; 5.0.2: Memory leak with WebSphere Portal Server V5.0.2
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Windows
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Solaris
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Linux
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Windows
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for HP-UX
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for AIX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Windows
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Solaris
5.0.2.14: WebSphere Application Server 5.0.2 Cumulative Fix 14 for Windows
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for AIX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Linux
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for HP-UX
5.1.1.12: WebSphere Application Server V5.1.1 Cumulative Fix 12 for Solaris
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for AIX
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Windows
5.0.2.13: WebSphere Application Server 5.0.2 Cumulative Fix 13
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for HP-UX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Solaris
5.1.1.13: WebSphere Application Server V5.1.1 Cumulative Fix 13 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for AIX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Linux
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for Windows
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Windows platforms
6.0.2: WebSphere Application Server V6.0 Refresh Pack 2 for Linux platforms
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Windows
5.1.1: WebSphere Application Server Version 5.1 Fix Pack 1 (Version 5.1.1)
5.0.2.18: WebSphere Application Server 5.0.2 Cumulative Fix 18 for Linux
5.1.1.11: WebSphere Application Server V5.1.1 Cumulative Fix 11 for HP-UX
5.1.1.14: WebSphere Application Server V5.1.1 Cumulative Fix 14 for HP-UX
5.0.2.15: WebSphere Application Server 5.0.2 Cumulative Fix 15 for HP-UX
5.0.2.16: WebSphere Application Server 5.0.2 Cumulative Fix 16 for Linux
5.1.1.10: WebSphere Application Server V5.1.1 Cumulative Fix 10 for Solaris
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for AIX
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for HP-UX
5.1.1.16: WebSphere Application Server V5.1.1 Cumulative Fix 16 for Linux
5.1.1.15: WebSphere Application Server V5.1.1 Cumulative Fix 15 for Linux
5.1.1.19: WebSphere Application Server V5.1.1 Cumulative Fix 19 for HP-UX



APAR status
Closed as program error.

Error description
Customer environment: WP 5.0.2, WAS 5.0.2 with the APAR

PQ81144. It is also running DB2 8.1 and IBM Directory
Server 5.1 FP2.  The box is on AIX 5.2 ML1.

There are 3 issues
1)100% CPU cylical
goes close to 100% for a minute, drops back down to
30%, goes back to 100%, drops back to 30% this only happened
when the heap grew large enough
before hand it was around 30% for 48+ hours after about 48 hours
it hit 100%.. then we saw increased latency

2)Outofmemory(OOM's)
Ct says they never even saw OOM's on this particular run
The problem seems to be happening due to heap exhaustion

3)No core file produced
  Ct has following settings
  User Limits (in bytes except for NOFILE and NPROC) -
-----------
RLIMIT_FSIZE   : 1073741312
RLIMIT_DATA    : 2147483645
RLIMIT_STACK   : 33554432
RLIMIT_CORE    : 1073741312
RLIMIT_NOFILE  : 32000
NPROC(max)     : 262144

<AF[15928]: Allocation Failure. need 1581072 bytes, 2620 ms
since last
AF>
<AF[15928]: managing allocation failure, action=2
(43507432/536803840)>
<GC(33308): Concurrent HALTED (state=64). Target=509269837
Traced=175847466 (134465010+41382456) Free=43507432>
<GC(33308): No Dirty Cards cleaned (Factor 0.147)>
<GC(33308): GC cycle started Tue Mar 30 10:26:08 2004
<GC(33308): freed 57695264 bytes, 18% free
(101202696/536803840), in
10090 ms>
  <GC(33308): mark: 1574 ms, sweep: 71 ms, compact: 8445 ms>
  <GC(33308): In mark: Final dirty Cards scan: 294 ms 26980
cards
(total:5731909 ms)
  <GC(33308): refs: soft 1 (age >= 32), weak 0, final 944,
phantom 0>
  <GC(33308): moved 6046724 objects, 301934528 bytes, reason=1,
used
2904 more bytes>
<AF[15928]: managing allocation failure, action=3
(101202696/536803840)>
<AF[15928]: managing allocation failure, action=4
(101202696/536803840)>
<AF[15928]: clearing all remaining soft refs>
<GC(33309): Concurrent ABORTED. Target=514388469 Traced=1259872
(0+1259872) Free=101202696>
<GC(33309): GC cycle started Tue Mar 30 10:26:10 2004
<GC(33309): freed 1997528 bytes, 19% free (103200224/536803840),
in 2121
ms>
  <GC(33309): mark: 2053 ms, sweep: 68 ms, compact: 0 ms>
  <GC(33309): refs: soft 392 (age >= 32), weak 0, final 157,
phantom 0>
<GC(33310): Concurrent ABORTED. Target=517270692 Traced=1259872
(0+1259872) Free=103200224>
<GC(33310): GC cycle started Tue Mar 30 10:26:19 2004
<GC(33310): freed 292744 bytes, 19% free (103492968/536803840),
in 8912
ms>
  <GC(33310): mark: 2091 ms, sweep: 68 ms, compact: 6753 ms>
  <GC(33310): refs: soft 0 (age >= 32), weak 0, final 0, phantom
0>
  <GC(33310): moved 2185684 objects, 113816280 bytes, reason=1,
used 16
more bytes>
<AF[15928]: managing allocation failure, action=6
(103492968/536803840)>
<AF[15928]: totally out of heap space>
Writing Heap dump ....
Written Heap dump to
/usr/WebSphere/AppServer/heapdump81348.1080663979.txt
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to
/usr/WebSphere/AppServer/javacore81348.1080664082.txt
<AF[15928]: completed in 290409 ms>

total of 433 MB of heap com/ibm/ws/security/web/WebAc
cessPermission contributing to total of 169MB,which
intern references java/util/Hashtable$Entry and
java/lang/ref/Finalizer objects
Local fix Problem summary
****************************************************************
* USERS AFFECTED: WebSphere Application Server users who have  *
*                 enabled security and are accessing protected *
*                 URIs.                                        *
****************************************************************
* PROBLEM DESCRIPTION: Servicing excesively large numbers of   *
*                      protecgted URIs will eventually consume *
*                      all memory available in the heap.       *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Servicing excesively large numbers of protecgted URIs will
eventually consume all memory available in the heap.  The
reason for this is some security information is cached by
the URIand the cache size was not limited.

Portal Server is one application which has been reported
to have this issue.
Problem conclusion
The cache size is now limited to 600 by default.  The
following jvm properties can be used to change the default
value.
com.ibm.websphere.security.core.phhashsize=<value>
com.ibm.websphere.security.web.hashsize=<value>
Temporary fix
ZE Fix Error 
PQ88445 04/05/07
Comments
APAR information
APAR number PQ87017
Reported component name WAS BASE 5.0
Reported component ID 5630A3600
Reported release 00A
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-04-01
Closed date 2004-04-19
Last modified date 2005-05-05

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:
PK05321

Modules/Macros

Publications Referenced

Fix information

Applicable component levels
R003 PSY    UP
R00A PSY    UP
R00H PSY    UP
R00I PSY    UP
R00P PSY    UP
R00S PSY    UP
R00W PSY    UP
R103 PSY    UP
R10A PSY    UP
R10H PSY    UP
R10I PSY    UP
R10P PSY    UP
R10S PSY    UP
R10W PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 00A
Software edition:
Reference #: PQ87017
IBM Group: Software Group
Modified date: May 5, 2005