PQ69054: MEMORY LEAK ON SUSE LINUX 390 WHEN STRESS TESTING WEBSPHERE APPLICATION SERVER 4.0.5 (ALL WAS PLATFORMS)

 Fixes are available

4.0.6: WebSphere Application Server Version 4.0 Fix Pack 6
***SUPERCEDED*** 1.3.1 Java SDK, Java Tech Edition for WebSphere v5



APAR status
Closed as program error.

Error description
Base orb in WAS 4.0.5 has memory leak
Operating system: Suse 7.0 Linux390
JDK short version: 1.3.1
JDK full version: J2RE 1.3.1 IBM build cx390131w-20021107 ORB130
Java components: IBM Java orb orbn130-200201003
Machine details: 512M memory, 140M swap
Symptoms: I'm running IBM's WebSphere's performance sample,
TRADE, on WebSphere (WAS) 4.0.5 build ptf50246.04 on
Suse 7.0 Linux390.  The setup for the testcase is to install
WebSphere on two different machines using a remote database
for both the repository and the application database, DB2 is
used in this case.  A seperate machine is also used for the web
server.  During the deployment of the Trade application the EJB
part of Trade is installed on one Linux390 system and the Trade
war and soap is installed on the other linux 390 system.  Using
the WAS admin GUI, I set the JVM heap size for the application
server running trade to 225M.  We use AKSTRESS running on a
seperate box to drive activity to the machine running the web
(HTTP) server.

The linux 390 systems I am using have 512M memory.  Before I
start the test I have 250M to 270M free memory.

The purpose of the test is to verify that WAS can run for three
days.  After about 30 minutes or so the WebSphere application
server runs out of memory with the ptf50246.04 build of WAS 405.
When I run with an earlier build the test completes
successfully.  We also tried running with replacing the
websphere orb (iwsorb.jar) from an earlier build and the test
still fails.  We then replaced the base orb (ibmorb.jar) file
and then the test runs successfully.  We have yet to test the
current iwsorb.jar file with an older ibmorb.jar file.  Doing
ORBInfo gives the following information:
IBM orb orbn130-20020613 with WAS orb J0229.02 works
IBM orb orbn130-20021003 with WAS orb K0244.01 fails

For the TRADE application I created 1000 stock quotes and
accounts and did "Std. No. reg." for "Workload Mix" and took
the defaults for everything else.
*Behaviour differences: Stress tests runs successfully on AIX.
Local fix
Determined that fix was created through a defect in JTC's
SOV family.  This fix was ORB related, so a stand alone
interim fix was created for selected versions of WebSphere.
However, the standard practice is to deliver fixes
developed by the JTC in a repackaged JDK:
Defect sov.57154, included in 12/17/2002 build. There is not a
standalone fix for this problem. The fix is incorporated into
WebSphere 4.0.6 and greater.
Problem summary
****************************************************************
* USERS AFFECTED: All WebSphere Application Server users of    *
*                 WAS 4.0.5                                    *
****************************************************************
* PROBLEM DESCRIPTION: Memory leak observed during stress      *
*                      testing WebSphere 4.0.5.                *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Memory leak observed during stress testing WebSphere 4.0.5 on
Suse 7.0 Linux 390. Internally observed while running
performance tests.  The memory leak quickly used available JVM
heap storage and system memory.  This caused
java.lang.OutOfMemory errors to show on the application server
and caused application server to restart itself. The problem was
identified in the ORB component of the IBM JDK shipped with
4.0.5.  Future JDK service refresh will address this problem
and efix PQ69054 has been provided to address this issue on
the current 4.0.5 levels.  The fix for PQ69054 is available on
all platforms.
Problem conclusion
This code change is happening in SOV source.

The original code pre Java build 20020613 had a hashtable in
ClientDelegate used to cache the servant IOR. This was
causing a leak in CICS applications.
A defect that went into Java 1.3.1 was to remove the
hashtable and store a reference of the IOR inside the ORB
directly.  The getServantIOR() method was moved from
ClientDelegate to com.ibm.rmi.corba.ORB. When trying to
backport this to 1.3.0, it was found that WebSphere
code was calling getServantIOR in ClientDelegate therefore
another solution had to be found.  The fix now simply
removes the hashtable keeping everything as it was before.

Removing the hashtable caused getServatIOR() to be invoked
every time to calculate the IOR by creating a tie and
calling orb.connect(tie).  The side effect of this is that
the tie is stored each time but it is not cached (no check
is made to see if there already is one).  Here is the leak!

The new fix is to go back to the initial complex solution.
Now it is the ORB containing the reference for the IOR and
even if the logic of the getServantIOR method is inside the
orb class, the method is kept in the ClientDelegate that
invokes the one in the orb.
Temporary fix Comments
APAR information
APAR number PQ69054
Reported component name WAS AE LINUX/ZS
Reported component ID 5724B6108
Reported release 400
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Submitted date 2002-12-11
Closed date 2002-12-16
Last modified date 2004-03-23

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros
ORB          

SRLS

Fix information
Fixed component name WAS AE LINUX/ZS
Fixed component ID 5724B6108

Applicable component levels
R400 PSY    UP


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > General
Operating system(s):
Software version: 400
Software edition:
Reference #: PQ69054
IBM Group: Software Group
Modified date: Mar 23, 2004