PK03327: WEBSPHERE ASID ABENDS WITH C9C21C3E, CR HAS MESSAGE BBOO0073E JAVA.LANG.OUTOFMEMORYERROR, SR HAS WTRN0008W: OBJ DESERIALIZED

 A fix is available

Obtain the fix for this APAR



APAR status
Closed as program error.

Error description
WebSphere Application Server for z/OS, Control Region abends.
with Reason code C9C21C3E.  Looking in the CR output, you see:
the following trace entry:
Trace: 2005/03/29 20:09:37.951 01 t=8A6088 c=UNK key=S2
                                                 (0000000A)
  Description: Log Boss/390 Error
  from filename: ./bbolrt.cpp
  at line: 1723
BBOO0073E Shasta Runtime function logJavaException detected that
the following exception occurred in the Java JVM.
Exception Message = java.lang.OutOfMemoryError
JVM Stack Trace = java.lang.OutOfMemoryError
.
Trace: 2005/03/29 20:10:16.685 01 t=8BD088 c=UNK key=P8
                                                     (13007002)
  FunctionName: com.ibm.ws390.tx.partnerLog.XaRecoveryTable
  SourceId: com.ibm.ws390.tx.partnerLog.XaRecoveryTable
  Category: ERROR
ExtendedMessage: WTRN0008W: Object cannot be deserialized. The
exception stack trace follows: java.io.StreamCorruptedException:
Caught EOFException while reading the stream header
.at java.io.ObjectInputStream.readStreamHeader
    (ObjectInputStream.java(Compiled Code))
.at java.io.ObjectInputStream.<init>
    (ObjectInputStream.java(Compiled Code))
.at com.ibm.ws390.tx.partnerLog.XaRecoveryTable.findRecoveryId
    (XaRecoveryTable.java:187)
.at com.ibm.ws390.tx.ResourceManagerData.getGlobalRecoveryID
    (ResourceManagerData.java:107)
.at com.ibm.ws390.tx.TransactionImpl.enlistResource
    (TransactionImpl.java:1303)
.at com.ibm.ws390.tx.TransactionManagerImpl.enlist
    (TransactionManagerImpl.java:861)
.at com.ibm.ws390.tx.TransactionManagerSet.enlist
    (TransactionManagerSet.java:508)
.at com.ibm.ejs.j2c.XATransactionWrapper.enlist
    (XATransactionWrapper.java:737)
.at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending
    (ConnectionEventListener.java:743)
.at com.ibm.ejs.jms.JMSManagedSession.interactionPending
    (JMSManagedSession.java:988)
.at com.ibm.ejs.jms.JMSManagedSession.enlist
    (JMSManagedSession.java(Compiled Code))
.at com.ibm.ejs.jms.JMSMessageProducerHandle.enlist
    (JMSMessageProducerHandle.java:716)
.at com.ibm.ejs.jms.JMSQueueSenderHandle.send
    (JMSQueueSenderHandle.java:175)
.at com.cust.app.code.task.TaskBean.addTaskToQueue(Unknown
.at com.cust app.code.task.EJSLocalStatelessTask_c0f1f37a.
     addTaskToQueue(Unknown Source)
.at com.cust.app.code.putMessage(Unknown Source)(
.at com.cust.app.code.doTask(Unknown Source)
.at com.cust.app.code.onMessage(Unknown Source)
.at com.ibm.ejs.jms.listener.MDBWrapper$PriviledgedOnMessage.
    run(MDBWrapper.java:205)
.at java.security.AccessController.doPrivileged(Native Method)
.at com.ibm.ejs.jms.listener.MDBWrapper.callOnMessage(
    MDBWrapper.java:194)
.
Looking in the SVC Dump that is generated, which captures the
CR asid, and has a dump title of:
 Dump Title: COMPON=WEBSPHERE,Z/OS,COMPID=5655I3500,
             ISSUER=BBORMCDP,ERRNO=C9C21C3E
Formatting out the TCB which issued the BBOO0073E message in the
job output, the stack trace of this TCB is:
Function
--------
BBORMCDX
RMCDP_Class::takeSVC_Dump(unsigned long)
RasProcessTerm(int,unsigned int)
checkJavaException(JNIEnv_*,unsigned char,unsigned char)
checkJavaException(JNIEnv_*)
BBOT_XaRecoveryTableManager::getCurrentSerializedTable
   (bbotxpdm*)
BBOT_TransactionManager_CR::work_request_dispatch(acrwObj&)
ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*)
ACR_ExecutionRoutine
(unknown)
.
 Note that this TCB may not be marked as an error tcb in the
summary format output of the svcdump.
Local fix Problem summary
****************************************************************
* USERS AFFECTED: All users of WebSphere Application Server    *
*                 V5.0 for z/OS                                *
****************************************************************
* PROBLEM DESCRIPTION: WebSphere Application Server for z/OS,  *
*                      Controller abends with minor code       *
*                      C9C21C3E with a                         *
*                      java.lang.OutOfMemoryError in the       *
*                      Controller JVM.                         *
*                                                              *
*                      Associated problem is a                 *
*                      LogFullException occuring due to        *
*                      unnecessarily logging repeated          *
*                      XAResource entries in the transaction   *
*                      log, corresponding to JMS resources     *
*                      over the lifetime of the server.        *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
When using JMS, the transaction log can fill up due to
unnecessarily logging.

This may occur when using JMS together with a Connection Factory
created under the Integral JMS Provider or under external
WebSphere MQ in CLIENT mode, since in both of these cases
entries will be added to the transaction log corresponding to
the associated XAResources.  The problem is especially likely to
occur if you set a nonzero "AgedTimeout" under your
ConnectionFactory's "Connection Pool" properties.

The following exception may be seen from the controller region
address space:

BBOO0220E WTRN0104E: Details of recovery log failure:
     com.ibm.ws.recoverylog.spi.LogFullException: Maximum file
     size of 1024k would be exceeded
 at com.ibm.ws.recoverylog.spi.RecoveryLogImpl.keypoint
 at com.ibm.ws.recoverylog.spi.LogHandle.getWriteableLogRecord
 at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.writeSections
 at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.forceSections
 at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.forceSections
 at com.ibm.ws390.recoverylog.spi.ControllerRequestProcessor.dri
 at com.ibm.ws390.recoverylog.spi.ControllerRequestProcessor.pro

A java.lang.OutOfMemoryError may also appear in the controller
region JVM.

An associated exception like the following may appear in the
servant region:

Trace: 2005/03/29 20:10:16.685 01 t=8BD088 c=UNK key=P8
(13007002)
FunctionName: com.ibm.ws390.tx.partnerLog.XaRecoveryTable
SourceId: com.ibm.ws390.tx.partnerLog.XaRecoveryTable
Category: ERROR
ExtendedMessage: WTRN0008W: Object cannot be deserialized. The
exception stack trace follows: java.io.StreamCorruptedException
Caught EOFException while reading the stream header
at java.io.ObjectInputStream.readStreamHeader
(ObjectInputStream.java(Compiled Code))
at java.io.ObjectInputStream.<init>
(ObjectInputStream.java(Compiled Code))
at com.ibm.ws390.tx.partnerLog.XaRecoveryTable.findRecoveryId
(XaRecoveryTable.java:187)
at com.ibm.ws390.tx.ResourceManagerData.getGlobalRecoveryID
(ResourceManagerData.java:107)
at com.ibm.ws390.tx.TransactionImpl.enlistResource
(TransactionImpl.java:1303)
at com.ibm.ws390.tx.TransactionManagerImpl.enlist
(TransactionManagerImpl.java:861)
at com.ibm.ws390.tx.TransactionManagerSet.enlist
(TransactionManagerSet.java:508)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist
(XATransactionWrapper.java:737)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending
(ConnectionEventListener.java:743)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending
(JMSManagedSession.java:988)
at com.ibm.ejs.jms.JMSManagedSession.enlist
(JMSManagedSession.java(Compiled Code))
at com.ibm.ejs.jms.JMSMessageProducerHandle.enlist
(JMSMessageProducerHandle.java:716)
at com.ibm.ejs.jms.JMSQueueSenderHandle.send
(JMSQueueSenderHandle.java:175)
at com.cust.app.code.task.TaskBean.addTaskToQueue
at com.cust app.code.task.EJSLocalStatelessTask_c0f1f37a.
 addTaskToQueue(Unknown Source)
at com.cust.app.code.putMessage(Unknown Source)
at com.cust.app.code.doTask(Unknown Source)
at com.cust.app.code.onMessage(Unknown Source)
Problem conclusion
Fixed a memory leak in the Application Server's Java heap
storage usage. Also fixed the transaction logging procedure
since we only need to add one entry to the transaction log
corresponding to the JMS resources for which we had previously
been logging multiple entries.

APAR PK03327 is associated with SERVICE LEVEL W502029 of
WebSphere Application Server V5.0 for z/OS.
Temporary fix Comments
APAR information
APAR number PK03327
Reported component name WEBSPHERE FOR Z
Reported component ID 5655I3500
Reported release 500
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2005-03-29
Closed date 2005-04-29
Last modified date 2005-06-02

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:
PK03607

Modules/Macros
BBOUBINF          

Publications Referenced

Fix information
Fixed component name WEBSPHERE FOR Z
Fixed component ID 5655I3500

Applicable component levels
R500 PSY UK03008    UP05/05/05 P F505

  Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.


Document Information


Current web document: swg1PK03327.html
Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PK03327
IBM Group: Software Group
Modified date: Jun 2, 2005