PK03327: WEBSPHERE ASID ABENDS WITH C9C21C3E, CR HAS MESSAGE BBOO0073E JAVA.LANG.OUTOFMEMORYERROR, SR HAS WTRN0008W: OBJ DESERIALIZED | |||||||||||||||||||||||||||||||||||||||||||
![]() |
|||||||||||||||||||||||||||||||||||||||||||
![]() APAR status Closed as program error. Error description WebSphere Application Server for z/OS, Control Region abends. with Reason code C9C21C3E. Looking in the CR output, you see: the following trace entry: Trace: 2005/03/29 20:09:37.951 01 t=8A6088 c=UNK key=S2 (0000000A) Description: Log Boss/390 Error from filename: ./bbolrt.cpp at line: 1723 BBOO0073E Shasta Runtime function logJavaException detected that the following exception occurred in the Java JVM. Exception Message = java.lang.OutOfMemoryError JVM Stack Trace = java.lang.OutOfMemoryError . Trace: 2005/03/29 20:10:16.685 01 t=8BD088 c=UNK key=P8 (13007002) FunctionName: com.ibm.ws390.tx.partnerLog.XaRecoveryTable SourceId: com.ibm.ws390.tx.partnerLog.XaRecoveryTable Category: ERROR ExtendedMessage: WTRN0008W: Object cannot be deserialized. The exception stack trace follows: java.io.StreamCorruptedException: Caught EOFException while reading the stream header .at java.io.ObjectInputStream.readStreamHeader (ObjectInputStream.java(Compiled Code)) .at java.io.ObjectInputStream.<init> (ObjectInputStream.java(Compiled Code)) .at com.ibm.ws390.tx.partnerLog.XaRecoveryTable.findRecoveryId (XaRecoveryTable.java:187) .at com.ibm.ws390.tx.ResourceManagerData.getGlobalRecoveryID (ResourceManagerData.java:107) .at com.ibm.ws390.tx.TransactionImpl.enlistResource (TransactionImpl.java:1303) .at com.ibm.ws390.tx.TransactionManagerImpl.enlist (TransactionManagerImpl.java:861) .at com.ibm.ws390.tx.TransactionManagerSet.enlist (TransactionManagerSet.java:508) .at com.ibm.ejs.j2c.XATransactionWrapper.enlist (XATransactionWrapper.java:737) .at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending (ConnectionEventListener.java:743) .at com.ibm.ejs.jms.JMSManagedSession.interactionPending (JMSManagedSession.java:988) .at com.ibm.ejs.jms.JMSManagedSession.enlist (JMSManagedSession.java(Compiled Code)) .at com.ibm.ejs.jms.JMSMessageProducerHandle.enlist (JMSMessageProducerHandle.java:716) .at com.ibm.ejs.jms.JMSQueueSenderHandle.send (JMSQueueSenderHandle.java:175) .at com.cust.app.code.task.TaskBean.addTaskToQueue(Unknown .at com.cust app.code.task.EJSLocalStatelessTask_c0f1f37a. addTaskToQueue(Unknown Source) .at com.cust.app.code.putMessage(Unknown Source)( .at com.cust.app.code.doTask(Unknown Source) .at com.cust.app.code.onMessage(Unknown Source) .at com.ibm.ejs.jms.listener.MDBWrapper$PriviledgedOnMessage. run(MDBWrapper.java:205) .at java.security.AccessController.doPrivileged(Native Method) .at com.ibm.ejs.jms.listener.MDBWrapper.callOnMessage( MDBWrapper.java:194) . Looking in the SVC Dump that is generated, which captures the CR asid, and has a dump title of: Dump Title: COMPON=WEBSPHERE,Z/OS,COMPID=5655I3500, ISSUER=BBORMCDP,ERRNO=C9C21C3E Formatting out the TCB which issued the BBOO0073E message in the job output, the stack trace of this TCB is: Function -------- BBORMCDX RMCDP_Class::takeSVC_Dump(unsigned long) RasProcessTerm(int,unsigned int) checkJavaException(JNIEnv_*,unsigned char,unsigned char) checkJavaException(JNIEnv_*) BBOT_XaRecoveryTableManager::getCurrentSerializedTable (bbotxpdm*) BBOT_TransactionManager_CR::work_request_dispatch(acrwObj&) ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*) ACR_ExecutionRoutine (unknown) . Note that this TCB may not be marked as an error tcb in the summary format output of the svcdump.Local fix Problem summary **************************************************************** * USERS AFFECTED: All users of WebSphere Application Server * * V5.0 for z/OS * **************************************************************** * PROBLEM DESCRIPTION: WebSphere Application Server for z/OS, * * Controller abends with minor code * * C9C21C3E with a * * java.lang.OutOfMemoryError in the * * Controller JVM. * * * * Associated problem is a * * LogFullException occuring due to * * unnecessarily logging repeated * * XAResource entries in the transaction * * log, corresponding to JMS resources * * over the lifetime of the server. * **************************************************************** * RECOMMENDATION: * **************************************************************** When using JMS, the transaction log can fill up due to unnecessarily logging. This may occur when using JMS together with a Connection Factory created under the Integral JMS Provider or under external WebSphere MQ in CLIENT mode, since in both of these cases entries will be added to the transaction log corresponding to the associated XAResources. The problem is especially likely to occur if you set a nonzero "AgedTimeout" under your ConnectionFactory's "Connection Pool" properties. The following exception may be seen from the controller region address space: BBOO0220E WTRN0104E: Details of recovery log failure: com.ibm.ws.recoverylog.spi.LogFullException: Maximum file size of 1024k would be exceeded at com.ibm.ws.recoverylog.spi.RecoveryLogImpl.keypoint at com.ibm.ws.recoverylog.spi.LogHandle.getWriteableLogRecord at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.writeSections at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.forceSections at com.ibm.ws.recoverylog.spi.RecoverableUnitImpl.forceSections at com.ibm.ws390.recoverylog.spi.ControllerRequestProcessor.dri at com.ibm.ws390.recoverylog.spi.ControllerRequestProcessor.pro A java.lang.OutOfMemoryError may also appear in the controller region JVM. An associated exception like the following may appear in the servant region: Trace: 2005/03/29 20:10:16.685 01 t=8BD088 c=UNK key=P8 (13007002) FunctionName: com.ibm.ws390.tx.partnerLog.XaRecoveryTable SourceId: com.ibm.ws390.tx.partnerLog.XaRecoveryTable Category: ERROR ExtendedMessage: WTRN0008W: Object cannot be deserialized. The exception stack trace follows: java.io.StreamCorruptedException Caught EOFException while reading the stream header at java.io.ObjectInputStream.readStreamHeader (ObjectInputStream.java(Compiled Code)) at java.io.ObjectInputStream.<init> (ObjectInputStream.java(Compiled Code)) at com.ibm.ws390.tx.partnerLog.XaRecoveryTable.findRecoveryId (XaRecoveryTable.java:187) at com.ibm.ws390.tx.ResourceManagerData.getGlobalRecoveryID (ResourceManagerData.java:107) at com.ibm.ws390.tx.TransactionImpl.enlistResource (TransactionImpl.java:1303) at com.ibm.ws390.tx.TransactionManagerImpl.enlist (TransactionManagerImpl.java:861) at com.ibm.ws390.tx.TransactionManagerSet.enlist (TransactionManagerSet.java:508) at com.ibm.ejs.j2c.XATransactionWrapper.enlist (XATransactionWrapper.java:737) at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending (ConnectionEventListener.java:743) at com.ibm.ejs.jms.JMSManagedSession.interactionPending (JMSManagedSession.java:988) at com.ibm.ejs.jms.JMSManagedSession.enlist (JMSManagedSession.java(Compiled Code)) at com.ibm.ejs.jms.JMSMessageProducerHandle.enlist (JMSMessageProducerHandle.java:716) at com.ibm.ejs.jms.JMSQueueSenderHandle.send (JMSQueueSenderHandle.java:175) at com.cust.app.code.task.TaskBean.addTaskToQueue at com.cust app.code.task.EJSLocalStatelessTask_c0f1f37a. addTaskToQueue(Unknown Source) at com.cust.app.code.putMessage(Unknown Source) at com.cust.app.code.doTask(Unknown Source) at com.cust.app.code.onMessage(Unknown Source)Problem conclusion Fixed a memory leak in the Application Server's Java heap storage usage. Also fixed the transaction logging procedure since we only need to add one entry to the transaction log corresponding to the JMS resources for which we had previously been logging multiple entries. APAR PK03327 is associated with SERVICE LEVEL W502029 of WebSphere Application Server V5.0 for z/OS.Temporary fix Comments
APAR is sysrouted FROM one or more of the following: APAR is sysrouted TO one or more of the following: PK03607 Modules/Macros
Publications Referenced
|
Document Information |
Current web document: swg1PK03327.html
Product categories: Software > Application Servers >
Distributed Application & Web Servers > WebSphere Application
Server for z/OS
Operating system(s):
Software version: 500
Software edition:
Reference #: PK03327
IBM Group: Software Group
Modified date: Jun 2, 2005
(C) Copyright IBM Corporation 2000, 2009. All Rights Reserved.