Fix (APAR): JR28088 Status: Fix Release: 6.0.2.2 Operating System: AIX,HP-UX,i5/OS,Linux,Solaris,Windows Supersedes Fixes: CMVC Defect: JR28088 Byte size of APAR: 7846 Date: 2008-01-28 Abstract: When the OutOfMemory exception is raised, WebSphere Process Server members create javacores and heapdumps at a rate of about 1 every 5 minutes. Description/symptom of problem: JR28088 resolves the following problem: ERROR DESCRIPTION: This APAR has a similar signature and observed behavior to JR27678 however it is a problem in a different section of code. An Out Of Memory error occurs when WebSphere Process Server (WPS) is handling a failure. A failure occurs during a flow, which should be handled by the WPS Recovery system. During this process, the RecoveryMDBHelper of manualrecovery tries to save the failed event, but it receives an exception in method getEObject(). This error is handled by the WebSphere Process Server using the First Failure Data Capture (FFDC) system. Currently,the processing for this logging might consume and require large memory allocations (10MB, 20MB, 30MB allocations) when processing these errors. These large memory allocations might lead to an Out Of Memory exception. This FFDC logging behavior will be changed so that the entire object is not logged. This will remove the need for these large allocations. It will also provide data to assist in identifying the root error which lead to the problem. The stack trace example for this error is as follows. Out Of Memory at: 4XESTACKTRACE at java.io.StringWriter.write(StringWriter.java(Compiled Code)) 4XESTACKTRACE at java.io.BufferedWriter.flushBuffer(BufferedWriter.java(Compiled Code)) 4XESTACKTRACE at java.io.BufferedWriter.flush(BufferedWriter.java(Compiled Code)) 4XESTACKTRACE at java.io.PrintWriter.flush(PrintWriter.java(Compiled Code)) 4XESTACKTRACE at com.ibm.ws.sca.logging.ObjectDumper.dump(ObjectDumper.java:450) 4XESTACKTRACE at com.ibm.ws.sca.logging.impl.JSR47LogImpl.ffdc(JSR47LogImpl.java: 218) 4XESTACKTRACE at com.ibm.ws.sca.internal.managed.WASLogImpl.ffdc(WASLogImpl.java: 44) 4XESTACKTRACE at com.ibm.ws.sca.logging.impl.JSR47LogImpl.ffdc(JSR47LogImpl.java: 229) 4XESTACKTRACE at com.ibm.ws.sca.internal.scdl.resources.impl.BusResourceImpl.getE Object(B usResourceImpl.java:101) 4XESTACKTRACE at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getEObject(R esourceS etImpl.java(Compiled Code)) 4XESTACKTRACE at org.eclipse.emf.ecore.util.EcoreUtil.resolve(EcoreUtil.java(Comp iled Code)) 4XESTACKTRACE at com.ibm.ws.sca.internal.headers.impl.ManagedInteractionHeaderImp l$1.run( ManagedInteractionHeaderImpl.java:70) 4XESTACKTRACE at java.security.AccessController.doPrivileged1(Native Method) 4XESTACKTRACE at java.security.AccessController.doPrivileged(AccessController.jav a(Compil ed Code)) 4XESTACKTRACE at com.ibm.ws.sca.internal.headers.impl.ManagedInteractionHeaderImp l.eResol veProxy(ManagedInteractionHeaderImpl.java:68) 4XESTACKTRACE at com.ibm.wsspi.sca.headers.impl.InteractionHeaderImpl.getSourcePo rt(Inter actionHeaderImpl.java(Compiled Code)) 4XESTACKTRACE at com.ibm.wbiserver.manualrecovery.ejb.RecoveryMDBHelper.getScaMes sageInfo (RecoveryMDBHelper.java:315) 4XESTACKTRACE at com.ibm.wbiserver.manualrecovery.ejb.RecoveryMDBHelper.saveFaile dEvent(R ecoveryMDBHelper.java:109) 4XESTACKTRACE at com.ibm.wbiserver.manualrecovery.ejb.RecoveryMDBHandler.onMessag e(Recove ryMDBHandler.java:130) 4XESTACKTRACE at com.ibm.wsspi.sca.async.bean.impl.ServiceSIBusMessageBean.onMess age(Serv iceSIBusMessageBean.java:103) LOCAL FIX: Correct the problem that led to the error handling scenario. In this case, a messaging endpoint was unavailable, leading to the failure scenario. Correcting the messaging endpoint avoids the failure scenario. PROBLEM SUMMARY USERS AFFECTED: WebSphere Process Server, version 6.0.2, users. PROBLEM DESCRIPTION: When the OutOfMemory exception is raised, WebSphere Process Server members create javacores and heapdumps at a rate of about 1 every 5 minutes. RECOMMENDATION: None This error is handled by the WebSphere Process Server using the First Failure Data Capture (FFDC) system. Currently, the processing for this logging might consume and require large memory allocations (10MB, 20MB, 30MB allocations) when processing these errors. These large memory allocations might lead to an Out Of Memory exception. This FFDC logging behavior will be changed so that the entire object is not logged. This will remove the need for these large allocations. It will also provide data to assist in identifying the root error which lead to the problem. PROBLEM CONCLUSION: Apply the interim fix to resolve the problem. Directions to apply fix: Download and follow Update Installer installation instructions. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 1) Copy the pak file to the directory maintenance in the UpdateInstaller directory 2) Shutdown WebSphere Process Server. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 3) Run the UpdateInstaller and follow the prompts. 4) Restart WebSphere Process Server. Directions to remove fix: NOTE: FIXES MUST BE REMOVED IN THE REVERSE ORDER IN WHICH THEY WERE APPLIED. DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED. YOU MAY REAPPLY ANY REMOVED FIX. Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, then fix2 may be removed 1) Shutdown WebSphere Process Server. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 2) Follow the Fix instructions that are packaged with the Fix Installer on how to uninstall and reinstall the Fix. 3) Restart WebSphere Process Server. Directions to re-apply fix: 1) Shutdown WebSphere. It is important that you perform a controlled and complete shutdown of the server to ensure that all transactions have completed, before installing the fix. 2) Run the UpdateInstaller and follow the instructions. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 3) Restart WebSphere Process Server. Additional Information: