Fix (APAR): PM62615 Status: Fix Release: 7.0.0.25,7.0.0.23,7.0.0.21,7.0.0.19,7.0.0.17,7.0.0.15,7.0.0.13,7.0.0.11 Operating System: AIX,HP-UX,Linux,Solaris,Windows,z/OS Supersedes Fixes: CMVC Defect: PM62615 Byte size of APAR: 27040 Date: 2012-12-06 Abstract: Messages stuck on Messaging engine (ME) hosting publication point Description/symptom of problem: PM62615 resolves the following problem: ERROR DESCRIPTION: A JMS application publishes messages to a WebSphere Application Server v7.0 Service Integration Bus Topic. Messages on the topicspace start building up when the number of transactions per second is increased. Even when the transactions per second is ramped down, the messages are consumed very slowly. Messages reporting Hung threads and HMGR0152W messages are written to the SystemOut.log: WSVR0605W: Thread "SIBFAPInboundThreadPool : 3" (00000067) has been active for 618131 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:167) at com.ibm.ws.sib.jfapchannel.impl.rldispatcher.Receive ListenerDispatchBarrier.pass(ReceiveListenerDispatchBarrier. java:60) at com.ibm.ws.sib.jfapchannel.impl.rldispatcher.Receive ListenerDispatchQueue.enqueue(ReceiveListenerDispatch Queue.java:313) at com.ibm.ws.sib.jfapchannel.impl.rldispatcher.Receive ListenerDispatcher.queueInvocationCommon(ReceiveListener Dispatcher.java:571) at com.ibm.ws.sib.jfapchannel.impl.rldispatcher.Receive ListenerDispatcher.queueDataReceivedInvocation(Receive ListenerDispatcher.java:667) at com.ibm.ws.sib.jfapchannel.impl.InboundTransmissionParser. dispatchToConversationListenerMethod(InboundTransmission Parser.java:1157) at com.ibm.ws.sib.jfapchannel.impl.InboundTransmissionParser. dispatchToConversation(InboundTransmissionParser.java:996) at com.ibm.ws.sib.jfapchannel.impl.InboundTransmissionParser. parseConversationPayload(InboundTransmissionParser.java: 732) at com.ibm.ws.sib.jfapchannel.impl.InboundTransmissionParser. parse(InboundTransmissionParser.java:293) at com.ibm.ws.sib.jfapchannel.impl.ConnectionReadCompleted Callback.complete(ConnectionReadCompletedCallback.java:249) at com.ibm.ws.sib.jfapchannel.framework.impl.CFWIOReadRequest Context$1.complete(CFWIOReadRequestContext.java:198) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener. futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback (AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions (AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java: 138) at com.ibm.io.async.ResultHandler.complete(ResultHandler. java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop (ResultHandler.java:816) at com.ibm.io.async.ResultHandler$2.run(ResultHandler. java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java: 1550) HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 31 seconds. LOCAL FIX: NONE PROBLEM SUMMARY USERS AFFECTED: Users of the default messaging provider for IBM WebSphere Application Server version 7.0,8.0 and 8.5 PROBLEM DESCRIPTION: Messages stuck on Messaging engine (ME) hosting publication point RECOMMENDATION: None This problem is seen in a publish subscribe environment. The messages were not flowing from the source ME (the point where the messages were published) to the destination ME (the point where the durable subscription home was defined). This was caused due to a gap in ticks( internal state of the ME) at the source ME (where the messages are stuck). The messages remain in pending acknowledgment mode. This problem comes because the source ME has just now discovered a subscriber is added at the destination ME for first time. On the discovery of this, at the source ME a gap is formed in tick range ( internal book keeping data structure which is used to transfer messages between MEs). PROBLEM CONCLUSION: The code has been modified to fill the gap in the tick range. This APAR will ensure that message flow between MEs . The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.27 ,8.0.0.7 and 8.5.0.2. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980 Directions to apply fix: Fix applies to Editions: Release 7.0 _X_ Application Server (Express or BASE) _X_ Network Deployment (ND) __ Edge Components __ Developer Install Fix to all WebSphere installations unless special instructions are included below. Special Instructions: None NOTE: The user must: * Logged in with the same authority level when unpacking a fix, fix pack or refresh pack. * Be at V7.0.0.0 or newer of the Update Installer. Certain iFixes may require a newer version of the Update Installer and the Update Installer will inform you during the installation process if a newer version is required. This can be checked by reviewing the level of the Update Installer in file /updateinstaller/version.txt. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 1) If your iFix is delivered as a single file with a .pak extension, Copy the .pak file directly to the maintenance directory. If your iFix is delivered as a single file with a .zip extension, unzip the file into the maintenance directory. 2) Shutdown WebSphere Application Server. Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that maintenance is being applied to. 3) For IBM i users, use the update command to install and uninstall the interim fix. The IBM Information Center can provide additional details, if needed, on the use of this command. http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-iseries&topic=rins_update. For non-IBM i users, launch the Update Installer and click the Next button on the Welcome page. 4) Enter the directory path of the installation location of the WebSphere product you want to update, and click the Next button. 5) Select the "Install maintenance package" operation and click the Next button. 6) Enter the directory path of your maintenance directory where you have the maintenance packages (.pak files) and click the Next button. 7) The Available Maintenance Package to Install page should list all maintenance packages (.pak files) that it finds in the directory path provided in the previous step. The Update Installer will select the correct maintenance packages based on your system configuration and will not allow an invalid combination to be installed. Please keep the Update Installer recommendations and click the Next button and continue with the installation of the maintenance package. To determine why some maintenance packages have been identified as not applicable, see description in log found in /logs/tmp*/updatelogs.txt 8) For all platforms except Windows. In pre-install summary panel, use the "verify permission" feature to verify the user has permissions to apply updates to files associated with the selected maintenance. Correct any file permissions before clicking next to start the install. 9) Restart WebSphere Application Server. Directions to remove fix: NOTE: * The user must have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * FIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED * DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED * YOU MAY REAPPLY ANY REMOVED FIX Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, fix2 removed, and fix3 re-applied. 1) Shutdown WebSphere Application Server. Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that uninstall is being run against. 2) Start Update Installer 3) Enter the installation location of the WebSphere product you want to remove the fix. 4) Select "Uninstall maintenance package" operation. 5) Enter the file name of the maintenance package to uninstall (PKxxxxx.pak). 6) UnInstall maintenance package. 7) Restart WebSphere Directions to re-apply fix: 1) Shutdown WebSphere Application Server. 2) Follow the Fix instructions to apply the fix. 3) Restart WebSphere Application Server. Additional Information: