Fix (APAR): PI17533 Status: Fix Release: 8.5.5.3 Operating System: AIX,HP-UX,IBM i,Linux,Solaris,Windows,z/OS Supersedes Fixes: CMVC Defect: xxxxxx Byte size of APAR: 281902 Date: 2019-05-13 Abstract: jms message producer and consumer applications fail to produce and consume messages from service integration bus destination Description/symptom of problem: PI17533 resolves the following problem: ERROR DESCRIPTION: JMS application is configured to send messages to Websphere Application Server v7.0.x Service Integration Bus destination and an MDB is configured to consume messages from the destination.The message producer and consumer application suddenly stop producing and consuming messages from the destination.Javacores collected from the server hosting the applications and the server hosting the messaging engine show the threads in blocked state with the following stack: On the server hosting the messaging engine,7 out of 8 SpillDispatcher threads were hung while adding messages to the filestore: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/ws/objectManager/AbstractSingleFileObjectStore.reserve(A bstractSingleFileObjectStore.java:718(Compiled Code)) at com/ibm/ws/objectManager/ManagedObject.reserveSpaceInStore(Manag edObject.java:663(Compiled Code)) at com/ibm/ws/objectManager/ManagedObject.getSerializedBytes(Manage dObject.java:637(Compiled Code)) at com/ibm/ws/objectManager/InternalTransaction.add(InternalTransac tion.java:780(Compiled Code)) at com/ibm/ws/objectManager/Transaction.add(Transaction.java:243(Co mpiled Code)) at com/ibm/ws/sib/msgstore/persistence/objectManager/PersistableImp l.addToStore(PersistableImpl.java:454(Compiled Code)) at com/ibm/ws/sib/msgstore/persistence/objectManager/BatchingContex tImpl.insert(BatchingContextImpl.java:205(Compiled Code)) at com/ibm/ws/sib/msgstore/task/AddTask.persist(AddTask.java:371(Co mpiled Code)) at com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S pillDispatcherThread.writeBatch(SpillDispatcher.java:1669(Compil ed Code)) at com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S pillDispatcherThread.run(SpillDispatcher.java:711(Compiled Code)) at java/lang/Thread.run(Thread.java:738(Compiled Code)) On the server hosting the message producer and consumer applications 90+ threads were found with the following stack: Consumer threads: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo mplete(ExchangeReceiveListener.java:227(Compiled Code)) at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver sationImpl.java:988(Compiled Code)) at com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange(JFAPCo mmunicator.java:386(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy.performReceive( ConsumerSessionProxy.java:914(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy._receiveWithWai t(ConsumerSessionProxy.java:825(Compiled Code)) at com/ibm/ws/sib/comms/client/ConsumerSessionProxy.receiveWithWait (ConsumerSessionProxy.java:733(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receiveInboundMes sage(JmsMsgConsumerImpl.java:1114(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receive(JmsMsgCon sumerImpl.java:481(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueReceiverImpl.receive(JmsQueu eReceiverImpl.java(Compiled Code)) Producer threads: at com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi on(ConnectionProxy.java:1007(Compiled Code)) at com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio n(ConnectionProxy.java:932(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.(JmsMsgProd ucerImpl.java:369(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueSenderImpl.(JmsQueueSe nderImpl.java:60(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsQueueSessionImpl.instantiateProdu cer(JmsQueueSessionImpl.java:224(Compiled Code)) at com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer(JmsSes sionImpl.java:866(Compiled Code)) LOCAL FIX: None PROBLEM SUMMARY: USERS AFFECTED: Users of the default messaging provider for IBM WebSphere Application Server V7.0, V8.0 and V8.5 PROBLEM DESCRIPTION: Message producer application suddenly stops producing messages to the destination because dispacher threads are hung. RECOMMENDATION: None When persisting messages into the filestore, an object manager needs to ensure free space in the filestore to store the messages. So the "SpillDispatcher" thread is indefinitely waiting in AbstractSingleFileObjectStore.reserve(). Another thread notifies the waiting thread based on a condition which is controlled by a boolan variable which is not thread safe. So the waiting "SpillDispatcher" thread may not get notified and waits forever and this may eventually happen to all the dispacther threads. When all the dispacther threads are hung, the message producer hangs as there are no dispacther threads to persist messages. Consumer application waits on receiveWithWait() because there are no messages in the destination. This is a rare problem due to timing especially when there are more CPUs. This problem is applicable if file store is being used as a message store. PROBLEM CONCLUSION: Source code has been modified to ensure that the boolean variable is thread safe. The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.37, 8.0.0.10, and 8.5.5.4. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980 Directions to apply fix: NOTE: Mark with an X the: 1) Release the fix applies to 2) The Editions that apply 3) And then DELETE THIS NOTE Fix applies to Editions: Release 8.5 __ Application Server (Express or BASE) __ Network Deployment (ND) __ Liberty Core __ Edge Components __ Developer Install Fix to all WebSphere installations unless special instructions are included below. Special Instructions: None NOTE: The user must: * Logged in with the same authority level when unpacking a fix, fix pack or refresh pack. * Be at V1.4.3 or newer of the Installation Manager. Certain iFixes may require a newer version of the Installation Manager and the Installation Manager will inform you during the installation process if a newer version is required. The IBM Knowledge Center can provide details, if needed, on the use of the Installation Manager to apply the iFixes. http://publib.boulder.ibm.com/infocenter/install/v1r4/index.jsp. Shutdown WebSphere Application Server before applying the iFixes. Restart WebSphere Application Server after applying the iFixes. Directions to remove fix: The IBM Knowledge Center can provide details, if needed, on the use of the Installation Manager to remove the iFixes. http://publib.boulder.ibm.com/infocenter/install/v1r4/index.jsp. Shutdown WebSphere Application Server before removing the iFixes. Restart WebSphere Application Server after removing the iFixes. Directions to re-apply fix: 1) Shutdown WebSphere Application Server. 2) Follow the Fix instructions to apply the fix. 3) Restart WebSphere Application Server. Additional Information: