Use this set of specific tips to help you troubleshoot problems with service integration messaging engines.
A messaging engine fails to start and the following error is displayed in the WebSphere® Application Server administrative console:
The messaging engine <name> cannot be started as there is no runtime
initialized for it yet, retry the operation once it has been initialized.
If dynamic configuration reload is enabled for this bus, then the servers
must be restarted.
Prior to trying to start the messaging engine again, make sure that you have restarted the server. For the runtime to initialize successfully, the application server must be started.
When attempting to use the DB2® Universal JDBC type 2 driver to store data on the z/OS platform, the messaging engine does not start up and "Storage Allocation Error" messages similar to the following message might appear in the WebSphere Application Server SystemOut.log file:
BBOO0220E: [SB6NLA1:SB6NLA1.server1-SB6NLA1] CWSIP0002E: An
internal messaging error occurred in com.ibm.ws.sib.processor.im
pl.MessageProcessor, 1:1469:1.365, com.ibm.ws.sib.msgstore.Messa
geStoreRuntimeException: com.ibm.ws.sib.msgstore.PersistenceExce
ption: CWSIS1501E: The data source has produced an unexpected
exception: com.ibm.db2.jcc.t2zos.y: [IBM/DB2][T2zos/2.5.48]T2zo
sPreparedStatement.readPrepareDescribeOutput_:processDescribeOut
put:1563:Storage Allocation Error at com.ibm.ws.sib.msgstore.cac
he.links.AbstractItemLink.readDataFromPersistence(AbstractItemLi
nk.java:2487) at
com.ibm.ws.sib.msgstore.cache.links.AbstractItemLink._restoreIte
m(AbstractItemLink.java:639)
The fullyMaterializeLobData custom property determines whether LOB data is fully materialized in the JDBC driver when a row is fetched, or is retrieved in pieces as needed. The actual behavior depends on whether the database server supports progressive streaming. Refer to your DB2 documentation for more information about this property. The default value is true.
When attempting to use the Informix® JDBC driver 3.00JC1 to store data, the messaging engine cannot start up and the following error message might appear in the WebSphere Application Server SystemOut.log file:
00000022 SibMessage E [RetireBus:retire_web.000- RetireBus] CWSIS0002E: The messaging engine encountered an exception while starting. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: java.sql.BatchUpdateException: Unique constraint (informix.u114_62) violated. 00000022 SibMessage E [RetireBus:retire_web.000- RetireBus] CWSID0035E: Messaging engine retire_web.000-RetireBus cannot be started; detected error reported during com.ibm.ws.sib.msgstore.impl.MessageStoreImpl start() 00000022 SibMessage E [RetireBus:retire_web.000- RetireBus] CWSID0027I: Messaging engine retire_web.000-RetireBus cannot be restarted because a serious error has been reported.T] 00000022 SibMessage I [RetireBus:retire_web.000- RetireBus] CWSID0016I: Messaging engine retire_web.000-RetireBus is in state Stopped.
There is a known defect (PTS 172471) in the Informix JDBC Driver 3.00JC1. To avoid this error, upgrade the Informix JDBC Driver to 3.00JC2.
AdminControl.invoke(AdminControl.queryNames("type=SIBMessagingEngine,
name=messagingenginename,*"),
"dump", "com.ibm.ws.sib.msgstore.*")
$AdminControl invoke [$AdminControl queryNames type=SIBMessagingEngine,
name=messagingenginename,*]
dump com.ibm.ws.sib.msgstore.*
The dump is created as an XML file in the $WAS_HOME/logs/server1 directory. The file is named according to the format: messaging_engine_nameUUIDtimestamp.xml
<MessageStore>
<itemStreams>
<ItemStreamLink id="0" state="Available">
<class>com.ibm.ws.sib.msgstore.ItemStream</class>
<priority>5</priority>
<canExpireSilently></canExpireSilently>
<storageStrategy>STORE_NEVER</storageStrategy>
<expiryTime>0</expiryTime>
<sequence>0</sequence>
<tranID>null</tranID>
<tickValue>0</tickValue>
<items>
<ItemLink id="2" state="Available" refCount="3" refCountDecreasing="false">
<class>com.ibm.ws.sib.msgstore.Item</class>
<priority>5</priority>
<canExpireSilently></canExpireSilently>
<storageStrategy>STORE_NEVER</storageStrategy>
<expiryTime>0</expiryTime>
<sequence>1</sequence>
<tranID>null</tranID>
<tickValue>0</tickValue>
</ItemLink></items></ItemStreamLink></itemStreams></MessageStore>
CWSIS1546I: The messaging engine, ME_UUID={0}, INC_UUID={1}, has lost an existing lock or failed to gain an initial lock on the data store.
java.lang.Exception: java.sql.SQLException: null userid not supported DSRA0010E: SQL State = null, ErrorWhen you create a new Network Attached Apache Derby data store, by default you get a blank authentication alias.If you use Apache Derby in Network Attached mode with the DB2 Universal JDBC Driver (that is, you use the "JDBC provider for Derby Network Server using the (DB2) Universal JDBC Driver"), you must specify an authentication alias. This requirement is documented in Data source minimum required settings for Apache Derby.
To solve this problem, you must follow the procedure described in Resolving indoubt transactions.
The XAResourceNotAvailableException exception can also be thrown when a server in a cluster bus member fails over. In this case, no operator intervention is required to recover and resolve transactions.
[8/11/04 21:55:01:439 CDT] 0000000f SibMessage I [LateBus:xyzsun15.server1-LateBus] isAlive: MessagingEngine suffered common mode error. Correct error (see logs) and restart server. [8/11/04 21:55:01:468 CDT] 0000000f SibMessage I [LateBus:xyzsun15.server1-LateBus] isAlive: MessagingEngine will be stopped because of common mode error. No failover will occur. [8/11/04 21:55:01:493 CDT] 0000000f SibMessage I [LateBus:xyzsun15.server1-LateBus] Messaging Engine xyzsun15.server1-LateBus not in state from which stop is valid: Starting [8/11/04 21:55:01:513 CDT] 0000000f SibMessage I [LateBus:xyzsun15.server1-LateBus] isAlive: MessagingEngine stopped because of common mode error. Correct error (see logs) and restart server. [8/11/04 21:57:01:431 CDT] 0000000e SibMessage I [LateBus:xyzsun15.server1-LateBus] isAlive: MessagingEngine suffered common mode error. Correct error (see logs) and restart server.
The messaging engine failed to start because the database directory for the messaging engine still exists once deletion of the bus and you must manually remove it. To delete the Apache Derby database for a non-existent messaging engine, you must delete the database directory that is located in profile_root/databases/com.ibm.ws.sib, where profile_root is the directory in which profile-specific information is stored.
You must stop WebSphere Application Server prior to deleting the database files.
For other databases, you can either delete all of the rows from the data store tables or you can drop all of the data store tables. These tables are in the schema that you configured for the data store. For a list of the tables, refer toData store tables.
For more information, see Data store life cycle.
To enable communication between buses, a foreign bus and a service bus integration link must be created. On the first bus, the name of the foreign bus must match the name of the second bus that becomes a foreign bus, and the name of the foreign bus for this second bus must match the name of the first bus. The service integration bus link must be have the same name on both buses.
You may encounter the following type of error if your configuration is not correct, for example because the service integration bus links do not match:
SibMessage E [TechBus:TechCluster.000-TechBus] CWSIT0057E: The inter-bus connection BookstoreBus failed in the remote messaging engine on host aixp401.rchland.ibm.com with reason: CWSIT0067E: Inter-bus connection BookstoreBus in bus BookstoreBus is not available.
The administrative console panel used for configuring the properties of a service integration bus link can also be used to change the foreign bus name that the link is pointing to. However, you must not alter the name of the foreign bus once it has been configured. If you do, any messaging engines that already hold state information about the link will not be able to use the link until the foreign bus name is reset to its previous value.
When the number of messages held by a destination reaches its limiting threshold, any attempt to send a message to that destination fails with a JMSException with a wrapped SILimitExceeded exception. The destination continues to fail with this exception until the number of messages held by the destination is reduced below the limiting threshold.
To obtain an accurate count of the number of available messages, you can monitor the Available Message Count PMI statistic for queue and topicspace destinations. If the number of available messages increases, take action to balance the system. Consider stopping producers from sending new messages until the destination consumes the available messages.
The ideal balance is for the number of messages produced and the number of messages consumed to be equal over a period of time. If your system is unbalanced and the producing application sends more messages than the destination can consume, the producing application eventually throws a JMSException.
Consider cloning your application across multiple servers in a non-clustered environment. By default, applications are cloned in a clustered server environment. To enable subscribers in a non-clustered environment, set the cloned flag in the TopicConnectionFactory JNDI setting for DurableSubscriptions.
It is possible, although rare, for a messaging engine, destination or link to be corrupted once a restart of the system. If this corruption occurs you will see a message indicating the problem. If the problem lies with the messaging engine, the messaging engine will not start. If a destination or link is corrupted, the relevant messaging engine will start, but the destination or link will not be usable on that messaging engine.
If you do not know the cause of the problem, contact your IBM service representative to establish the cause prior to attempting to resolve the situation.
To be able to retrieve the status of messaging engines, you must be logged into the administrative console with at least monitor authority. If you do not have this authority, the messaging engine status is displayed as "Unavailable", even if the messaging engine has started.
[4/20/05 10:49:57:083 CDT] 0000004b RoleBasedAuth A SECJ0305I: The role-based authorization check failed for admin-authz operation SIBMessagingEngine:stateExtended. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: administrator, operator, configurator, monitor.Where the user ID shown in the message is the user ID that you used to log in to the administrative console.
If an application depends on a messaging engine being available, then the messaging engine must be started prior running the application. If you want application server to start an application automatically, you should develop your applications to test that any required messaging engine has been started and, if needed, wait for the messaging engine. If this technique is used in a startup bean, then the startup bean method should perform the test and wait work in a separate thread (using the standard WorkManager methods), so that the application server startup is not delayed.
For an example of code to test and wait for a messaging engine, see Applications with a dependency on messaging engine availability.
CWSIV0759W: During activation of a message-driven bean, no suitable active messaging engines were found in the local server on the bus {0}.
When the messaging engines start, another information message confirms this and message handling can then occur.
CHFW0030E: Error starting chain {0} because of exception {1}
When the messaging engines start, another information message confirms this and message handling can then occur.
These messages only appear under certain circumstances, for example these messages might appear if you change ports during migration
Trace: 2009/06/17 08:24:41.434 01 t=9C6B58 c=UNK key=P8 (00000011) Description: Log Java Message Message: CHFW0030E: Error starting chain _InboundTCPProxyBridgeService because of exception com.ibm.wsspi.channel.framework.exception.RetryableChannelException: An exception was thrown when attempting to start the TCPProxyChannel com.ibm.ws.channel.framework.imp l.ChannelFrameworkImpl
Exception = com.ibm.wsspi.channel.framework.exception.RetryableChannelException Source = com.ibm.ws.channel.framework.impl.ChannelFrameworkImpl.startChainInternal probeid = 2577 Stack Dump = com.ibm.wsspi.channel.framework.exception.RetryableChannelException: An exception was thrown when attempting to start the TCPProxyChannel at com.ibm.ws.tcpchannelproxy.jfap.impl.TCPProxyInboundChannel.start(TCPProxyInboundChannel.java:153) at com.ibm.ws.channel.framework.impl.ChannelFrameworkImpl.startChannelInChain(ChannelFrameworkImpl.java:1410) at com.ibm.ws.channel.framework.impl.ChannelFrameworkImpl.startChainInternal(ChannelFrameworkImpl.java:2863) at com.ibm.ws.channel.framework.impl.WSChannelFrameworkImpl.startChainInternal(WSChannelFrameworkImpl.java:960) at com.ibm.ws.channel.framework.impl.ChannelFrameworkImpl.startChainInternal(ChannelFrameworkImpl.java:2794) at com.ibm.ws.channel.framework.impl.ChannelFrameworkImpl.startChain(ChannelFrameworkImpl.java:2779) at com.ibm.ws.runtime.component.ChannelFrameworkServiceImpl.startChain(ChannelFrameworkServiceImpl.java:666) at com.ibm.ws.sib.jfapchannel.framework.impl.ChannelFrameworkReference$TCPProxyBridgeServiceInboundChainStartupRunnable .run(ChannelFrameworkReference.java:1641) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550) Caused by: com.ibm.ws.tcpchannelproxy.jfap.NotYetInitializedException: Server is not yet initialized at com.ibm.ws.tcpchannelproxy.jfap.TCPProxyBridgeServicesImpl.startListening(TCPProxyBridgeServicesImpl.java:558) at com.ibm.ws.tcpchannelproxy.jfap.impl.TCPProxyInboundChannel.start(TCPProxyInboundChannel.java:131) ... 8 moreEventually the following message should be displayed indicating that the z/OS TCP proxy channel has started up correctly:
Trace: 2009/06/17 08:24:51.449 01 t=9C6B58 c=UNK key=P8 (13007002) ThreadId: 00000003 FunctionName: com.ibm.ws.channel.framework.impl.WSChannelFrameworkImpl SourceId: com.ibm.ws.channel.framework.impl.WSChannelFrameworkImpl Category: AUDIT ExtendedMessage: BBOO0222I: CHFW0019I: The Transport Channel Service has started chain _InboundTCPProxyBridgeService.
A messaging engine that is hosted on WebSphere Application Server Version 8.5 cannot fail over to a server that is hosted on a different WebSphere Application Server version. If you have a cluster bus member that consists of servers that are hosted on different WebSphere Application Server versions, you must ensure that the high availability policy is configured to prevent failovers.
To prevent failover of a Version 8.5 messaging engine to a server that is hosted on different version, configure the high availability policy for the messaging engine so that the cluster is effectively divided into two sets of servers, one set for Version 8.5 and another set for the servers that are hosted on different Versions. The high availability configuration ensures that the Version 8.5 messaging engine is restricted to the Version 8.5 servers only. For more information, see Configuring messaging engine failover for mixed version clusters