Tips for troubleshooting WS-Notification publish and subscribe messaging for web services.
To help you identify and resolve WS-Notification problems, use the WebSphere® Application Server trace and logging facilities as described in Tracing and logging configuration Setting up component trace (CTRACE).
To enable trace for WS-Notification, set the application server trace string to SIBWsn=all=enabled:com.ibm.ws.sib.webservices.*=all=enabled. If you encounter a problem that you think might be related to WS-Notification, you can check for error messages in the WebSphere Application Server administrative console, and in the application server SystemOut.log file. You can also enable the application server debug trace to provide a detailed exception dump.
A list of the main known restrictions that apply when using WS-Notification is provided in WS-Notification: Known restrictions.
WebSphere Application Server system messages are logged from a variety of sources, including application server components and applications. Messages logged by application server components and associated IBM products start with a unique message identifier that indicates the component or application that issued the message. The prefix for the WS-Notification component is CWSJN.
The Troubleshooter reference: Messages topic contains information about all WebSphere Application Server messages, indexed by message prefix. For each message there is an explanation of the problem, and details of any action that you can take to resolve the problem.
<wsdl:operation name="oneWayRawSubscriptionNotify"> <soap:operation soapAction="http://docs.oasis-open.org/wsn/bw-2/NotificationConsumer/Notify" /> <wsdl:input name="oneWayRawSubscriptionNotifyRequest"> <soap:body use="literal" /> </wsdl:input> </wsdl:operation>
<wsdl:operation name="oneWayRawSubscriptionNotify"> <wsdl:input message="impl:oneWayRawSubscriptionNotifyRequest" name="oneWayRawSubscriptionNotifyRequest" wsam:Action="http://docs.oasis-open.org/wsn/bw-2/NotificationConsumer/Notify"> </wsdl:input> </wsdl:operation>
[ERROR] the following naming conflicts occurred: com.ibm.websphere.wsn.publisher_registration_manager.ResourceNotDestroyedFault line 2 of file:/path_to_wsdl/PublisherRegistrationManager.wsdl
This fault occurs because the WSDL for the publisher registration manager uses both the WS-Notification and the WS-ResourceLifetime specifications; both of which refer to a ResourceNotDestroyedFault element that shares the same message name. Here is the relevant part of the publisher registration manager WSDL file:
<wsdl:operation name="DestroyRegistration"> <wsdl:input name="DestroyRegistrationRequest" message="wsn-brw:DestroyRegistrationRequest" wsam:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest" wsaw:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest"/> <wsdl:output name="DestroyRegistrationResponse" message="wsn-brw:DestroyRegistrationResponse" wsam:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse" wsaw:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse"/> <wsdl:fault name="ResourceUnknownFault" message="wsrf-rw:ResourceUnknownFault" wsam:Action="http://docs.oasis-open.org/wsrf/fault" wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/> <wsdl:fault name="ResourceNotDestroyedFault" message="wsn-brw:ResourceNotDestroyedFault" wsam:Action="http://docs.oasis-open.org/wsn/fault" wsaw:Action="http://docs.oasis-open.org/wsn/fault"/> </wsdl:operation> <!-- Some parts have been omitted --> <!-- An extract from WS-ResourceLifetime --> <wsdl:operation name="Destroy"> <wsdl:input name="DestroyRequest" message="wsrf-rlw:DestroyRequest" wsam:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest" wsaw:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest"/> <wsdl:output name="DestroyResponse" message="wsrf-rlw:DestroyResponse" wsam:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse" wsaw:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse"/> <wsdl:fault name="ResourceNotDestroyedFault" message="wsrf-rlw:ResourceNotDestroyedFault" wsam:Action="http://docs.oasis-open.org/wsrf/fault" wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/> <wsdl:fault name="ResourceUnknownFault" message="wsrf-rw:ResourceUnknownFault" wsam:Action="http://docs.oasis-open.org/wsrf/fault" wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/> <wsdl:fault name="ResourceUnavailableFault" message="wsrf-rw:ResourceUnavailableFault" wsam:Action="http://docs.oasis-open.org/wsrf/fault" wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/> </wsdl:operation>
To resolve this naming conflict, you must include the JAX-WS bindings file ibm-wsn-jaxws.xml as an argument to the wsimport command. This bindings file ensures that the conflicting elements are mapped to different class names.
c:\was\bin\wsimport -b ibm-wsn-jaxws.xml -keep PublisherRegistrationManager.wsdl
You have a JAX-WS demand-based publisher registered with a Version 7.0 type of WS-Notification service. When the service invokes any of the operations on the PausableSubscriptionManager interface of the publisher, the publisher responds with a triggerActionNotSupportedFault exception message. The operations that trigger this message are Renew, Unsubscribe, PauseSubscription or ResumeSubscription.
You see messages similar to the following text in the SystemOut.log file for the server. In this example the fault message is triggered in response to the broker attempting to unsubscribe from the demand-based publisher.
triggerActionNotSupportedFault triggerActionNotSupportedFault: messageContext: [MessageContext: logID=urn:uuid:13616A3EB4F278A3DC1221827497002] problemAction: http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest CWSJN1029W: An attempt was made to perform an operation on a remote NotificationProducer located with endpoint Address[[xxx/prod_service/NotificationProducerPort]], ReferenceParams[] but the operation could not be completed. This operation will be retried automatically in 2 seconds. The operation failed due to com.ibm.ws.sib.wsn.webservices.WSNWSException: CWSJN5035E: An internal error occurred: Unable to unsubscribe from remote publisher at endpoint reference Address[[xxx/prod_service/PausableSubscriptionManagerPort]], ReferenceParams[] due to javax.xml.ws.soap.SOAPFaultException: The [action] cannot be processed at the receiver.
The server continues to unsuccessfully attempt to unsubscribe from the publisher at ever-increasing time intervals.
To resolve this problem, in the WSDL file for your demand-based publisher you must explicitly specify the SOAP action to use for each of the operations on the PausableSubscriptionManager interface. The action URI to use for each operation is defined in the Web Services Base Notification 1.3 (WS-BaseNotification) specification available from http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsn For example, the WS-Addressing action for the Unsubscribe operation is defined in the specification as http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest, therefore you must use this action for the Unsubscribe operation in your publisher WSDL file. Here is an excerpt from the binding section of such a WSDL file:
<wsdl:binding name="PausableSubscriptionManagerBinding" type="wsn-bw:PausableSubscriptionManager"> <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" /> <wsdl:operation name="Unsubscribe"> <soap:operation soapAction="http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest" />
Similarly you must specify the corresponding actions for the Renew, PauseSubscription and ResumeSubscription operations in your publisher WSDL file.
If you see WebServicesFault( "Connection timed out" ) errors in your WS-Notification broker server logs, and you have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the maximum number of connections in the HTTP outbound connector connection pool when sending outbound notification messages to subscribed consumers.
To increase the number of outbound HTTP connections in the pool, set the com.ibm.websphere.webservices.http.maxConnection custom property on the server or servers on which your broker is running. For more information about this property, see HTTP transport custom properties for Web services applications.
If you see "Out of memory" errors in your WS-Notification broker server logs, and you have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the available JVM heap size allocated to the server on which it is running.
To increase the maximum heap size for the server or servers on which your broker is running, see Tuning the IBM virtual machine for Java.
The WSDL and schema files shipped with WebSphere Application Server Version 7.0 are updated to reflect the final Version 1.3 WS-Notification standards.
You need not change your existing WS-Notification services. Your existing client applications will also continue to work unchanged, but if they are now also working with new WS-Notification services, and you want them to explicitly handle the new fault conditions, then regenerate the client stubs by using the WSDL file from the new service.
If you try to create a Version 6.1 WS-Notification service, and you get the following stack trace, then the SDO repository is not configured correctly. To resolve this problem, see Installing and configuring the SDO repository.
java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: Failed to store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker due to the following exception: com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: CWSWS1007E: The following exception occurred: com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: javax.transaction.TransactionRolledbackException: ; nested exception is: javax.ejb.TransactionRolledbackLocalException: ; nested exception is: com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: PMGR1014E: Exception occurred when getting connection factory: com.ibm.websphere.naming.CannotInstantiateObjectException: threw NameNotFoundException while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is javax.naming.NameNotFoundException: Context: smeagolNode03Cell/nodes/smeagolNode03/servers/server1, name: jdbc/com.ibm.ws.sdo.config/SdoRepository: First component in name com.ibm.ws.sdo.config/SdoRepository not found. [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.
In some situations you might receive more notifications at a given notification consumer than the number of event notifications that have been inserted into the notification broker by a publisher. For example you might publish 4 messages, and receive 8, 12, 16 (or some other multiple of four) messages at the notification consumer.
This is usually caused by there being two or more active subscriptions that target the notification consumer - a situation that can occur if the subscriber application is run more than once. Each time the Subscribe operation is called, a new subscription must be created by the notification broker (see section 4.2 of the Web Services Base Notification specification), which causes duplicate messages to be delivered if a previous subscription exists.
To check whether this is what is happening, examine the SubscriptionReference property of the notifications received by the notification consumer. This endpoint reference contains the identifier of the subscription that caused the notification to be sent. If you find several different subscription identifiers, then there is more than one subscription active.
Subscriber applications should tidy up subscriptions when they are not required (or register them with a timeout), however you can tidy them up administratively using the runtime panels as described in Listing or deleting active WS-Notification subscriptions.
You should be wary of deleting and recreating messaging engines on bus members for which WS-Notification-administered subscribers have been configured, because in some cases this can leave the remote web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it.
To avoid this situation you should delete the WS-Notification configuration, or just the administered subscribers, in a separate step to deleting the messaging engine. When the dynamic configuration update is then processed, or the server restarted, the remote web service subscription is tidied up cleanly.
When you remove messaging engines from a cluster, remove them in numerical order from highest to lowest so as to avoid a situation where, for example, there are messaging engines numbered 001 and 002 and not 000. This is to avoid problems if you use WS-Notification, which attaches special significance to the first-created messaging engine in a cluster.
In a clustered topology there can be more than one messaging engine running on the "bus member" (cluster). Administered subscribers are defined against a service point (bus member) and so there are several alternatives when choosing the messaging engine that is responsible for creating the subscription to the remote web service. In this situation, the "first" messaging engine in the cluster is responsible for making the subscription. For example in a cluster containing three messaging engines the messaging engines will have names following the pattern xxx-000-yyy, xxx-001-yyy, xxx-002-yyy, and the administered subscriber subscriptions will be managed by the "000" messaging engine.
If you delete the "000" messaging engine from the cluster then restart the servers, the administered subscriptions are now managed by the "001" messaging engine - being the lowest number engine in the cluster. However, as previously mentioned, deleting and recreating messaging engines on bus members for which administered subscribers have been configured can leave the remote web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it. Therefore if another messaging engine is later added to the cluster and there is no xxx-000-yyy messaging engine currently defined the new engine takes on the name of xxx-000-yyy. Therefore, in this instance it is possible for two messaging engines to concurrently believe that they manage the administered subscription, resulting in multiple subscriptions being made to the remote web service.
Usually, a bus destination is used as described in Bus destinations. However, this is not the case for Version 6.1 WS-Notification services. The destination that is associated with a Version 6.1 WS-Notification service does not relate to the topics for which the WS-Notification service can handle requests, and you should not alter or mediate the destination. In WS-Notification, the configuring of topics is handled through topic namespaces. For more information, see Creating a new WS-Notification permanent topic namespace.
Applications that want to publish event notifications into the broker make use of the Notify operation. This is defined as a one-way (web services) operation which means that it is not possible to return a fault (exception) if it is not possible to complete the operation. Thus the application will assume that the notification was successful, but subscribing applications will not receive the notification message.
You should monitor this type of exception closely, because it might indicate a denial of service attack and certainly indicates that the application is not functioning correctly. The first time an inbound notification fails from a particular producing application, a warning message is sent to the SystemOut log of the server. If there are further notification failures for that producer, subsequent timed warning messages are logged at 30 minute intervals. Additional information is provided with each timed message to indicate how many failed notifications were received for that producer during the 30 minute interval.
An outbound web service invocation (broker to application) does not succeed when a remote application is unavailable for invocation. This might be the result of an application failure, a network error, or a firewall configuration issue. When event notifications are not passed to subscribed applications, messages build up on the subscriptions held on the server. The messages held on a given subscription can be observed by using the runtime panels as described in Listing or deleting active WS-Notification subscriptions. Subscriptions for which the most recent event notification attempt has failed in this way are marked as being in ERROR state when viewed in the WS-Notification subscription runtime administration panel.
If the WS-Notification service point does not successfully notify a NotificationConsumer application, a warning message is sent to the application server SystemOut log and the subscription is told to wait for 2 minutes. Reasons for a failure of this type might be that the remote web service is not currently available, or that network conditions prevent contact between the local server and the service.
After 2 minutes, the notification is retried. If delivery is still not possible then the subscription is put back into a wait state for another 2 minutes. If the failure is caused by a transient I/O error, this pattern is repeated indefinitely, until the notification is either successfully delivered or you delete the subscription. If the error is caused by an application failure on the remote side then the notification will be retried up to the number of times defined in the "Maximum failed deliveries" setting of the service integration bus topic space destination from which the message is being received. After the first warning message is output to the SystemOut log, subsequent timed warning messages are logged at 30 minute intervals.
The act of subscribing to the broker or registering a publisher creates a stateful resource on the server that consumes system resources while it is active. Usually an application specifies a termination time as part of the act of creating these resources, and thus they are automatically deleted when the termination time is reached. However it is also possible for the application to request an infinite lifetime for the resource. If this is done then it is possible for resources to remain on the server indefinitely even though the application might never be coming back to use (or destroy) the resource.
You can view the stateful resources (subscriptions and publisher registrations) by using the runtime panels described in Interacting at run time with WS-Notification. These panels also provide the ability to administratively delete the items if required. Only do this if you are sure that the application is no longer using the resource because it will cause application failures if the resource is referenced after being deleted.
When you create a subscription by using a WS-Notification application, that is by using the Subscribe operation, one or more durable subscriptions are created in the relevant service integration bus topic space destination. You can view these durable subscriptions in the service integration bus runtime panels for the publication point.
To delete a subscription that was created by a WS-Notification application, use the runtime panels provided by the WS-Notification implementation, as described in Interacting at run time with WS-Notification. This approach closes the active consumer and automatically deletes the related service integration bus durable subscriptions.
WebSphere Application Server depends on being able to access a running service integration bus messaging engine to send and receive messages, and to create and retrieve state for the various web service resources that are created.
You can stop a messaging engine by using the MBean interface or runtime panels. This prevents WS-Notification from successfully servicing any requests from applications that might come in during the time that the messaging engine is stopped. In this situation, error messages are logged as described in An inbound (application to broker) notification is not successful and An outbound (broker to application) notification is not successful. When you stop a messaging engine, all WS-Notification processing stops and all messaging applications cease to function. When you restart the messaging engine, WS-Notification processing resumes.
The WS-Notification configuration artifacts often depend on objects defined in other areas of the server configuration. For example (for Version 6.1 WS-Notification services) the endpoint listeners through which application requests are received, and the service integration bus topic spaces to and from which messages are sent.
The following items describe the action that is taken by the WS-Notification runtime code when it meets relevant changes in the objects upon which it depends.
The service integration bus topic space is the primary messaging object upon which WS-Notification depends at run time. Notification messages from an application are published to the topic space specified by the (permanent) topic namespace mapping specified by the administrator.
Deleting the topic namespace mapping that was used to establish a (currently active) subscription has the same effect as deleting the underlying service integration bus topic space as defined previously, and subscriptions that were created using this namespace mapping are deleted.
Publisher registrations and pull points associated with the deleted topic namespace mapping are also deleted.
The fields of a permanent topic namespace mapping are read-only fields, so the only way to "change" the fields is to delete the namespace mapping and recreate it with new values. The effect of deleting a permanent topic namespace mapping is described in the previous item.
java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: Failed to store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker due to the following exception: com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: CWSWS1007E: The following exception occurred: com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: javax.transaction.TransactionRolledbackException: ; nested exception is: javax.ejb.TransactionRolledbackLocalException: ; nested exception is: com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: PMGR1014E: Exception occurred when getting connection factory: com.ibm.websphere.naming.CannotInstantiateObjectException: threw NameNotFoundException while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is javax.naming.NameNotFoundException: Context: KADGINNode01Cell/nodes/KADGINNode01/servers/server1, name: jdbc/com.ibm.ws.sdo.config/SdoRepository: First component in name com.ibm.ws.sdo.config/SdoRepository not found. [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.For details on how to configure the SDO repository, see Installing and configuring the SDO repository.
WSDLException (at /definitions/import[1]): faultCode=OTHER_ERROR: Unable to resolve imported document at 'http://docs.oasis-open.org/wsn/brw-2.wsdl', relative to 'http://localhost:9082/WSNService1WSNServicePt1NB/Service/WEB-INF/wsdl/NotificationBroker.wsdl': java.net.UnknownHostException: docs.oasis-open.org
Configure the client to use a local copy of the WSDL file instead, by following the instructions in the topic Configuring a JAX-WS client to resolve a WS-Notification service WSDL without following web links.