Troubleshooting the situation update forwarder IBM Tivoli Monitoring and Tivoli Netcool/OMNIbus Troubleshooting the situation update forwarder In this training module, you learn how to check logs and troubleshoot issues that can arise when using the situation update forwarder on Tivoli® Monitoring and Tivoli Netcool®/OMNIbus. Debugging the situation update forwarder Debugging the situation update forwarder Set Loglevel to Verbose ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/etc-> vi situpdate.conf fileSize=50000 fileNumber=10 fileLocation="/opt/IBM/Netcool/omnibus/SitForwarder/persistence" pollingInterval=3 crcBytecount=50 cmsSoapUrl=cms/soap bufferFlushRate=10 logLevel=verbose If you encounter a problem while using the situation update forwarder, open the system logs, which are synched with Tivoli Monitoring and are in the /tmp folder. The most useful log is the trace log, which shows how the situation update forwarder handles an event. Be sure the Log Level is set to verbose. If you think the situation update forwarder is not running when you examine the logs, open the persistence directory. If a running file is present, the situation update forwarder is running. SitUser.conf SitUser.conf ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/etc-> vi situser.conf serverid=itm-vm100.tivlab.austin.ibm.com userid=root passwordfile=/opt/IBM/Netcool/omnibus/SitForwarder/etc/itm-vm100.tivlab.austin.ibm.com.pwd serverid=sles10.tivlab.austin.ibm.com userid=root passwordfile="/opt/IBM/Netcool/omnibus/SitForwarder/etc/sles10.tivlab.austin.ibm.com.pwd" Situser.conf is a file that is sent to the two Tivoli Enterprise Monitoring Servers whose information is displayed. The situpdate.conf file is in SitForwarder/etc. Situation update forwarder executables Situation update forwarder executables ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/bin-> ls launch.sh sitconfigsvruser.log startSUF.sh nohup.out stopSUF.sh query_state.sh sitconf.sh sitconfig.log sitconfuser.sh test.sh sitconfig.log.lck situpdate.sh These executables are the situation update forwarder executables. To change passwords, set up another server ID, location, or user. Then, use sitconfuser.sh or sitconf.sh. /tmp/itmsynch/logs/synch_tracelogs /tmp/itmsynch/logs/synch_tracelogs synch_trace.log 2009.11.19 16:48:12.611-06:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP SOAP string: rootxxxxxloginssles10:LZ 2009.11.19 16:48:12.611-06:00 com.tivoli.candlenet.SituationUpdateForwarder getServerUrlConnection IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Entry, parm 1 = op=a,sn=logins,sh=sles10:LZ,sv=sles10.tivlab.austin.ibm.com:3661,et=,di= 2009.11.19 16:48:12.612-06:00 com.tivoli.candlenet.SOAPConnection sendRequest IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP SOAP URL is: https://sles10.tivlab.austin.ibm.com:3661///cms/soap/kshhsoap.htm 2009.11.19 16:48:12.612-06:00 com.tivoli.candlenet.SOAPConnection sendRequest IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Method name is: CT_Acknowledge 2009.11.19 16:48:12.663-06:00 com.tivoli.candlenet.SOAPConnection getResponseCode IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Response code is: 200 In this example, you see a synch_tracelog. Find the dialog CT_Acknowledge near sles10 Tivoli Austin IBM, on port connection 3661. Now, find code is: 200 at the bottom of the trace log. This code confirms that the event was sent to Tivoli Enterprise Monitoring Server. Check NCO_PA.log if event is not in /persistence Check NCO_PA.log if event is not in /persistence ROOT@TORPEDO /opt/IBM/Netcool/omnibus/log-> cat NCO_PA.log Two procedures are running to check the situation update forwarder status; one is running the wrong syntax The correct version has the full path; the incorrect version is missing /omnibus/send_event_cmd in the path Correct "usr/local/app2/opt/IBM/SitForwarder/omnibus/send_event_cmd /usr/local/app2/opt/IBM/SitForwarder Check_SUF_Status Check_SUF_Status“ Incorrect /usr/local/app2/opt/IBM/SitForwarder Check_SUF_Status Check_SUF_Status“ Check the persistence directory for your event. Check the NCO_PA log for your event. You should always check the NCO_PA log, even if your event did not register in the persistence directory. The NCO_PA log is shown. Find the send_event command. In this example, the send_event command is present, but the path is incorrect. If you receive an error stating that an event did not arrive, ensure that the path is correct. opt/IBM/Netcool/omnibus/SitForwarder/persistence opt/IBM/Netcool/omnibus/SitForwarder/persistence drwxr-xr-x 2 root system 256 19 Nov 16:51 . drwxr-xr-x 13 root system 4096 04 Nov 18:13 .. -rw-r--r-- 1 root ncoadmin 26 19 Nov 16:51 lastwritten -rw-r--r-- 1 root system 0 03 Dec 12:43 running -rw-r--r-- 1 root ncoadmin 1444 19 Nov 16:51 situpdate_1257380230 This is the persistence directory. If a situpdate file is not present in the persistence directory, then the situation update forwarder is not running. The situpdate example displayed is an event that has arrived. Open the file and check the event name. If the file name has been shortened or truncated in the situation update forwarder file, you must replace the shortened name with the original name. A truncated file name causes an error that prevents the event from being delivered. Check for the insert statement in NCOMS.log Check for the insert statement in NCOMS.log Look for the insert statement proof of the Tivoli Monitoring SQL triggers firing in NCOMS.log if event is not in /persistance 08/25/09 11:00:21: Debug: D-OBJ-105-010: Client language command [probe][PROBE][tivoli_eif][pokdevlab75] [insert into alerts.status values ( 'MS_Offline:usrdtep01i9012b:UA::ITM_ManagedSystem',0,'usrdtep01i9012b:UA','usrdtep01i9012b:UA','EIF Probe on pokdevlab75','EIF Probe on pokdevlab75','ITM_ManagedSystem','MS_Offline',4,'MS_Offline',1251216021,1251216021,1251216021,1251216021,0,20,0,6601,1,'',65534,0,0,0,'',0,0,0,'','',0,0,'',0,'',0,0,'',0,0,'','','','','','','','',0,0,'','','USPK0P0ALLD',0,'','','','','08/25/2009',0,'N','','','08/25/2009 12:00:21.000','usrdham01i901xa.raleigh.ibm.com','3661','U','','S','HUB_usrdham01i901xa','','','','','','','','MS_Offline','','','',0,'',0,'','','','usrdtep01i9012b:UA','','','','LAB','','','',0,'','',0,0,1251216021,1251216021,0,'','','',0,0,1251216021,'',0,'',1251216021,0,0,0,'',0,1251216021,0,0,1251216021,0,'','','',0,0,0,0,0,1251216021,0,'','','','',0) updating (ITMStatus,ITMTime,ITMHostname,ITMResetFlag);]. This is the NCOMS log. When set to debug, it is also the Tivoli Netcool/OMNIbus log. Look for any errors in the insert statement for the Tivoli Monitoring SQL triggers firing. Test SOAP connection Test SOAP connection Stop the situation update forwarder Delete all situation update forwarder cache files and file called lastwritten (in $NCHOME/ITM/SitForwarder/persistence/) Make sure you are logged in as the same user used to install the situation update forwarder (typically, this is Netcool) Restart the situation update forwarder with StartSUF.sh Run test.sh to test SOAP connection to Tivoli Monitoring Acknowledge events in OMNIbus and confirm that they are modified in Tivoli Monitoring If you have not found the source of an error, test the SOAP connection. Stop the situation update forwarder. Delete all the situation update forwarder cache files called lastwritten. Make sure you are logged in as the same user that installed the situation update forwarder. Typically, this user is netcool. Restart the situation update forwarder using StartSUF.sh and run test.sh to test the SOAP connection to Tivoli Monitoring. You are also confirming that events are modified within Tivoli Monitoring. If the test completes, it confirms that the situation update forwarder is communicating with the Tivoli Enterprise Monitoring server. Iptrace of SOAP Iptrace of SOAP Iptrace on SOAP connections Start Iptrace on Omnibus server tracing SOAP ports 1920 and 3661 startsrc -s iptrace -a " -a -p 1920,3661 -b /tmp/trace.out" Run an IP trace on SOAP connections. Start on the Tivoli Netcool/OMNIbus server tracing SOAP ports 1920 and 3661. WireShark output of IP trace WireShark output of IP trace WireShark is a free downloadable tool that can display the SSL log connection and the can-ferret-ssl connection. You can use this tool to find out what connections you are receiving over the port. Here is the WireShark output of the IP trace checking connectivity of the Tivoli Netcool/OMNIbus server and the Tivoli Enterprise Monitoring Server. Summary Summary In this presentation, you learned how to check logs to troubleshoot issues that might arise in the situation update forwarder In this presentation, you learned how to check logs to troubleshoot issues that might arise in the situation update forwarder. Feedback Feedback Your feedback is valuable You can help improve the quality of IBM Education Assistant content to better meet your needs by providing feedback. Did you find this module useful? Did it help you solve a problem or answer a question? Do you have suggestions for improvements? Click to send email feedback: mailto:iea@us.ibm.com?subject=Feedback_about_troubleshooting_suf.ppt This module is also available in PDF format at: ../troubleshooting_suf.pdf You can help improve the quality of IBM Education Assistant content by providing feedback. Trademarks