1. ITM/OMNIBUS Synchronization (SUF) Integration Presenters: Calvin Moody & Ralph Baldwin

2. Presentation Objectives Install of Situation Update Forwarder (SUF) Sending ITM events to Omnibus and Resynching them back to ITM Trouble Shooting various aspects of SUF

3. SUF Instalation Install Guide for SUF http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?topic=/com.ibm.itm.d oc/itm_install313.htm http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?topic=/com.ibm.itm.d oc/itm_install313.htm PassPort Advantage Part # C167UEN.tar You can downloaded ITM Tools 6.2 C167UEN.tar contents are agentbuilder dm_upgrade itm5_upgrade tec Install guide indicates to cd tec... First action was to run ESync2000Aix.bin which installs to AIX /opt/IBM/SitForwarder All other OS's are contained within this download, under /TEC.

4. Install Wizard Page

5. Installation Wizard

6. License Agreement

7. Pathway to Install

8. Situpdate.conf

9. Persistence Directory

10. Situpdate.conf File fileLocation="/opt/IBM/SitForwarder/persistence" pollingInterval=3 crcBytecount=50 cmsSoapUrl=cms/soap bufferFlushRate=10 logLevel=verbose

11. TEM Hostname,UID,Passwd

12. situser.conf File serverid=sles10.tivlab.austin.ibm.com userid=root passwordfile="/opt/IBM/Netcool/omnibus/S itForwarder/etc/sles10.tivlab.austin.ibm.co m.pwd"

13. Install

14. Install Finish

15. Process Automation Setup

16. Stop/Restart PA

17. Udate Omnibus DB SQL

18. Cmds for updating SQL

19. Error regarding SQL Scripts

20. Write Permissions regarding SQL's

21. Eif Probe and Rules

22. tivoli_eif.rules ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/omnibus-> ls checkSUFStatus.sh itm.1.event itm_sync.sql tivoli_eif.rules controlsync.sh itm.2.event postz.conf writeevent.sh ereplay.pl itm_db_update.sql postz.minbari.conf errorevent.conf itm_proc.sql postzmsg

23. tivoli_eif.props Inactivity : 0 Server : "NCOMS" Manager : "Omnibus" Buffering : 0 BufferSize : 10 MessageLevel : "debug" MessageLog : '/opt/IBM/Netcool/omnibus/probes/eif/eif.log' Help : 0 Version : 0 StoreAndForward : 1 AutoSAF : 0 RawCapture : 0 Portnumber : 9999 RulesFile : '$OMNIHOME/probes/aix5/tivoli_eif.rules' EIFCacheFile : '$OMNIHOME/var/tivoli_eif.cache' #StreamCapture : true # End of file. MessageLog : 'stdout'

24. Setting Error Event Flow

25. Adding another TEMS server

26. Adding a TEMS server continued;

27. Start/Stop of SUF

28. TEMS Open Event

29. ITM Open Event in Omnibus

30. ITM “Ack'd” Event in Omnibus

31. Resynched Ack'd Event in TEMS

32. ITM synch logs ITM synch logs in /tmp ROOT@TORPEDO /tmp-> ls -al itm* -rw------- 1 root ncoadmin 0 11 Nov 16:08 itm_sync_error.cache -rwxr-xr-x 1 root ncoadmin 368 11 Nov 15:57 itmsyncerror.out -rw-r--r-- 1 root system 36000 19 Nov 15:37 itm_tec_event_sync_install.log itmsynch drwxr-xr-x 3 root system 256 03 Sep 13:54. drwxrwxrwt 32 bin bin 16384 02 Dec 17:00.. drwxr-xr-x 2 root system 256 02 Dec 13:02 logs ROOT@TORPEDO /tmp/itmsynch/logs-> ls synch_msg.log synch_trace2.log synch_trace4.log synch_trace1.log synch_trace3.log synch_trace.log

33. Debugging SUF set Loglevel to Verbose ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/etc-> vi situpdate.conf fileSize=50000 fileNumber=10 fileLocation="/opt/IBM/Netcool/omnibus/SitForwarder/persistence" pollingInterval=3 crcBytecount=50 cmsSoapUrl=cms/soap bufferFlushRate=10 logLevel=verbose

34. SitUser.conf ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/etc-> vi situser.conf serverid=itm-vm100.tivlab.austin.ibm.com userid=root passwordfile=/opt/IBM/Netcool/omnibus/SitForwarder/etc/itm-vm100.tivlab.austin.i bm.com.pwd serverid=sles10.tivlab.austin.ibm.com userid=root passwordfile="/opt/IBM/Netcool/omnibus/SitForwarder/etc/sles10.tivlab.austin.ibm.com.pwd"

35. Suf executables ROOT@TORPEDO /opt/IBM/Netcool/omnibus/SitForwarder/bin-> ls launch.sh sitconfigsvruser.log startSUF.sh nohup.out stopSUF.sh query_state.sh sitconf.sh sitconfig.log sitconfuser.sh test.sh sitconfig.log.lck situpdate.sh

36. /tmp/itmsynch/logs/synch_tracelog s synch_trace.log 2009.11.19 16:48:12.611-06:00 com.tivoli.candlenet.SituationUpdateForwarder formatEvent IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP SOAP string: root xxxxx logins sles10: LZ 2009.11.19 16:48:12.611-06:00 com.tivoli.candlenet.SituationUpdateForwarder getServerUrlConnection IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Entry, parm 1 = op=a,sn=logins,sh=sles10:LZ,sv=sles10.tivlab.austin.ibm.com:3661,et=,di= 2009.11.19 16:48:12.612-06:00 com.tivoli.candlenet.SOAPConnection sendRequest IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP SOAP URL is: https://sles10.tivlab.austin.ibm.com:3661///cms/soap/kshhsoap.htm 2009.11.19 16:48:12.612-06:00 com.tivoli.candlenet.SOAPConnection sendRequest IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Method name is: CT_Acknowledge 2009.11.19 16:48:12.663-06:00 com.tivoli.candlenet.SOAPConnection getResponseCode IBM Tivoli Monitoring TEC Synchronization torpedo.tivlab.austin.ibm.com IP Response code is: 200

37. Check NCO_PA.log if Event doesn't show up in /persistance ROOT@TORPEDO /opt/IBM/Netcool/omnibus/log-> cat NCO_PA.log We have two procedures running to check the SUF status, and one has the wrong syntax one is running "usr/local/app2/opt/IBM/SitForwarder/omnibus/ send_event_cmd /usr/local/app2/opt/IBM/SitForwarder Check_SUF_Status Check_SUF_Status" which looks right The other is not running "send_event_cmd” / usr/local/app2/opt/IBM/SitForwarder Check_SUF_Status Check_SUF_Status" which is wrong...the second won't work since it needs the full path notice omnibus is missing from it's pathway.

38. /opt/IBM/Netcool/omnibus/SitForw arder/persistence drwxr-xr-x 2 root system 256 19 Nov 16:51. drwxr-xr-x 13 root system 4096 04 Nov 18:13.. -rw-r--r-- 1 root ncoadmin 26 19 Nov 16:51 lastwritten -rw-r--r-- 1 root system 0 03 Dec 12:43 running -rw-r--r-- 1 root ncoadmin 1444 19 Nov 16:51 situpdate_1257380230

39. Check for the “insert” in NCOMS.log if event doesn't show up in /persistance Look for the insert statement proof of the ITM sql triggers firing insert into alerts.status values 08/25/09 11:00:21: Debug: D-OBJ-105-010: Client language command [probe][PROBE][tivoli_eif][pokdevlab75] [insert into alerts.status values ( 'MS_Offline:usrdtep01i9012b:UA::ITM_ManagedSystem',0,'usrdtep01i9012b:UA','usrd tep01i9012b:UA','EIF Probe on pokdevlab75','EIF Probe on pokdevlab75','ITM_ManagedSystem','MS_Offline',4,'MS_Offline',1251216021,125121 6021,1251216021,1251216021,0,20,0,6601,1,'',65534,0,0,0,'',0,0,0,'','',0,0,'',0,'',0,0,'',0,0,'','','','','','','','',0,0,'','','USPK0P0ALLD',0,'','','','','08/25/2009',0,'N','','','08/25/2009 12:00:21.000','usrdham01i901xa.raleigh.ibm.com','3661','U','','S','HUB_usrdham01i90 1xa','','','','','','','','MS_Offline','','','',0,'',0,'','','','usrdtep01i9012b:UA','','','','LAB','','','',0,'','',0, 0,1251216021,1251216021,0,'','','',0,0,1251216021,'',0,'',1251216021,0,0,0,'',0,125121 6021,0,0,1251216021,0,'','','',0,0,0,0,0,1251216021,0,'','','','',0) updating (ITMStatus,ITMTime,ITMHostname,ITMResetFlag); ].

40. Test Soap Connection 1) Stop SuF (2) Delete all SuF cache files and file called lastwritten (in $NCHOME/ITM/SitForwarder/persistence/ ) (3) Make sure you are logged in as the same user used to install SuF, typically this would be netcool (4) Restart SuF via StartSUF.sh (5) run test.sh to test SOAP connection to ITM (6) Acknowledge ITM events in OMNIbus and confirm they are modified in ITM

41. Iptrace of Soap Iptrace on Soap Connections: Start Iptrace on Omnibus server tracing soap ports 1920 and 3661 startsrc -s iptrace -a " -a -p 1920,3661 -b /tmp/trace.out"

42. WireShark output of Iptrace

43. Questions for Ralph or Calvin Email address: moody1@us.ibm.com Rbaldwin@us.ibm.com