1. TBSM v6.1 Troubleshooting Events
September, 2013
Joy Hawkins

2. Agenda Technote UserID’s (Roles & Groups) TBSM Event Environment
Event Results Walk-Through from GUI
Enable Event Logging
Results from event logging in trace.log
How to use event logging filters 2

3. Technote: (Events)
MustGather - for IBM support to help analyze and understand a TBSM event issue?

4. Technote
UserID’s (Roles & Groups)
Section Separator

5. UserID’s - Roles & User Groups
(admin1) - (or use ‘tbsmadm’ from install)
Access Service Availability and Service Administration Pages
(tbsmAdminUserRole) assigned by default to tbsmAdmins Group
(user1) – (or use ‘tbsmuser” from install)
Access Service Availability Page
(tbsmReadOnlyUserRole) assigned by default to tbsmReadOnly Group

6. (admin1) UserID’s access
UserID’s - Roles & User Groups
(admin1) UserID’s access

7. UserID’s - Roles & User Groups
(user1) access

8. UserID’s - Roles & User Groups
Netcool_Omnibus_Admin Group
[ncw_user, ncw_gauges_editor,ncw_admin, ncw_dashboard_editor, netcool_rw]

9. UserID’s - Roles & User Groups
to run tool
Netcool_Omnibus_User Group
[ncw_user, netcool_ro]

10. UserID’s - Roles & User Groups
ISQWrite – omnibus privileges to run tool

11. UserID’s - Roles & User Groups

12. UserID’s (Roles & Groups)
TBSM Event Environment
Section Separator

13. TBSM Environment: (Templates)

14. TBSM Environment: (Incoming Status Rule)

15. TBSM Environment: (Preview data)

16. TBSM Environment: (Identification fields tab)

17. TBSM Environment: (ObjectServer is not Data Source)

18. TBSM Environment: (Data Fetcher)

19. TBSM Environment: (Data Fetcher’s View Data)
Select
DATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE
from TBSMDEMO.TICKETS
where STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’

20. TBSM Environment: (Rule’ Preview Data)

21. Section Separator TBSM Event Environment
Event Results Walk-Through from GUI
Section Separator

22. TBSM Environment: (Service Instances & send test Event)

23. Event Results: (Service Administration Portlets)

24. Event Results: (AEL)

25. Event Results: (Event Details)

26. Section Separator Event Results Walk-Through from GUI
Enable Event Logging
Section Separator

27. Enabling Event Logging
Available from TBSM FP3, or TBSM 6.1 FP1, or TBSM GA.
Use setTraceLevel command to add the trace level for the EventLogging package.
UNIX: $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data –addtrace "EventLogging=finer"
Windows: %TBSM_HOME%\bin\setTraceLevel.bat <tipadmin user> <tipadmin password> data -addtrace “EventLogging=finer“

28. Enabling Event Logging
For best effect, set any other TBSM trace levels to “info”
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -addtrace "com.micromuse.sla.*=info:com.ibm.tbsm.*=info:EventLogging=finer”

29. Enabling Event Logging
Other setTraceLevel.sh options
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -display
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -setstartup
Event Logging should not be enabled permanently on customer sites – turn it off when not required.
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -resetall OR
$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -rmtrace "EventLogging=finer"

30. Section Separator Enable Event Logging
Results from event logging in trace.log
Section Separator

31. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Collect new trace logs
Run <installdir>/tbsm/bin/collect_logs
After running the utility on the TBSM Data server, the result will be a single zip file <installdir>/tbsm_logs_<datetimestamp>.zip
$TIP_HOME/TBSMProfile/logs/server1/ (trace.log, SystemOut.log, SystemErr.log)
6 – 31
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

32. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
With tracing set to EventLogging.
The first entry is an event receipt – shows all fields in the Event Container. TBSM will use the filters to match the event to a service.
[7/22/13 13:42:10:946 EDT] EventLogging 2 RADEventProcessor execute "Prepared by TBSMOMNIbusEventReader"=(EventContainer=(RAD_TimeWindowStart=0, AlertKey=mail, Location=, RAD_TimeWindowEnd=0, RemotePriObj=, LocalRootObj=, RAD_FunctionName=, NmosEntityId=0, RemoteNodeAlias=, X733EventType=0, ReceivedWhileImpactDown=0, RAD_UserFunctionName=NCOMS564, RAD_TotalRawMetric=, TaskList=0, ProbeSubSecondId=0, RAD_ServiceID=0, RAD_TimeWindowLength=0, RemoteRootObj=, Tally=1, KeyField=564, NodeAlias=, RAD_NextThresholdRawMetric=, RAD_ServiceTypeName=, NmosDomainName=, RAD_LastCount=1, RAD_CurrentFunctionState=, RAD_FilterIDList=2,3, Serial=564, X733CorrNotif=, … …
<also seen in same message, I can see…>
Identifier=steJoyTESTEvent110,
Node=Mail3PortC,
Class=0,
Summary=JoySendEvent3,
Severity=3,…),
<the message ends with the following…>
Policies={Main}, PolicyName=Main, ReportsPolicyName=Main, ActionNodeName={Main}, ActionType=1)
6 – 32
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

33. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
Three parts of an Incoming Status Rule:
(Discriminator, Filter Thresholds and Instance fields).
1. Discriminator (or Class) is used to set the RAD_FilterIDList on the event.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters No Filters found in Event Container so must get filters.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters Using discriminator: 0.
6 – 33
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

34. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
2. Threshold Filter. Each field in the Event Container is tested against the Filter fields, to check if they match the threshold.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing AlertKey with comparator = 'mail'.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.
[7/22/13 13:42:10:947 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch Testing Severity with comparator >= '3'.
[7/22/13 13:42:10:947 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter addFilterIDToEvent New RAD_FilterIDList for Event is to set to: 2.
[7/22/13 13:42:10:946 EDT] EventLogging 2 MemoryBasedMatchingRawAttributeGetter println testThresholdMatch Threshold matched and status is greater than 0.
6 – 34
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

35. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
3. Instance field. TBSM reads the value for the Event fields from the event, and then returns a list of Service Instances which match this.
[7/22/13 13:42:10:948 EDT] EventLogging 2 AbstractMatchingInstanceGetter getInstanceIDFieldValueMappingForRawAttribute Returning field expression and value for attribute: {Node=Mail3PortC}.
[7/22/13 13:42:10:948 EDT] EventLogging 2 MemoryBasedMatchingInstanceGetter getMappingStringForRawAttribute Returning event string for raw attribute: Mail3PortC.
[7/22/13 13:42:10:948 EDT] EventLogging 2 GlobalInstanceFieldStore getCommaSeparatedInstanceIDs TagId is 85. attributeID is 1. eventMatchingString is Mail3PortC. List of Service Instance Ids which match raw attribute is: 89.
[7/22/13 13:42:10:948 EDT] EventLogging 2 MemoryBasedMatchingInstanceGetter getMatchingInstanceIDsFromDataStore Returning matching instances: [89]. 2 1 3 4
6 – 35
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

36. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
Update State in TBSM
[7/22/13 13:42:10:948 EDT] EventLogging 2 RADEventProcessor sendInstancesToStateModel1 Send to statemodel for ServiceInstanceName Mail3PortC Service instanceID: 89.
[7/22/13 13:42:10:948 EDT] EventLogging 2 StateModelUpdater passToStateModel Status is 3.0.
6 – 36
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

37. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
Update Object Server service_deps table
[7/22/13 13:42:10:948 EDT] EventLogging 2 StateModelNodeAttributeState updateServiceDeps About to update service_deps in memory and at object server for SIB: Mail3PortC.
[7/22/13 13:42:10:948 EDT] EventLogging 2 ObjectServerUpdater updateServiceDeps updateEventIDs for Mail3PortC and eventInstanceID NCOMS:_:564.
6 – 37
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

38. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
Update Service Name in Event
[7/22/13 13:42:10:948 EDT] EventLogging 2 StateModelUpdater setVarsInContexts RAD_ServiceName is: Mail3PortC being set in Event Container.
6 – 38
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

39. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Trace log Entries
Propagate up parents
[7/22/13 11:41:22:151 EDT] EventLogging 2 StateModelNodeAttributeState stateChanged About to update parents for current SIB.
[7/22/13 11:41:22:151 EDT] EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: Mail3PortC.
[7/22/13 11:41:22:153 EDT] EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: MailServer3.
[7/22/13 11:41:22:156 EDT] EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent: App-Root.
6 – 39
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

40. Section Separator Results from event logging in trace.log
How to use event logging filters
Section Separator

41. Event Logging with Filters
IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filters reduce the number of events which are eligible for Event Logging
Filters are dynamic
Filters are persistent
Steps:
First eventloggingfilter Table must created
Then use radshell commands to set up filters for Event Logging
6 – 41
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

42. Event Logging with Filters
IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
For TBSM create the postgres table eventloggingfilter
Created by running the SQL file: createEventLoggingFilterTable.sql (This is shipped with TBSM FP3)
UNIX:
cat ./createEventLoggingFilterTable.sql | $TBSM_HOME/bin/rad_db log
Windows:
type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log
6 – 42
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

43. Event Logging with Filters
IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
For TBSM 6.1.X - create the DB2 table eventloggingfilter
From a command line start db2
db2
connect to TBSM (to connect to DB)
6 – 43
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

44. Event Logging with Filters
IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
For TBSM 6.1.X - create the DB2 table eventloggingfilter
Create Table
create table TBSMBASE.eventloggingfilter (
filterfieldname varchar(255),
filterfieldvalue varchar(1024) )
6 – 44
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

45. Event Logging with Filters
IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Use radshell commands to set up filters for Event Logging
Displays the current filter setting
Clear the filter
Adds fields and values to the current filter
Removes a field from the filter
$TBSM_HOME/bin/rad_radshell
displayEventLoggingFilter();
clearEventLoggingFilter();
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
removeEventLoggingFilterValue(fieldName);
6 – 45
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

46. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filter examples: add a field to current filter:
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
Only events with Node = “Mail3PortC” are logged, when the trace level for EventLogging package is finer
addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);
6 – 46
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

47. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filter examples: add a field to current filter:
addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);
To change the filter, so that only events with Node = “Mail3PortC”,AND AlertKey = “mail”, are logged, when the trace level for EventLogging package is finer
addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);
addUpdateEventLoggingFilterValue(“AlertKey”,”mail”);
6 – 47
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

48. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filter examples:
6 – 48
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

49. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filter examples: remove a field from current filter:
removeEventLoggingFilterValue(fieldName);
To remove the Node fields from the filter (now filter will just match on AlertKey)
removeEventLoggingFilterValue(“Node”);
6 – 49
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

50. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters
Filter examples: remove a field from current filter:
clearEventLoggingFilter();
Remove ALL filter fields – Logging will be done for all events if the trace level for EventLogging package is finer.
6 – 50
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

51. IBM Tivoli Netcool/Real time Active Dashboards 3.0 Fundamentals
Event Logging with Filters- Turn off afterwards
The filter set up by the addUpdateEventLoggingFilterValue radshell command, is only used when the trace level for the EventLogging package is set to finer.
If EventLogging package is not set, the radshell filter has NO effect.
6 – 51
©Copyright IBM Corp Course materials may not be reproduced in whole or in part without the prior written permission of IBM.