1. Fault Management – Detection and Diagnosis

2. Outline  Fault management functionality  Event correlations concept  Techniques

3. Definitions  A fault may cause hundreds of alarms.  We need to be able to do the following: oDetect the existence of faults oLocate faults  An alarm oExternal manifestations of faults —Generated by components —Observable, e.g. via messages  An alarm represents a symptom of a fault.  An event oAn occurrence of interest, e.g. an alarm message

4. Fault Management Functionalities  Fault detection oShould be real-time oTechniques can be based on active schemes (e.g., polling) or event-based schemes (where a system component says that it has detected a failure).  Fault location oIs it a link or system component or application component?  Determine corrective actions  Carry out corrective actions and determine effectiveness

5. Alarm (Event) Correlation  Alarm explosion oA single problem might trigger multiple symptoms (e.g., router is down)  There could be too many alarms for an administrator to handle; Techniques used to help: oCompression: reduction of multiple occurrences of an alarm into a single alarm oCount: replacement of a number of occurrences of alarms with a new alarm oSuppression: inhibiting a low-priority alarm in the presence of a higher priority alarm oBoolean: substitution of a set of alarms satisfying a condition with a new alarm oRoot cause determination

6. Faults and Alarms

7.  The previous figure shows that correlation c1 detects the fault f1 and that correlation c2 detects the fault f2.  Correlating c1 and c2 into the correlation c0 allows the diagnosis of the fault f0.

8. Example  Let a1, a2, a3, a4, a5 be alarms generated by client processes indicating that a client process is not getting a response from a server.  Correlation techniques can be used to show that since a1, a2, a3 were generated by client processes by trying to contact the same server then the server may be the problem. Similar comments apply to a4 and a5.

9. Example  From the perspective of client processes, the servers (at the second level of the previous figure) are at fault.  However, it may be observed that alarms were generated by these two servers. Both alarms indicate that each of the two servers are not getting a response and that both were trying to contact the same server. This is another correlation.

10. Fault Diagnosis  Major application of alarm correlation (often called event correlation) is fault diagnosis  Useful in fault location

11. Rule-Based Reasoning  Based on expert systems  Intended to represent heuristic knowledge as rules.  Components oKnowledge Base (KB): Contains the expert rules that describe the action to be taken when a specific condition occurs e.g., if-then-else oWorking Memory(WM): Stores information such as the system/network topology and data collected through the monitoring of application and network components.

12. Rule-Based Reasoning  Components (continued) oInference engine: matches the current state (as represented by the monitored data) of the system against the left-side of a rule in the knowledge base in order to trigger the action.  The rules are meant to encapsulate expert knowledge  Why rule-based reasoning? oRules are interpreted which means that rules can be changed without recompiling. oSince expert knowledge can be wrong and/or complete, this feature is very useful.

13. Rule-Based Reasoning  Operation oThe WM constantly scanned for facts (e.g., alarms) that can satisfy any of the left hand sides of the rules. oIf a rule is found then the rule “fires” I.e., the right hand side is executed. oThe result of the execution may result in facts being inserted into WM.  Example: oFailed-connection (Y,X) and Failed-connection(X,Z)  faulty(Z).  Used by commercial systems such as Tivoli (from IBM) and HP Openview.

14. Approaches  Fault propagation  Model traversing  Case-based reasoning

15. Fault Propagation  Based on models that describe which symptoms will be observed if a specific fault occurs.  Monitors typically collect managed data at network elements and detect out of tolerance conditions, generating appropriate alarms.  An event model is used by a management application to analyze these alarms.  The event model represents knowledge of events and their causal relationships.

16. Fault Propogation (Coding Approach)  Correlation is concerned with analysis of causal relations among events.  The notation e  f is to denote causality of the event f by the event e.  Causality is a partial order between events.  The relation  may be described by a causality graph whose directed edges represent causality.  Distinguish between faults problems) and symptoms.  Nodes of a causality graph may be marked as problems (P) or symptoms (S).  Some symptoms are not directly caused by faults, but rather by other symptoms.

17. Fault Propagation (Coding Approach) 7 6 1 8 9 11 3 5 4 10 2 Example Causality Graph

18. Fault Propagation (Coding Approach)  The correlation problem oA correlation p  s means that problem p can cause a chain of events leading to the symptom s. oThis can be represented by a graph.

19. Fault Propagation (Coding Approach) 1 9 11 10 2 6 A Correlation Graph

20. Fault Propagation (Coding Approach)  For each fault (problem) p, the correlation graphs provides a vector that summarizes information available about correlation and symptoms and problems.  This is referred to as the code of the problem.  Alarms may also be described using a vector assigning measures of 1 and 0 to observed and unobserved symptoms.  The alarm correlation problem is that of finding problems whose codes optimally match an observed alarm vector.

21. Fault Propagation (Coding Approach)  Example codes (look at correlation graph example) o1 = (0,1,1) – This indicates that problem 1 causes symptoms 9 and 10 o2 = (1,0,1) – This indicates that problem 2 causes symptoms 6 and 10 o11 = (0,1,1) – This indicates that problem 11 causes symptoms 9 and 10.

22. Fault Propagation (Coding Approach)  Example alarm vector oAssume that alarms indicating symptoms 9 and 10 have been observed. oa = (0,1,1)  We can infer that either 1 or 11 match the observation a.  These two problems have identical codes and hence are indistinguishable.  The fault management application may have to do additional tests.

23. Fault Propagation (Coding Approach)  A Codebook is an array of the vectors just defined.  The number of symptoms associated with a single problem may be very large. oSometimes a much smaller set of symptoms is selected to accomplish a desired level of distinction among problems.

24. Fault Propagation (Coding Approach) Example Codebook p1p2p3p4p5p6 1 100101 2 111100 4 101010

25. Fault Propagation (Coding Approach) Example Codebook p1 p2 p3 p4 p5 p6 1 1 0 0 1 0 1 3 1 1 0 1 0 0 4 1 0 1 0 1 0 6 1 1 1 0 0 1 9 0 1 0 0 1 1 18 0 1 1 1 0 0

26. Fault Propagation (Coding Approach)  Distinction among problems is measured by the Hamming Distance between their codes  The radius of a codebook is one half of the minimal Hamming distance among codes.  When the radius is 0.5, the code provides distinction between problems.

27. Fault Propagation (Coding Approach)  Is this easy to apply to application processes? oNo  Why oApplications are dynamic oThe coding approach assumes the system is fairly static.

28. Model Traversing  Reconstruct fault propagation at run time using relationships between objects  Begins with managed object that generated event  Work best when object relationship is graph-like and easy to obtain since it must be obtained at run-time oPerformance oPotential parallelism  Weaknesses oLack of flexibility oNot well-structured like fault propagation

29. Model Traversing  Characteristics oEvent-Driven: Fault management application is passive until an event arrives. This event is the reporting of a symptom. oCorrelation : Decides whether two events result from the same primary fault. oRelationship Exploration: The fault management application correlates events by detecting special relationships between the source objects of those events.

30. Model Traversing  Event reports should have the following information: oSymptom type oSource oTarget oetc  If symptom s i ’s target is the same as s j ’s source then this is an indication that s i is a secondary symptom. This allows us to ignore certain alarms.

31. Model Traversing  For each event, construct a graph of objects (models) related to the source object of that event.  When two such graphs touch each other, i.e. contain at least one common object, the events which initiated their construction are regarded to be correlated. Possibly these two events are the result of the same fault.  If s i is correlated with s j and s j is correlated with s k then through transitivity we can conclude that s i is a secondary symptom.

32. Model Traversing  The process of eliminating symptom reports may result in reports that have the same target.  Example: os 1 and t os 2 and t  It might be necessary to construct possible paths of objects between s 1 and t as well as s 2 and t  Nodes in common are good candidates for the faults.

33. Model Traversing  We will now discuss the building of graphs  The algorithm for building graphs uses relationships between network hardware and software components to search for the root cause of a problem.  Assumes that information about the relationships between the components are available (e.g., through a database).  Assumes that there are functions including these: ogetNextHop(source, target,B): Get the node representing the next entity (that comes after B) in the path between source and target. Note that this may return more than one entity.

34. Model Traversing Example  Assume the following configuration of processes and machines. All machines are connected through the Ethernet. oP1 is on chocolate; P2 is on peppermint oP3 is on vanilla; P4 is on strawberry oP5 is on doublefudge; P6 is on mintchip  Communication is through remote procedure calls. This basically requires that all communication go through a daemon process on the server host’s machine. We will call this rpcd

35. Model Traversing P4 P3 P5 P1 P2 P6 Call structure is depicted in the following graph:

36. Model Traversing Example  Assume that P4 terminates abnormally causing a cascade of timeouts  Correlation will result on focusing on these event reports: o(P1,P4) o(P3,P4)  Not enough to diagnose the fault. oIt’s all at the process level. oThere are still many entities or objects to examine since you do not want everything generating a message.

37. Model Traversing  Example  Starting with P1 the next component (node) along the path of the connection between P1 and P4 is identified.  Between P1 and P4 are many entities. We will start out with a vertical search which basically results in the fact that P1 is running on a host machine called chocolate

38. Model Traversing  chocolate is connected to the hub through an ethernet cable.  The hub is connected to strawberry through an ethernet connection cable where P2 is running. Thus we can say that the path is the following: oP1, chocolate,ethernet connection cable,hub,strawberry,ethernet connection cable, rpcd.strawberry,P4  The path between P3 and P4 is the following: oP3, vanilla, ethernet connection cable, hub, ethernet connection cable, strawberry, rpcd.strawberry, P4

39. Model Traversing Example  This suggests that we can narrow down the problem to hub, ethernet connection cable, strawberry.rpcd, strawberry, P 4.  At this point, the fault management application may want to poll for additional information. The polling may check to see if something is up or not. An example is applying the ping operation to the host machine called strawberry.  What if every entity is up? This may indicate that strawberry is overloaded. An indication of an overload can be found by measuring the CPU load.

40. Model Traversing  Building the graphs requires structural information and the use of rules.

41. Model Traversing Implementation  What management services are needed? oTo detect and report symptoms, one could use application instrumentation. oThe instrumentation library should most likely talk with a management process (or agent). oThe agent sends an event report to the event server. oThe event server may have a set of rules for symptom correlation. oAfter correlation, a task may be invoked that does relationship exploration and the final diagnosis.

42. Model Traversing Implementation  Information Needed oInformation representing the relationships between hardware components and software components is needed. oThis needs to be stored in a database or a directory service (e.g., X500) oAn API needs to be defined to retrieve this information. oRules can be used to help construct the graph.

43. Model Traversing Implementation  Information Needed oHow is the information collected? oMany different techniques. Examples include: —Processes (using instrumentation) may have to register and have their information put into the database. —Network information may have to be entered manually.

44. Model Traversing  Summary  Performs very quickly once model is built oModel can be constructed incrementally during normal processing; do not have to wait until failure  Can operate in parallel  Can accommodate multiple events; different starting points can result in same problem element  Does require model reflective of run-time oOne that changes too fast is a problem

45. Case-based Reasoning (CBR)  Objective oLearn from experience oSolutions to novel problems oAvoid extensive maintenance  Basic idea: recall, adapt and execute episodes of former problem-solving in an attempt to deal with a current problem

46. Case-based Reasoning Approach

47. Case-Based Reasoning  Strategy  Useful for domains in which a body of knowledge with a case structure exists or is easily obtainable  Case structure: oSet of fields or “slots” oCapture “essential” information  Yield discriminators oSet of fields highly correlated with problems or solutions  Need to find “closest” match

48. Case-Based Reasoning Adapt

49. Case-based Reasoning  Summary  Needs well-defined cases  Likely to work well when problems are “close” to existing solutions  Problem selecting solutions when “not so close” oDangerous in following actions? oHow to adapt?

50. Summary  Variety of approaches  Mostly applied in network management scenarios oMore controlled? oBetter understanding of problems?  Limited experience in application management