1. EEC 688/788 Secure and Dependable Computing
Lecture 6
Wenbing Zhao
Department of Electrical and Computer Engineering
Cleveland State University

2. EEC688/788: Secure & Dependable Computing
Outline
Recovery oriented computing
Overview
Application level fault detection
Structural behavior monitoring
Path shape analysis
Microreboot and System-Level Undo/Redo
12/2/2018
EEC688/788: Secure & Dependable Computing

3. Recovery-Oriented Computing
On availability of soft realtime systems
Availability = MTTF/(MTTF+MTTR)
MTTF: mean time to failure
MTTR: mean time to recover
Availability can be improved by increasing MTTF as well as reducing MTTR
Recovery-oriented computing: focusing on reducing MTTR
Making fault detection faster and more accurate
Making recovery faster

4. Fault Detection and Localization
Fault detection: determine if some component in the system has failed
Fault localization: pinpoint the particular component that failed
Low-level fault detection mechanism
Based on timeout, probing each component periodically with a heartbeat message
Cannot detect many application-level faults
Recovery-oriented computing: focusing on application-level fault detection and localization
75% of the recovery time is spent on application-level fault detection

5. Microreboot and System-Level Undo/Redo
Microreboot: many problems can be fixed by simply restarting the faulty component
Works best with component-based systems
For problems cannot be fixed by microreboot, performs system-level undo, fixed the problem, then carries out system-level redo
Based on checkpointing and logging

6. System Model for Recovery-Oriented Computing
Three-tier architecture
Separating application logic and data management
Middle-tier is stateless or maintains only session state
Component-based middleware
Java Platform, Enterprise Edition (Java EE often referred to as J2EE)
Key component: Enterprise Java Bean (EJB)

7. Application-Level Fault Detection
Fail-stop faults can be detected using timeouts
Application-level faults can only be detected in the application level
One plausible fault detection method: acceptance test
Developer would have to develop effective and efficient acceptance test routings
Not practical for Internet apps due to their scale, complexity and rapid rate of changes
ROC-based approach: measure and monitor structural behaviors of an app
May detect app-level faults without a priori knowledge of the app details

8. Structural Behavior Monitoring
Interaction patterns between different components reflect the app-level functionality
Each component implements a specific app function, e.g.,
Stateful session bean to manage a user’s shopping cart
A set of singleton session beans to keep track of inventory
The internal structural behavior can be monitored to infer whether or not the app is functioning normally
To monitor
Log runtime path for each end-user request, including all incoming msgs, outgoing msgs, method invocations, etc.

9. Structural Behavior: Runtime Path Example
Runtime path for a single end-user request
Span 5 components
Consist of 10 events

10. Structural Behavior: Machine Learning
Train reference models using machine learning
Historical reference model: training with aggregated runtime path data
Objective: anomaly detection based on historical behavior
May use real workload as well as synthetic workload that resembles real workload
Peer reference model: train with most recent runtime path data
Objective: anomaly detection with respect to peer components
Must train with real workload
Fault (anomaly) detection: comparing observed patterns with those in the reference models

11. Component Interactions Modeling
Focus on interactions between a component instance and all other component classes
More scalable: can cope with cases when there are many instances of each class
Suitable for using the Chi-square test for anomaly detection

12. Component Interactions Modeling
Given a system with n component classes, the interaction model for a component instance consists of a set of n-1 weighted links between the instance and all the other n-1 component classes
We assume instances of the same class do not interact with each other
We assume that interactions are symmetric (i.e., request and reply)
Weight assigned to each link is the probability of the component instance interacting with the linked component class
The sum of the weight on all links is 1, i.e., the component instance has probability of 1 to interact with other component classes

13. Component Interaction Model: Example
Class A: web component, handles end-user requests
Class B: app logic, handles conversations with end-users, 3 instances
Class C and Class D: also app logic, representing shared state
Class E: database server, persistent state

14. Component Interaction Model: Example
Machine learning: determine link weight based on training data
Training data
A issued 400 remote invocations on b1
b1 issued 300 local method invocations on C, and 300 invocations on D
Not important what happened between C & E, D & E
Link weight calculation
Total number interactions occurred at b1 instance: 1000
P(b1-A) = 400/1000 = 0.4
P(b1-C) = 300/1000 = 0.3
P(b1-D) = 300/1000 = 0.3

15. Anomaly Detection
Comparison of current behavior with the trained behavior: use Chi-Square test
Prepare the observed data as a histogram
Compare distribution using formula:
n: number of cells in the histogram
ei: expected frequency in cell i
oi: observed frequency in cell i
If ei is 0, the cell should be pruned off
Each link is regarded as a cell
For observation period of m requests, expected frequency for link i: ei = m * pi
No anomaly: D = 0 ideally. In practice, D is not 0 due to randomness, it follows a chi-square distribution

16. Anomaly Detection: Chi-Square Test
Anomaly detected: D > the 1-a quantile of the chi-square distribution with degree of freedom of k=n-1 at a level of significance a
Higher level of a => more sensitive => more false positive
High level of significance => higher probability to reject that there is no relationship => higher probability to confirm that there is a relationship => higher probability of detecting abnormality
Level of significance: the probability of rejecting the null hypothesis in a statistical test when it is true
The null hypothesis refers to a general statement or default position that there is no relationship between two measured phenomena. Rejecting or disproving the null hypothesis—and thus concluding that there are grounds for believing that there is a relationship between two phenomena

17. Anomaly Detection: Chi-Square Test: Example
Observation period: 100 requests
A issued 45 requests on b1
b1 issued 35 invocations on C, and 20 invocations on D
Link(A-b1): expected value is 100*0.4=40, observed 45
Link(C-b1): expected: 100*0.3=30, observed 35
Link(D-b1): expected: 100*0.3, observed 20
D=(45-40)2/40 + (35-30)2/30+(20-30)2/30 = 4.79
Chi-square test: degree of freedom is 2 (only 3 cells), for a=0.1, 90% quantile is 4.6 => anomaly detected

18. Path Shapes Modeling
The shape of a runtime path is defined to be the ordered set of component classes
A path shape is represented as a tree in which a node represents a component class
The directional edge represents the causal relationship between two adjacent nodes

19. Path Shapes Modeling
The probabilistic context-free grammar (PCFG) is used for path shape modeling (in Chomsky Normal Form, CNF)
A list of terminal symbols, Tk,
component classes in a path shape form Tk
A list of nonterminal symbols, Ni
Denote the stages of the production rules
N1: start symbol, often denoted as S
$: the end of a rule
All other nonterminal symbols are to be replaced by production rules (see below)
A list of production rules, Ni -> zj (a list of terminals and nonterminals)
A list of probabilities Rij = P(Ni -> zj )

20. Path Shape Modeling: Example
Path shape for 4 end-user requests
100% probability for the call to transit from A to B
R1j: SA, p=1.0
R2j: AB, p=1.0

21. Path Shape Modeling: Example
For B, 3 possible transitions: to C with 25%, to D with 25%, and to both C&D with 50 probability
R3j: BC, p=0.25 | BD, p=0.25 | BCD, p=0.5
Once a call reaches C or D, it must transit to E, hence:
R4j: CE, p=1.0
R5j: DE, p=1.0
E is the last stop for all
R5j: E$, p=1.0

22. Path Shape Modeling: Anomaly Detection
The path shape of new requests can be judged to see if they confirm to the grammar
An anomaly is detected if a path shape does not conform to the grammar
PCFG itself only detect fault, but not pinpoint root cause (localization of fault)
Need to use other method, such as decision tree

23. Microreboot
Microreboot: many problems can be fixed by simply restarting the faulty component
Works best with component-based systems
System design guideline
Component based: such as Java EE, with EJB
Separating application logic execution and state management
Reboot should be cause state loss
Loose coupling: to enable localized microreboot
Reduce dependency among components: either self-contained, or interaction with other components should be mediated (e.g., via Java EE container)
Key: any instance of the referenced component should be able to get the job done => when one under gone microreboot, another instance can provide same service
Resilient inter-component interactions
Lease-based resource management

24. Microreboot Automatic recovery with microreboot
Equipping with a fault monitor and a recovery management
The fault monitor implements some of the fault detection and localization algorithms described in the previously
The recovery manager is responsible to recover the system from the fault recursively:
by microrebooting first the identified faulty component,
if the symptom does not disappear, a group of components according to a fault-dependency graph.
If microrebooting does not work, the entire system is rebooted.
The final resort is to notify a human operator

25. Microreboot
Fault-depency graph (f-map): consists of components as nodes and the fault-propagation paths as edges
Equipping with a fault monitor and a recovery management
Can be obtained using automatic failure-path inference (AFPI)
AFPI
Constructed by observing the system’s behaviors when faults are injected
F-map is then refined during normal operation
Cycles in the f-map: nodes in the cycle are grouped as a single node; entire group will be microrebooted as a single unit; f-map => r-map

26. Microreboot Automatic recovery with microreboot
Reboot both reported faulty component and all components that are immediately downstream from the component
If faulty symptom persists, the upstream component in the r-map is also microrebooted
Recovery is carried out recursively until entire system is rebooted

27. Microreboot Implications of microreboot
Microreboot faulty components before node-level failure
Tolerating more false positives
Proactive microreboot for software rejuvenation
Enhance fault transparency for end-users

28. Overcoming Operator Errors
System dependability is significantly reduced because of human errors
Checkpointing and logging useful but not sufficient
Operating system level
State repair and selective replay
System-level undo (rewind), repair, system-level redo (replay)

29. Exercise 1. Identify the set of most recent checkpoints that can be used to recover the system shown here after the crash of P1
12/2/2018
12/2/2018
EEC693: Secure and Dependable Computing
EEC688: Secure & Dependable Computing
Wenbing Zhao

30. EEC688: Secure & Dependable Computing
Exercise 2.Chandy and Lamport distributed snapshot protocol is used to produce a consistent global state of the system shown below. Draw all control msgs sent in the CL protocol, the checkpoints taken at P1 and P2, and specify the channel state for the P0 to/from P1 channels, the P1 to/from P2 channels, and P2 to/from P0 channels
Software control will be elaborated in more details in the next slide
12/2/2018
EEC688: Secure & Dependable Computing
Wenbing Zhao 30

31. Exercise 3: Prove that the Chandy and Lamport Distributed Snapshot Protocol produces consistent checkpoints of the system.

32. Exercise 4: The following are the interactions that occurred in a system at instance b1 during a period, the total invocations on b1 at an instance are The remote invocation on b1 by A, the local method invocation by C, D, E and F are 300,200,300,200 and 200. If remote invocations on A by b1, the local method invocations on C, D, E and F observed are 35, 25,20,15, and 25 then find if anomalies are present in the system?