1. EEC 688/788 Secure and Dependable Computing Lecture 12 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University wenbing@ieee.org

2. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Outline Reminder: Project outline due today! Fault tolerance and replication Event ordering

3. Midterm#2 High: 100, low: 60, mean: 84 Q1: 43/50, Q2: 10/10, Q3: 18/20, Q4: 13/20

4. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Redundancy To tolerant fault, some form of redundancy must be used  Replication in time (transaction processing)  Replication in space  Redundancy in software design (n-version programming) The three types of replication techniques are complimentary to each other

5. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replication in Time Replication in time  From the present state, apply one or more operations  If the system fails before completion, abort and rollback to the original state  Start all over again Example: transaction processing  Essence: roll-backward recovery

6. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replication in Space Replication in space:  Run multiple instances (replicas) of the systems so that if one fails, one or more replicas can take over  Must assume independent faults  Must ensure consistency among the replicas

7. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao N-version Programming N-version programming (redundancy in software design)  The system (or component) has n-different designs and implementations  In case of permanent software bugs, a different version is used for each replica, or for repeated executions

8. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replication is not a Trivial Task Suppose we want to replicate a server using the most popular (an inexpensive) approach  We run two servers on separate computers  The primary sends a log (its state, and/or logged incoming messages) to the backup  If primary crashes, the backup soon catches up and can take over

9. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Split Brain Syndrome primary backup Clients initially connected to primary, which keeps the backup up to date. Backup collects the log and updates its state log

10. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Split Brain Syndrome Transient problem causes some links to break but not all. Backup thinks it is now primary, primary thinks backup is down primary backup

11. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Split Brain Syndrome Some clients still connected to primary, but one has switched to backup and one is completely disconnected from both primary backup

12. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replica Consistency Replicas must be coordinated appropriately so that we can achieve strong replica consistency:  At the end of each execution step, all replicas must have the same state  The outputs from every replica must be consistent all the time

13. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replica Consistency Techniques to achieve strong replica consistency  As a prerequisite, we need to have an agreement on the membership of the replica group  Ensure that the same set of inputs is delivered to all replicas and the inputs must be in the same order  Ensure deterministic execution due to each input at every replica (applicable to active and semi-active replication)

14. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao One Copy Image When a component or system is replicated, we must ensure that the replicated unit appears as a single copy to external components/systems that interact with it Need a gateway in between the replicated and non- replicated units Alternatively, the gateway module can be integrated with the non-replicated unit Main tasks of the gateway  Multicast requests/replies to the replicas  Detect and suppress duplicated replies/requests

15. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Replication Styles Active replication  Every input (request) is executed by every replica  Every replica generates the outputs (replies)  Voting is needed to cope with non-fail-stop faults Passive replication  One of the replicas is designated as the primary replica  Only the primary replica executes requests  The state of the primary replica is transferred to the backups periodically or after every request processing Semi-active replication  One of the replicas is designated as the leader (or primary)  The leader determines the order of execution  Every input is executed by every replica per the leader’s instruction

16. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Duplicate Invocation Suppressed Duplicate Responses Suppressed Active Replication Actively Replicated Client Object A Actively Replicated Server Object B RM

17. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Active Replication with Voting Question: to cope with f number of faults (non-malicious), how many replicas are needed?

18. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao State Transfer State Response Invocation Passive Replication Passively Replicated Client Object A Passively Replicated Server Object B Primary Replica Primary Replica RM Question: can passive replication tolerate non-fail-stop faults?

19. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Ordering info Response Invocation Semi-Active Replication Semi-Actively Replicated Client Object A Semi-Actively Replicated Server Object B Primary Replica Primary Replica RM

20. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Ensuring Strong Replica Consistency Possible strategies  For active replication, use a group communication system that guarantees total ordering of all messages (plus deterministic processing in each replica)  Passive replication with systematic checkpointing  Semi-active replication  Use two-phase commit

21. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Total Ordering of Messages What is total ordering of messages?  All replicas receive the same set of messages in the same order  Atomic multicast – If a message is delivered to one replica, it is also delivered to all non-faulty replicas With replication, we need to ensure total ordering of messages sent by a group of replicas to another group of replicas  FIFO ordering between one sender and a group is not sufficient m1 m2 m1 m2 m1

22. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Potential Sources of Non-determinisms Multithreading  The order of accesses of shared data by different threads might not be the same at different replicas System calls/library calls  A call at one replica might succeed while the same call might fail at another replica. E.g., memory allocation, file access Host/process specific information  Host name, process id, etc.  Local clocks - gettimeofday() Interrupts  Delivered and handled asynchronously – big problem

23. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Event Ordering “Time, Clocks, and the Ordering of Events in a Distributed System”, by Leslie Lamport, Communications of the ACM, July 1978, Volume 21, Number 7, pp.558-565  What usually matters is not that all processes agree on exactly what time it is, but rather, that they agree on the order in which events occur

24. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Happens-Before Relation Assumptions:  The system is composed of a collection of processes, each process consists of a sequence of events  The events of a process form a sequence, where a occurs before b in this sequence if a happens before b  The sending or receiving of a message is an event in a process

25. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Happens-Before Relation The happens-before relation “→” on the set of events of a system is the relation satisfying the following three conditions:  If a and b are events in the same process, and a comes before b, then a → b  If a is the sending of a message by one process and b is the receipt of the same message by another process, then a → b  If a → b and b → c, then a → c

26. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Partial Ordering Not all events have the happens-before relationship Two distinct events a and b are said to be concurrent if a → b and b → a  Neither event can causally affect the other  This introduces a partial ordering of events in a system with concurrently operating processes “a happens before b” means that information can flow from a to b “a is concurrent with b” means that there is no information flow between a and b

27. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao How to Capture the Partial Ordering? Use logical clocks to capture the partial ordering  Define a clock C i for each process P i. Assign a number C i (a) to any event a in that process  The entire system of clocks is represented by the function C which assigns to any event b the number C(b), where C(b) =C j (b) if b is an event in process P j  The clocks C i are logical clocks rather than physical clocks

28. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Lamport Clock A Lamport logical clock is a monotonically increasing software counter Each process P i keeps its own logical clock C i to apply Lamport timestamps to events To capture the happens-before relation →, processes must do the following:  Before each event at P i : C i := C i +1  When P i sends a message m, it piggybacks t = C i  When P j receives (m,t): C j := max(C j,t) + 1 e → e’  C(e) < C(e’)

29. 6/16/2015 EEC688/788: Secure & Dependable Computing Wenbing Zhao Lamport Clock: An Example