1. Fault Tolerance Chapter 7

2. Goal An important goal in distributed systems design is to construct the system in such a way that it can automatically recover from partial failures without seriously affecting the overall performance.

3. Basic Concepts Dependability Includes Availability A system is ready to be used immediately. Reliability A system can run continuously without failure. Safety Temporary failure does not cause troubles. Maintainability A failed system can be easily repaired.

4. Example Assuming: System A goes down for one millisecond every hour. System B never crashes but is shutdown for two weeks every August. Availability vs. Reliability?

5. Fault Fault: The cause of an error. Transient Intermittent Permanent Fault tolerance: a system can provide its services even in the present of faults.

6. Failure Models Different types of failures. Type of failureDescription Crash failureA server halts, but is working correctly until it halts Omission failure Receive omission Send omission A server fails to respond to incoming requests A server fails to receive incoming messages A server fails to send messages Timing failureA server's response lies outside the specified time interval Response failure Value failure State transition failure The server's response is incorrect The value of the response is wrong The server deviates from the correct flow of control Arbitrary failureA server may produce arbitrary responses at arbitrary times

7. Redundancy If a system is to be fault tolerant, the best it can do is to try to hide the occurrence of failures from other processes. Redundancy –Information redundancy –Time redundancy –Physical redundancy

8. Failure Masking by Redundancy Triple modular redundancy. Why three voters in each step?

9. 7.2 Process Resilience Process Groups –Organize several identical processes into a group –If one process fails, hopefully some other process can take over for it.

10. Flat Groups versus Hierarchical Groups a)Communication in a flat group. b)Communication in a simple hierarchical group

11. Failure Masking and Replication k fault tolerant system –It can survive faults in k components and still meet its specifications –Example 1: a group of k+1 components is enough to provide k fault tolerance. k of them simply stop, then the answer from the other one can be used. –Example 2: a minimum of 2k+1 processes are needed to achieve k fault tolerance. K failing processes accidentally generate the same reply, we need the remaining k+1 to beat them.

12. Agreement in Faulty Systems The general goal of distributed agreement algorithms is to have all the non-faulty processes reach consensus on some issue, and to establish that consensus within a finite number of steps. Is communication reliable? Are processes honest?

13. Two-army Problem (1) Let’s attack C at 8am (2) OK, let’s do it. (3) I got your message (2) (4) I got your message (3) …… 3000 5000

14. Byzantine Generals Problem

15. Agreement in Faulty Systems (1) The Byzantine generals problem for 3 loyal generals and1 traitor. a)The generals announce their troop strengths (in units of 1 kilosoldiers). b)The vectors that each general assembles based on (a) c)The vectors that each general receives in step 3. (1, 2, unknown, 4)

16. Agreement in Faulty Systems (2) The same as in previous slide, except now with 2 loyal generals and one traitor. Unknown

17. 7.3 Reliable C-S Communication Five different classes of failures for RPC –The client is unable to locate the server –The request message from the client to the server is lost –The server crashes after receiving a request –The reply message from the server to the client is lost –The client crashes after sending a request.

18. 7.4 Reliable Group Communication Basic Reliable-Multicasting Schemes –Assign a number to each message it multicasts –Assuming messages are received in the order they are sent. –Sender keeps the message in its history until each receiver has returned an ACK.

19. Basic Reliable-Multicasting Schemes A simple solution to reliable multicasting when all receivers are known and are assumed not to fail a)Message transmission b)Reporting feedback

20. 7.4 Reliable Group Communication Basic Reliable-Multicasting Schemes –Problem: Feedback implosion –Solution: returning only negative ACK Problem: the sender will be forced to keep a message in its history buffer forever in theory.

21. Nonhierarchical Feedback Control Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others. Problem: multicasting feedback also interrupts those processes to which the message has been successfully delivered.

22. Hierarchical Feedback Control The essence of hierarchical reliable multicasting. a)Each local coordinator forwards the message to its children. b)A local coordinator handles retransmission requests. Problem: How to construct a tree dynamically?

23. 7.5 Distributed Commit Distributed commit –An operation being performed by each member of a process group, or none at all. One-phase commit protocol –A coordinator tells all other processes that are also involved, called participants, whether or not to (locally) perform the operation. –Problem: there is no way to tell the coordinator that some participants cannot perform the operation. Two-phase commit (2PC)

24. Two-Phase Commit (1) a)The finite state machine for the coordinator in 2PC. b)The finite state machine for a participant.

25. Two-Phase Commit (2) Actions taken by a participant P when residing in state READY and having contacted another participant Q. State of QAction by P COMMITMake transition to COMMIT ABORTMake transition to ABORT INITMake transition to ABORT READYContact another participant

26. Two-Phase Commit (3) Outline of the steps taken by the coordinator in a two phase commit protocol actions by coordinator: while START _2PC to local log; multicast VOTE_REQUEST to all participants; while not all votes have been collected { wait for any incoming vote; if timeout { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote; } if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants; } else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; }

27. Two-Phase Commit (4) Steps taken by participant process in 2PC. actions by participant: write INIT to local log; wait for VOTE_REQUEST from coordinator; if timeout { write VOTE_ABORT to local log; exit; } if participant votes COMMIT { write VOTE_COMMIT to local log; send VOTE_COMMIT to coordinator; wait for DECISION from coordinator; if timeout { multicast DECISION_REQUEST to other participants; wait until DECISION is received; /* remain blocked */ write DECISION to local log; } if DECISION == GLOBAL_COMMIT write GLOBAL_COMMIT to local log; else if DECISION == GLOBAL_ABORT write GLOBAL_ABORT to local log; } else { write VOTE_ABORT to local log; send VOTE ABORT to coordinator; }

28. Two-Phase Commit (5) Steps taken for handling incoming decision requests. actions for handling decision requests: /* executed by separate thread */ while true { wait until any incoming DECISION_REQUEST is received; /* remain blocked */ read most recently recorded STATE from the local log; if STATE == GLOBAL_COMMIT send GLOBAL_COMMIT to requesting participant; else if STATE == INIT or STATE == GLOBAL_ABORT send GLOBAL_ABORT to requesting participant; else skip; /* participant remains blocked */