1. Fault tolerance and related issues in distributed computing Shmuel Zaks zaks@cs.technion.ac.il GSSI - Feb 2016114

2. Part 0: Part 0: An overview Part 1: Part 1: Lower bounds Part 2: Part 2: Computing in spite of faults Part 3: Part 3: Detecting faults Part 4: Part 4: Self-stabilization 115GSSI - Feb 2016

3. problem, task P1P1 P2P2 P3P3 input output consensus 1 0 0 1 1 1 116GSSI - Feb 2016

4. 117 Consensus Input: 1 or 0 to each processor Output: Agreement: all procssors decide 0 or 1 Termination: all processors eventually decide Validity: if all inputs x, then decide x

5. GSSI - Feb 2016118 The result: No completely asynchronous consensus protocol can tolerate even a single unannounced process death.

6. GSSI - Feb 2016119 This problem serves a role that is similar to the one served by “the halting problem” in computability theory. Many problems equivalent to consensus (or reduce to it)

7. GSSI - Feb 2016120 How protocols in practice deal with this outcome ? Weaken an assumption. For example: Computation model: e.g., assume bounded –delay network Computation model: e.g., assume bounded –delay network Fault model: e.g., assume faults only at start. Fault model: e.g., assume faults only at start.

8. GSSI - Feb 2016121 The Model Message System Message System Reliable Reliable Delivers all messages correctly Delivers all messages correctly Exactly once Exactly once Processing Model Completely Asynchronous No Assumptions about relative speeds No Assumptions about relative speeds Unbounded time in delivering message Unbounded time in delivering message

9. GSSI - Feb 2016122 System Model Communicate by means of one global message buffer Atomic step Attempt to receive a message Perform local computation Send arbitrary but finite set of messages

10. GSSI - Feb 2016123 Consensus Protocol N processes (N > 1) process p x p 0/1 y p 0/1/b memory(unboundd)PC input register output register memory Program counter

11. GSSI - Feb 2016124 Fixed starting valued at the memory (except the input register) Output register starts with b The output register is “write once” when a value is written to the output register, the process is “in a decision state”. Process acts deterministically according to a Transition function

12. GSSI - Feb 2016125 Communication System A message is a pair (p,m) p is the name of the destination m is a “message value” message buffer Maintains messages that have been sent but not yet delivered We assume a clique topology

13. GSSI - Feb 2016126 two operations by a process : send (p,m) – place (p,m) in the message buffer ( “message (p,m) is sent to process p”) receive (p) delete a message (p,m) from the message buffer and return m (note: no fifo required) ( “message (p,m) is received”) OR return  (message buffer unchanged)

14. GSSI - Feb 2016127 Message system nondeterministic. However, each message (p,m) in the message buffer: if receive(p) is performed  times, then (p,m) is eventually delivered. In other words: in response to receive(p) : if a message (p,m) is in the message buffer, then the message system can return , but only a finite number of times.

15. GSSI - Feb 2016128 (P 1,M) Message Buffer (P 0,M’) (P 2,M’’) (P 1,M’’’) Process P 0 Process P 2 Process P 1 receive( P 0 )  (P 0,M’)

16. GSSI - Feb 2016129 (P 1,M) Message Buffer Process P 2 Process P 1 receive(1) (P 2,M’’) (P 1,M’’’) send(P 2,m) (P 2,m) Process P 0 (P 0,M’)

17. GSSI - Feb 2016130 Configurations A configurations consists of Internal state of each process Contents of the message buffer initial configuration each process p starts with x p =0 or x p =1 the message buffer is empty

18. GSSI - Feb 2016131 event e = (p,m) (“m is received by p”). step of a single process p: receive(p) is performed ( p receives m) p enters a new internal state p sends a finite set of messages event and step: event: syntax step: semantic

19. GSSI - Feb 2016132 Events and Schedules e(C) – denotes the resulting configuration after applying event e to configuration C (“e can be applied to C”) The event (p,  ) can always be applied A schedule from C is a finite/infinite sequence  of events that can be applied from C. The associated sequence of steps is called a run. one: event - step many: schedule - run

20. GSSI - Feb 2016133 If a schedule  is finite,  (C) denotes the resulting configuration C’, which is “reachable from C “. C’ is accessible if it is reachable from an initial configuration.

21. GSSI - Feb 2016134 Lemma 1 (‘commutativity’) Lemma 1 : Suppose that from some configuration C, the schedules  1,  2 lead to configurations C 1,C 2, respectively. If the sets of processes taking steps in  1 and  2, respectively, are disjoint, then  2 can be applied to C 1, and  1 can be applied to C 2, and both lead to the same configuration C 3.

22. GSSI - Feb 2016135 C2C2C2C2 C0C0C0C0 C1C1C1C1 C3C3C3C3 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2

23. GSSI - Feb 2016136 C2C2C2C2 C0C0C0C0 C1C1C1C1 C3C3C3C3 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 First case: both  1 and  2 contain a single event First case: both  1 and  2 contain a single event  1 =(P 1,M 1 )  1 =(P 1,M 1 )  2 =(P 2,M 2 )  2 =(P 2,M 2 ) P 1 ≠ P 2 P 1 ≠ P 2

24. GSSI - Feb 2016137 (P 1,M 1 ) (P 2,M 2 ) (P 1,M 1 ) 1111 2222 1111 2222 The message buffer of C 3 The message buffer of C 1 The message buffer of C 2 The message buffer of C 0 Message buffer

25. GSSI - Feb 2016138 P 1 Internal state - A P 2 Internal state - X P 1 Internal state - B P 2 Internal state - Y P 1 Internal state - B P 2 Internal state - X P 1 Internal state - A P 2 Internal state - Y 1111 2222 1111 2222 All other processors – change unchanged states

26. GSSI - Feb 2016139 C2C2C2C2 C0C0C0C0 C1C1C1C1 C3C3C3C3 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 when  1 and  2 contain a single event (p,m) event - ok when  1 and  2 contain a single event (p,m) event - ok when  1 and  2 contain any run – use induction when  1 and  2 contain any run – use induction

27. GSSI - Feb 2016140 Recall: If a schedule  is finite,  (C) denotes the resulting configuration C’, which is “reachable from C “. C’ is accessible if it is reachable from an initial configuration.

28. GSSI - Feb 2016141 A configuration C has a decision value v if some process p is in a decision state with y p = v (v =0 or v=1). if some process p is in a decision state with y p = v (v =0 or v=1). A consensus protocol is partially correct if it satisfies two conditions: 1. No accessible configuration has more than one decision value. 2. For each v  {0,1}, some accessible configuration has decision value v. good news - it is non trivial - sometimes it decides - it never decides incorrectly bad news - termination not guaranteed - what about delivering all messages? -what about failures?

29. GSSI - Feb 2016142 A process p is nonfaulty in a run if it takes  steps. It is faulty otherwise. A process p is nonfaulty in a run if it takes  steps. It is faulty otherwise. bad news: a process can be declared faulty only at  !! A run is admissible if A run is admissible if - at most one process is faulty, and - at most one process is faulty, and - all messages sent to non-faulty - all messages sent to non-faulty processes are eventually received. processes are eventually received.

30. GSSI - Feb 2016143 A run is deciding if some process reaches a decision state. A run is deciding if some process reaches a decision state. A consensus protocol is totally correct in spite of one fault if it is: A consensus protocol is totally correct in spite of one fault if it is: partially correct, and partially correct, and every admissible run is a deciding run. every admissible run is a deciding run. (recall: A run is admissible if at most one process is faulty, and all messages sent to non-faulty processes are eventually received.

31. GSSI - Feb 2016144 Theorem (Fischr,Lynch,Paterson): No consensus protocol is totally correct in spite of one fault.

32. GSSI - Feb 2016145 Sketch of Proof: Assume that P is totally correct in spite of one fault. sshow an initial configuration from which each decision is still possible ( Lemma 2 ) sshow that from such a configuration one can always reach another similar configuration ( Lemma 3 ) cconclude – by induction – with an admissible run that never decides – a contradiction.

33. GSSI - Feb 2016146 Let be a configuration and let V be the set of decision values of configurations reachable from. Let C be a configuration and let V be the set of decision values of configurations reachable from C. is if C is bivalent if |V| = 2 is if C is univalent if |V| = 1 if then is if V = {0} then C is 0-valent if then is if V = {1} then C is 1-valent (Note: |V|≠0, since P is totally correct) No consensus protocol is Theorem: No consensus protocol is totally correct in spite of one fault. totally correct in spite of one fault. Proof: Assume that P is totally correct in spite of one fault. We will reach a contradiction.

34. GSSI - Feb 2016147 0-valent configuration From now on: 1-valent configuration 2-valent configuration Unknown

35. GSSI - Feb 2016148 : Proof: Assume there is no bivalent initial configuration. But P is partially correct. So, there are both 0-valent and 1-valent initial configurations. initial configurations. :. Lemma 2: P has a bivalent initial configuration.

36. GSSI - Feb 2016149...... bivalent configuration initial configurations C

37. GSSI - Feb 2016150 C0C0C0C0...... 0-valentconfiguration C1C1C1C1 initial configurations 1-valentconfiguration

38. GSSI - Feb 2016151 Two initial configurations are called if they differ only in the initial value of a single process. Two initial configurations are called adjacent if they differ only in the initial value of a single process. 0 1 0 1 1 0 1 0 1 0 x 0 x 1 x 2 x 3 x 4

39. GSSI - Feb 2016152 Claim: There exist a 0-valent initial configuration C 0 adjacent to a 1-valent initial configuration C 1.

40. GSSI - Feb 2016153 0 1 0 1 1 1 1 0 1 1 1 1 0 1 0 1 1 0 0 0 1 0 0 0 0 x 0 x 1 x 2 x 3 x 4 C0C0 C1C1 Proof by example:0-valent 1-valent

41. GSSI - Feb 2016154 So: There exist a 0-valent initial configuration C 0 adjacents to a 1-valent initial configuration C 1. Let p be the process in whose initial value they differ

42. GSSI - Feb 2016155 P is a consensus protocol that is totally correct in spite of one fault. P is a consensus protocol that is totally correct in spite of one fault. Consider an admissible deciding run (with schedule ) from in which process takes no steps. Consider an admissible deciding run (with schedule  ) from C 0 in which process p takes no steps. can be applied to  can be applied to C 1 The two corresponding configurations are identical, except for the internal state in The two corresponding configurations are identical, except for the internal state in p Both runs reach the same decision Both runs reach the same decision x

43. GSSI - Feb 2016156 x = 1 C 0 is bivalent x = 0 C 1 is bivalent Contradiction. C1C1C1C1 C0C0C0C0 C’   C’’ Decision: x x 0-valent 1-valent Lemma 2: P has a bivalent initial configuration. So, we proved:

44. GSSI - Feb 2016157 Lemma 3: Let: be a C be a bivalent configuration of P, e = (p,m) be an event that is applicable to C. S be the set of configurations reachable from C without applying e, and D=e(S)= { e(E)| E  S and e is applicable to E }.. Then, D contains a bivalent configuration. (Note: it is ok if we choose m= (Note: it is ok if we choose m= .)

45. GSSI - Feb 2016158 Note: e =(p,m) is applicable to C so: message (p,m) is in the message buffer, so: e is applicable to every E  S.

46. GSSI - Feb 2016159 E e2e2e2e2 e1e1e1e1 e4e4e4e4 e i ≠ e bivalent configuration e e e e S e D=e(S) e e5e5e5e5 e6e6e6e6 e7e7e7e7 C Need to prove: D contains a bivalent configuration

47. GSSI - Feb 2016160 Prove by contradiction Assume that D contains no D=e(S) e i ≠ e e e e e S e e C 0-valent 1-valent

48. GSSI - Feb 2016161 Step 1: Claim: D contains both and 0-valent 1-valent So: every configuration d  D is or The proof has three steps.

49. GSSI - Feb 2016162 S e D=e(S) D0D0D0D0 D1D1D1D1 e e=(p,m) Step 1

50. GSSI - Feb 2016163 C is bivalent There exist E i,, i=0,1, i-valent configurations reachable from C. e i ≠ e e e e e S e D=e(S) e C

51. GSSI - Feb 2016164 L et F 1 = e (E 1 ). E1E1E1E1 0 e2e2e2e2 e1e1e1e1 e4e4e4e4 e i ≠ e bivalent configurati on F1F1F1F1 e e e e S e D=e(S) e e5e5e5e5 e6e6e6e6 e7e7e7e7 C 0-valent 1-valent so: D contains

52. GSSI - Feb 2016165 e was applied in reaching E 0 so, either E 0 is in D, or there exists F 0  D from which E 0 is reachable. e2e2e2e2 e1e1e1e1 e4e4e4e4 e i ≠ e bivalent configurati on e e e e S e D=e(S) e e5e5e5e5 e6e6e6e6 e7e7e7e7 E0E0E0E0 C 0-valent 1-valent so: D contains

53. GSSI - Feb 2016166 e was applied in reaching E 0 so, either E 0 is in D, or there exists F 0  D from which E 0 is reachable. e2e2e2e2 e1e1e1e1 e4e4e4e4 e i ≠ e bivalent configurati on e e e e S e D=e(S) e e5e5e5e5 e6e6e6e6 e7e7e7e7 F0F0F0F0 E0E0E0E0 C 0-valent 1-valent so: D contains F0F0F0F0

54. GSSI - Feb 2016167 So, D contains both and 0-valent 1-valent End of step 1 Start of step 2

55. GSSI - Feb 2016168 Step 2 Claim: There exist C 0, C 1  S such that: C 0 and C 1 are neighbors ( C 1 = e’(C 0 ), e’=(p’,m’) ) D 0 = e(C 0 ) is and D 1 = e(C 1 ) is, or D 0 = e(C 0 ) is and D 1 = e(C 1 ) is. (two configurations neighbors if one results from the other in a single step.) 0-valent 1-valent

56. GSSI - Feb 2016169 S e D=e(S) D0D0D0D0 D1D1D1D1 e’ C1C1C1C1 C0C0C0C0 e e=(p,m) e’=(p’,m’) Step 2

57. GSSI - Feb 2016170 e(C) is or. If : There are and in D. They have predecessors in S. e(C) S D=e(S) e(C) C e e 0-valent 1-valent

58. GSSI - Feb 2016171 Consider the path in S from C to the predecessor of e(C) S e D=e(S) e e(C) C 0-valent 1-valent

59. GSSI - Feb 2016172 Applying e to each configuration on this path, we get a configuration in D, which is or. bivalent configurati on S e D=e(S) e e(C) e e e C

60. GSSI - Feb 2016173 So we get two configurations C 0 and C 1, that are neighbors in S; i.e., there is e’ s.t. S e D=e(S) e(C) D0D0D0D0 D1D1D1D1 e’ C1C1C1C1 C0C0C0C0 C e

61. GSSI - Feb 2016174 e(C) is or. If : There are and in D. They have predecessors in S. e(C) S D=e(S) e(C) C e e 0-valent 1-valent

62. GSSI - Feb 2016175 Consider the path in S from C to the predecessor of e(C) S e D=e(S) e e(C) C 0-valent 1-valent

63. GSSI - Feb 2016176 So we get two configurations C 0 and C 1, that are neighbors in S; i.e., there is e’ s.t. S e D=e(S) e(C) D0D0D0D0 D1D1D1D1 e’ C1C1C1C1 C0C0C0C0 C e

64. GSSI - Feb 2016177 End of step 2 Start of step 3 So we proved:2 Claim: There exist C 0, C 1  S such that: C 0 and C 1 are neighbors ( C 1 = e’(C 0 ), e’=(p’,m’) ) D 0 = e(C 0 ) is and D 1 = e(C 1 ) is, or D 0 = e(C 0 ) is and D 1 = e(C 1 ) is.

65. GSSI - Feb 2016178 D 1 = e’(D 0 ) by Lemma 1 Case 1 : Case 1 : p’ ≠ p contradiction S e D=e(S) e(C) D0D0D0D0 D1D1D1D1 e’ C1C1C1C1 C0C0C0C0 C e Step 3: get to a contradiction e=(p,m), e’=(p’,m’) D0D0D0D0 assume

66. GSSI - Feb 2016179 S e D=e(S) D0D0D0D0 D1D1D1D1 e’ C1C1C1C1 C0C0C0C0 e e=(p,m) e’=(p’,m’) p’ = p Case 2 : Case 2 : p’ = p recall:

67. GSSI - Feb 2016180 C1C1C1C1 C0C0C0C0 D0D0D0D0 D1D1D1D1 A Case 2 : Case 2 : p’ = p e  - deciding run from C 0 in which p takes no steps A =  (C 0 )  deciding run 1-valent 0-valent e e’ e e   E0E0E0E0 E1E1E1E1   is a deciding run. But A cannot be and it cannot be. a contradiction !!!

68. GSSI - Feb 2016181 Lemma 3: Let: C be a bivalent configuration of P, C be a bivalent configuration of P, e = (p,m) be an event that is applicable to C. e = (p,m) be an event that is applicable to C. S be the set of configurations reachable from S be the set of configurations reachable from C without applying e, and C without applying e, and D = e(S) = {e(E)| E  S and e is applicable to E}. D = e(S) = {e(E)| E  S and e is applicable to E}. Then, D contains a bivalent configuration. Lemma 2: P has a bivalent initial configuration. So, we proved:

69. GSSI - Feb 2016182

70. GSSI - Feb 2016183 Any deciding run from a bivalent initial configuration goes to univalent configuration, so there must be some single step that goes from a bivalent to univalent configuration. We construct a run that avoids such a step: bivalent configuration deciding run bivalent configuration … univalent configuration end of proof:

71. GSSI - Feb 2016184 we construct an infinite non-deciding run bivalent configuration non-deciding run bivalent configuration … …

72. GSSI - Feb 2016185 Start with a bivalent initial configuration ( Lemma 2) The run constructed in stages. Every stage starts with a bivalent configuration and ends with a bivalent configuration A queue of processes, initially in arbitrary order Message buffer is ordered according to the time messages were sent

73. GSSI - Feb 2016186 In each stage: C is a bivalent configuration that the stage starts with. Suppose that process p heads the queue Suppose that m is the earliest message to p in the message buffer if any (or  otherwise) e = (p,m)

74. GSSI - Feb 2016187 By Lemma 3 there is a bivalent configuration C’ reachable from C by a schedule in which e is the last event. After applying this schedule: move p to the back of the queue

75. GSSI - Feb 2016188 in any infinite sequence of stages every process takes infinitely many steps every process receives every message sent to it Therefore, the constructed run is admissible never reaches a univalent configuration The protocol never reaches a decision The protocol is not totally correct in spite of one fault. contradiction

76. GSSI - Feb 2016189 Conclusion Theorem: No consensus protocol is totally correct in spite of one fault. Q: which process fails in the infinite run that was constructed for the proof?

77. GSSI - Feb 2016190 One importance lesson: In an asynchronous system, there is no way to distinguish between a faulty process and a slow process. Other tasks not solvable with one faulty processor: Input graph – connected Output graph - disconnected Many extensions and uses

78. GSSI - Feb 2016191 References M. Fischer, N. Lynch, M. Paterson, Impossibility of distributed consensus with one faulty processor, JACM, 1985. O. Biran, S. Moran and S. Zaks, A Combinatorial Characterization of the Distributed Tasks Which Are Solvable in the Presence of One Faulty Processor, J. of Algorithms, 1990.