Presentation is loading. Please wait.

Presentation is loading. Please wait.

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 1 Principles of Reliable Distributed Systems Lecture 8: Failure Detectors.

Similar presentations


Presentation on theme: " Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 1 Principles of Reliable Distributed Systems Lecture 8: Failure Detectors."— Presentation transcript:

1  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 1 Principles of Reliable Distributed Systems Lecture 8: Failure Detectors Spring 2009 Prof. Idit Keidar

2  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 2 Material Chandra and Toueg, Unreliable Failure Detectors for Reliable Distributed Systems. Mostefaoui and Raynal, Solving Consensus using Chandra-Toueg’s Unreliable Failure Detectors: A General Approach. Keidar and Rajsbaum, On the Cost of Fault- Tolerant Consensus When There are no Faults: A Tutorial.

3  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 3 Reminder: Consensus Each process has an input, should irrevocably decide an output Agreement: correct processes’ decisions are the same Validity: decision is input of one process Termination: eventually all correct processes decide

4  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 4 Asynchronous Model No bounds on message delays, processing times Good for unpredictable settings, e.g., Internet

5  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 5 Asynchronous Model with Crash Failures Asynchronous –Messages can be delayed arbitrarily Safety or liveness?? –Processes take steps at asynchronous times No clocks Crash failures –A process that crashes at any point in a run is faulty in that run Reliable links

6  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 6 Fault-Tolerant Asynchronous Consensus is Impossible Every asynchronous fault-tolerant consensus algorithm has a fair run in which no process decides [ FLP85 ] Fair run – if some event can happen (is enabled) long enough, this event happens –E.g., with reliable links, every sent message is eventually delivered Note: fairness is a condition on the environment, not the consensus protocol

7  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 7 Key Difficulty Distinguish slow process from faulty one When to timeout?

8  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 8 So What Should We Do? Use synchronous model? –Always possible: messages never take more than 2 days –Use long rounds (conservative timeouts) to ensure that all messages arrive on time In practice, avg. latency can be < [Cardwell, Savage, Anderson 2000], [Bakr-Keidar 2002] max. latency 100 long timeout

9  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 9 Motivation: Choosing a Model Example network: –99% of packets arrive within 10 µsec –Upper bound of 1000 µsec on message latency What would we choose the round duration for a round-based synchronous system? –Implication? We would like to choose a timeout of 10 µsec, but without violating safety…

10  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 10 The Middle Ground We can choose timeouts that usually hold –During long stable periods, delays and processing times are bounded like synchronous model –Some unstable periods like asynchronous model We can design algorithms that always ensure safety, but ensure liveness only at stable times

11  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 11 How Do We Model This? Assume that in each run there is a Global Stabilization Time (GST) after which the system is stable Unbounded Unknown

12  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 12 Eventual Synchrony (ES) Model [Dwork, Lynch, Stockmeyer 88] Processes have clocks with bounded drift There are upper bounds –  on message delay, and –  on processing time GST, global stabilization time –Until GST, unstable: bounds do not hold –After GST, stable: bounds hold –GST unbounded, unknown

13  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 Eventual Synchrony (ES) in Practice For , , choose bounds that hold with high probability Stability forever? –Model: assume yes – clean model –In practice, no need for it – stability has to last “long enough” for given algorithm to terminate Does it make it a bad model? 13

14  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 A Note on “Good Models” Accurate – analysis yields truths about the analyzed system/object Tractable – analysis is possible Accurate and tractable models are hard to define –Need to abstract away issues that do not affect the phenomena of interest –Include exactly those attributes that do 14

15  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 15 Why Assume Stability Forever? Real world: Model: That was too short – no decision yet That was long enough – decided Algorithms that work in the real-world work in the model and vice versa!

16  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 16 Time-Free Algorithms Describe algorithms using a failure detector abstraction [Chandra, Toueg 96] Goal: abstract away time, get simpler algorithms What makes a good abstraction? –Implementable in abstracted model (ES) –Sufficient for applications (consensus)

17  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 17 The Failure Detector Abstraction [Chandra, Toueg 96] Each process has a local failure detector oracle –Typically outputs list of processes suspected to have crashed at any given time Algorithm A 1 FD {p 3,p 7 } Network Algorithm A n FD {p 3 } …

18  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 18 A Natural Failure Detector Implementation in Eventual Synchrony Model Send heartbeat messages at regular times Implement failure detector using timeouts: –When expecting a message process i should send at time t, wait until t +  clock skew before suspecting i –Whenever a message from i arrives, unsuspect i In stable periods,  always hold, hence no false suspicions FD Builder

19  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 19 The Resulting Failure Detector Is ◊P - Eventually Perfect Strong Completeness: From some point on, every faulty process is suspected by every correct process Eventual Strong Accuracy: From some point on, no correct process is suspected Is it implementable in asynchronous systems?

20  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 20 t 0 q does not suspect p 00 t 0 p crashes '0'0 t 1 q suspects p t 0 p’s msgs delayed 11 t 1 q suspects pt 2 q does not suspect p '1'1 Are we done? Now,  1 is fair Build a Fair Run W/out Failures s.t. There Is No Time After Which q Does Not Suspect p t0t0 t 1 q suspects p t 2 p crahses t 3 q suspects p 22 t0t0 t 1 q suspects p t 2 p’s msgs delayed t 3 q suspects p Continue by induction to build an infinite fair run in which q is correct, suspected at t 1,t 3,t 5, …

21  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 21 The Failure Detector Abstraction Asynchronous model with failure detectors Higher level abstraction than ES model –Forget about ,  Each process has a failure detector oracle Alg Builder Algorithm A 1 FD {p 3,p 7 } ES is hidden here

22  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 22 Weaker Failure Detector: ◊S – Eventually Strong Strong Completeness Eventual Weak Accuracy: There exists some correct process that is not suspected by any correct process from some point on –Processes do not know who this process is I suspect Josh and Joe I suspect Joe and John I suspect Joe Joanne Josh Joe I suspect Josh John

23  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 23 Some Notes on ◊S ◊P is a subset of ◊S –Every failure detector of class ◊P is also of class ◊S Strictly weaker than ◊P –Sometimes homework question Equivalent to the weakest for consensus

24  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 24 Model n processes 1,…,n t<n/2 of them can crash –This is optimal; we will show later Reliable links between correct processes Asynchronous with ◊S oracle Alg Builder

25  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 25 ◊S-based Consensus: MR Algorithm [ Mostefaoui, Raynal 99 ] Asynchronous rounds: –Each process locally progresses through rounds r = 1, 2, 3, … –Different processes can progress at different times Rotating coordinator –Process i mod n is the coordinator of round i Each round consists of two phases

26  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 26 MR Algorithm val  input; est   || for r =1, 2, … do coord  (r-1 mod n)+1 if I am coord, then send (r,val) to all wait for ( (r, v) from coord OR suspect coord (by ◊S)) if receive v from coord then est  v else est   send (r, est) to all wait for (r,e) from n-t processes if any non-  value e received then val  e if all received e’s have same non-  value v then send (“decide”, v) to all return(v) || Upon receive (“decide”, v), forward to all; return(v) 1 2

27  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 27 Failure-Free Suspicion-Free Run 11 2 n...... (1, v 1 ) 1 2 n...... all have est = v 1 all decide v 1 (decide, v 1 )

28  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 28 Coordinator is Suspected 11 2 n...... (1, v 1 ) 1 2 n...... (1,  ) all have est =  delayed (2, v 2 ) delayed no decision

29  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 29 One Suspicion Per-Round is Enough for FLP 11 2 n...... (1, v 1 ) 1 2 n...... (1,  ) est =  delayed no decision

30  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 30 Phase 1 Rationale Ensure that for every p i : est i  {val coord,  } –Do all processes have the same est? Progress –Why does the 1 st phase terminate?

31  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 31 Phase 2 Rationale Ensure Agreement –If process p i decides v during round r, and process p j progresses to round r+1, then p j does so with val j = v. Progress –Why does the 2 nd phase terminate?

32  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 32 Phase 2 Rationale (Cont’d) The 2 nd phase ends upon receiving (r, est) from a majority of processes (n-t is a majority) Why is the majority important? –Every two majority sets intersect –If one process gets n-t messages with v, then every other correct process gets at least one message with v

33  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 33 Possible Scenarios in Phase 2 p i gets only v –p i decides –All other processes get v at least once p i gets only  –All other processes get  at least once –Nobody decides p i gets both v and  –Some other process might decide v –p i sets val i to v Can p i get two different values v and v’?

34  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 34 Validity Proof For every i, val i and est i always store the initial value of some process or  By induction on the length of the execution: –Initially, for every process i, val i stores i’s initial value, and est i is  –Subsequently, they can only change to store a val j or est i value sent by some process j

35  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 35 Lemmas Lemma 1: If in some round r, two messages (r,v) and (r,v’) are sent such that v ≠  and v’ ≠ , then v=v’. Lemma 2: If in some round r, n-t processes send (r,v), then for every round r’>r, if a message (r’,v’) with v’ ≠  is sent, then v=v’. –Hint: n-t > n/2.

36  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 36 Agreement Proof Assume by contradiction that two different decisions, v ≠ v’ are made. Let r (r’) be the first round in which some process i (i’) decides v (v’) when it receives n-t (r,v) ((r’,v’)) messages. By Lemma 1, r ≠ r’, and by Lemma 2, neither r > r’ nor r’>r. A contradiction.

37  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 37 Termination Proof Steps Progress: until some process decides, no process is ever “stuck” in a round forever First decision: some correct process eventually decides Subsequent decisions: if some correct process decides, then all correct processes eventually decide

38  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 38 What Do We Need “Decide” For? val  input; est   || for r =0,1, 2, … do coord  (r mod n)+1 if I am coord, then send (r,val) to all wait for ( (r, val) from coord OR suspect coord (by ◊S)) if receive val from coord then est  val else est   send (r, est) to all wait for (r,est) from n-t processes if any non-  est received then val  est if all ests have same non-  value v then send (“decide”, v) to all return(v) od

39  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 39 Why Send “Decide”? 11 2 n...... (1, v 1 ) 1 2 n...... suspect 1 est =  delayed no decision delayed (1,  ) Decide

40  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 40 Disseminating the Decision OK, so we need the 1 st “decide”. Why forward to all? Hint: reliable broadcast

41  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 41 Why Forward “Decide”? n=4, t=1 11 2 4 (1, v 1 ) 1 2 4 suspect 1 est =  no decision (1,  ) decide 3 3 2 4 3 (2, v 1 ) X X 4 2 stuck, no n-t

42  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 42 How Long Does It Take? The algorithm can take unbounded time –What if no failures occur? Is this inevitable? Can we say more than “decision is reached eventually” ?

43  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 43 Performance Metric Number of communication steps in well-behaved runs Well-behaved: –No failures –Stable (synchronous) from the beginning –With failure detector: no false suspicions Motivation: common case

44  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 44 The Algorithm’s Running Time in Well-Behaved Runs In round 1, the coordinator is correct, not suspected by any process All processes decide at the end of phase two of round 1 –Decision in two communication steps –Halting (stopping) takes three steps –Same as in synchronous model For Uniform Consensus

45  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 45 Back to Our Example Example network: –99% of packets arrive within 10 µsec –Upper bound of 1000 µsec on message latency Now we can choose a timeout of 10 µsec, without violating safety! Most of the time, the algorithm will be just as fast as a synchronous uniform consensus algorithm –We did pay a price in resilience, though

46  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 46 Indulgent Algorithms ◊S or ◊P failure detector’s output can be wrong (even arbitrary) for an unbounded (finite) prefix of a run An algorithm that tolerates unbounded periods of asynchrony is called indulgent [ Guerraoui 98 ]

47  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 47 Observations on Indulgent Consensus Algorithms Every indulgent consensus algorithm also solves uniform consensus [ Guerraoui 98 ] It is impossible to solve t-resilient indulgent consensus when t ≥ n/2 [ Chandra, Toueg 96; Guerraoui 98 ] –Proof on the board (see next slide)

48  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 48 33 Example: n=4, t=2 p1 p2 P Decide 0 by validity and termination p3 p4 Q p1 p2 P Decide 1 p3 p4 Q 0 0 1 0 0 1 1 1 11 22 x x x x Decide 0 by validity and termination Can all fail because |Q| ≤ t Decide 1

49  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 49 33 Example: n=4, t=2 p1 p2 P Decide 0 by validity and termination p3 p4 Q 0 0 1 1 Decide 0 by validity and termination Decide 1 All messages between P and Q arrive after decisions are already made

50  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 50 Alternative Weak Failure Detector  – Leader –Outputs one trusted process –From some point, all correct processes trust the same correct process Can easily implement ◊S Is the weakest for consensus [Chandra, Hadzilacos, Toueg 96]

51  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 51 A Natural  Implementation Use ◊P implementation Output lowest id non-suspected process

52  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 52 Summary Prepare for the worst –Safety under asynchrony Hope for the best –Liveness & good performance in common cases Nice clean models –Eventual stability –Time-free abstractions: unreliable failure detectors


Download ppt " Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2009 1 Principles of Reliable Distributed Systems Lecture 8: Failure Detectors."

Similar presentations


Ads by Google