1.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 1 Principles of Reliable Distributed Systems Lecture 7: Failure Detectors Spring 2006 Dr. Idit Keidar

2.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 2 Material Chandra and Toueg, Unreliable Failure Detectors for Reliable Distributed Systems. Mostefaoui and Raynal, Solving Consensus using Chandra-Toueg’s Unreliable Failure Detectors: A General Approach. Keidar and Rajsbaum, On the Cost of Fault- Tolerant Consensus When There are no Faults: A Tutorial.

3.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 3 Fault-Tolerant Asynchronous Consensus is Impossible Every asynchronous fault-tolerant consensus algorithm has a fair execution in which no process decides [ FLP85 ] It is possible to design asynchronous consensus algorithms that don’t always terminate

4.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 4 Should We Give Up? We can always model a system as synchronous with the right timeout –messages never take more than 2 days But modeling systems as synchronous requires conservative timeouts –problem?

5.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 5 Motivation: Choosing a Model Example network: –99% of packets arrive within 10 µsec –upper bound of 1000 µsec on message latency What would we choose the round duration for a round-based synchronous system? –implication? We would like to choose a timeout of 10 µsec, but without violating safety…

6.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 6 The Middle Ground We can choose timeouts that usually hold –during long stable periods, delays and processing times are bounded like synchronous model –some unstable periods like asynchronous model We can design algorithms that always ensure safety, but ensure liveness only at stable times

7.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 7 How Do We Model This? Option 1: Change the problem statement to require conditional liveness –e.g., replace termination with: “if there is a time after which the system is stable (synchronous) then all correct processes eventually decide” Option 2: Change the model –assume there is a Global Stabilization Time (GST) after which the system is stable

8.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 8 These Options Are Equivalent If a safety property holds for a given algorithm in a partial synchrony model, the same property also holds for runs of that algorithm in an asynchronous model –because safety properties are prefix-closed The liveness properties are conditional on the existence of GST

9.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 9 Eventual Synchrony Model [Dwork, Lynch, Stockmeyer 88] Processes have clocks with bounded drift There are upper bounds –  on message delay, and –  on processing time GST, global stabilization time –Until GST, unstable: bounds do not hold –After GST, stable: bounds hold –GST unbounded, unknown

10.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 10 Eventual Synchrony in Practice For , , choose bounds that hold with high probability Stability forever? –we assume that once stable remains stable –in practice, has to last “long enough” for given algorithm to terminate

11.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 11 Time-Free Algorithms Describe algorithms using a failure detector abstraction [Chandra, Toueg 96] Goal: abstract away time, get simpler algorithms What makes a good abstraction?

12.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 12 The Failure Detector Abstraction [Chandra, Toueg 96] Each process has local failure detector oracle –typically outputs list of processes suspected to have crashed at any given time In each execution step, a process –receives a message (if there is one ready) –queries its failure detector oracle –makes a transition to a new state –may send messages to other processes

13.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 13 A Natural Failure Detector Implementation in Eventual Synchrony Model Implement failure detector using timeouts: –When expecting a message from a process i, wait  clock skew before suspecting i In stable periods,  always hold, hence no false suspicions

14.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 14 The resulting failure detector is ◊P - Eventually Perfect Strong Completeness: From some point on, every faulty process is suspected by every correct process Eventual Strong Accuracy: From some point on, no correct process is suspected Is it implementable in asynchronous systems?

15.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 15 t 0 q does not suspect p 00 t 0 p crashes '0'0 t 1 q suspects p t 0 p’s msgs delayed 11 t 1 q suspects pt 2 q does not suspect p '1'1 Are we done? Now,  1 is fair Build a fair run without failures s.t. there is no time after which q does not suspect p t0t0 t 1 q suspects p t 2 p crahses t 3 q suspects p 22 t0t0 t 1 q suspects p t 2 p’s msgs delayed t 3 q suspects p Continue by induction to build an infinite run in which q is correct, suspected at t 1,t 3,t 5, …

16.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 16 Unreliable Failure Detectors Failure detector’s output can be wrong (even arbitrary) for an unbounded (finite) prefix of a run Captures eventual synchrony An algorithm that tolerates unbounded periods of asynchrony is called indulgent [ Guerraoui 98 ]

17.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 17 Observations on Indulgent Consensus Algorithms Every indulgent consensus algorithm also solves uniform consensus [ Guerraoui 98 ] It is impossible to solve t-resilient indulgent consensus when t ≥ n/2 [ Chandra, Toueg 96; Guerraoui 98 ] –henceforward, assume t < n/2

18.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 18 Weaker Failure Detector ◊S – Eventually Strong –Strong Completeness –Eventual Weak Accuracy: From some point on, some correct process is not suspected by any correct process ◊P is a subset of ◊S –Every failure detector of class ◊P is also of class ◊S Strictly weaker than ◊P –homework question Equivalent to the weakest for consensus

19.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 19 Model n process 1,…n t<n/2 of them can crash Reliable links between correct processes Asynchronous with ◊S

20.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 20 ◊S-based Consensus [ Mostefaoui, Raynal 99 ] val  input; est   || for r =1, 2, … do coord  (r-1 mod n)+1 if I am coord, then send (r,val) to all wait for ( (r, v) from coord OR suspect coord (by ◊S)) if receive v from coord then est  v else est   send (r, est) to all wait for (r,e) from n-t processes if any non-  value e received then val  e if all received e’s have same non-  value v then send (“decide”, v) to all return(v) || Upon receive (“decide”, v), forward to all; return(v) 1 2

21.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 21 Failure-Free Suspicion-Free Run 11 2 n...... (1, v 1 ) 1 2 n...... all have est = v 1 all decide v 1 (decide, v 1 )

22.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 22 Validity Proof For every i, val i and est i always store the initial value of some process or  By induction on the length of the execution Initially, for every process i, val i stores i’s initial value, est i is  Subsequently, they can only change to store a val j or est i value sent by some process j

23.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 23 Lemmas Lemma 1: If in some round r, two messages (r,v) and (r,v’) are sent such that v ≠  and v’ ≠ , then v=v’. Lemma 2: If in some round r, n-t processes send (r,v), then for every round r’>r, if a message (r’,v’) with v’ ≠  is sent, then v=v’. –Hint: n-t > n/2.

24.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 24 Agreement Proof Assume by contradiction that two different decisions, v ≠ v’ are made. Let r (r’) be the first round in which some process i (i’) decides v (v’) when it receives n-t (r,v) ((r’,v’)) messages. By Lemma 1, r ≠ r’, and by Lemma 2, neither r > r’ nor r’>r. A contradiction.

25.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 25 Termination Proof Steps Progress: until some process decides, no process is ever “stuck” in a round forever First decision: some correct process eventually decides Subsequent decisions: if some correct process decides, then all correct processes eventually decide

26.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 26 How Long Does It Take? The algorithm can take unbounded time –what if no failures occur? Is this inevitable? Can we say more than “decision is reached eventually” ?

27.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 27 Performance Metric Number of communication steps in well-behaved runs Well-behaved: –No failures –Stable (synchronous) from the beginning –No false suspicions Motivation: common case

28.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 28 The Algorithm’s Running Time in Well-Behaved Runs In round 1, the coordinator is correct, not suspected by any process All processes decide at the end of phase two of round 1 –decision in two communication steps –halting (stopping) takes three steps

29.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 29 Alternative Weak Failure Detector  – Leader –Outputs one trusted process –From some point, all correct processes trust the same correct process Can easily implement ◊S Is the weakest for consensus [Chandra, Hadzilacos, Toueg 96]

30.  Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 30 A Natural  Implementation Use ◊P implementation Output lowest id non-suspected process