1. Timeliness, Failure Detectors, and Consensus Performance Alex Shraer Joint work with Dr. Idit Keidar Technion – Israel Institute of Technology In PODC 2006

2. How do you survive failures and achieve high availability?

3. Replication

4. State Machine Replication aaa bb c Replicas are identical deterministic state machines Process operations in the same order  remain consistent

5. Consensus Building block for state machine replication Each process has an input, should decide on an output so that – Agreement: decisions are the same Validity: decision is input of one process Termination: eventually all correct processes decide

6. Keidar & Shraer, Technion, Israel PODC 2006 Basic Model Message passing Links between every pair of processes –do not create, duplicate or alter messages (integrity) Process and link failures

7. Keidar & Shraer, Technion, Israel PODC 2006 Synchronous Model a b c Very convenient for algorithms –understanding performance –early decision with no/few failures a b c a b c d d d

8. Keidar & Shraer, Technion, Israel PODC 2006 Synchronous Model Known bound Δ on message delay, processing Very convenient for algorithms Requires very conservative timeouts –in practice: avg. latency < –Computation might be too sloooow! max. latency 100 [Cardwell, Savage, Anderson 2000], [Bakr-Keidar 2002]

9. Keidar & Shraer, Technion, Israel PODC 2006 Asynchronous Model Unbounded message delay Much more practical Fault-tolerant consensus impossible [FLP85]

10. Keidar & Shraer, Technion, Israel PODC 2006 The DLS solution [Dwork, Lynch, Stockmeyer 88] Asynchronous fault-tolerant service you can live with –Always safe – no matter how slow the network is –Usually live What does “usually” mean? Option 1Option 2 Your account balance is:

11. Keidar & Shraer, Technion, Israel PODC 2006 Eventual Synchrony! (DLS) Formally capture liveness conditions Bounds known, but hold only eventually, from some unknown time GST onward –“ eventually ” formally models “ most of the time ” (in stable periods) In practice, does not have to be forever, just “ long enough ” for the algorithm to finish its task

12. Keidar & Shraer, Technion, Israel PODC 2006 Unreliable Failure Detectors [Chandra, Toueg 96] Asynchronous distributed system (no bounds). Reliable links (sent messages eventually arrive to correct processes) Each process has local failure detector oracle –Typically outputs list of processes suspected to have crashed at any given time Unreliable: failure detector output can be arbitrary for unbounded (finite) prefix of run

13. Keidar & Shraer, Technion, Israel PODC 2006 Failure Detectors - Examples ◊S - Eventually Strong –Strong Completeness: From some point on, every faulty process is suspected by every correct process –Eventual Weak Accuracy: From some point on, some correct process is not suspected  - Leader –Outputs one trusted process –From some point, all correct processes trust the same correct process Equivalent Weakest for consensus

14. Keidar & Shraer, Technion, Israel PODC 2006 Eventually Stable (Indulgent) Models Initially asynchronous –for unbounded period of time Eventually reach stabilization –GST (Global Stabilization Time) –following GST certain assumptions hold Examples –ES (Eventual Synchrony) – starting from GST all links have a bound on message delay [Dwork, Lynch, Stockmeyer 88] –failure detectors Example:  leader) failure detector –Outputs one trusted process –From some point, all correct processes trust the same correct process [Chandra, Toueg 96], [Chandra, Hadzilacos, Toueg 96]

15. Keidar & Shraer, Technion, Israel PODC 2006 Indulgent Models: Research Trend Weaken post-GST assumptions as much as possible [Guerraoui, Schiper96], [Aguilera et al. 03, 04], [Malkhi et al. 05] Weaker = better?

16. Keidar & Shraer, Technion, Israel PODC 2006 You only need ONE machine with eventually ONE timely link. Buy the hardware to ensure it, set the timeout accordingly, and EVERYTHING WILL WORK. Indulgent Models: Research Trend

17. Keidar & Shraer, Technion, Israel PODC 2006 Consensus with Weak Assumptions Network Why isn’t anything happening ???Don’t worry! It will eventually happen!

18. Keidar & Shraer, Technion, Israel PODC 2006 Consensus with Weak Assumptions Network

19. Keidar & Shraer, Technion, Israel PODC 2006 What’s Going On? In practice, bounds just need to hold “long enough” for the algorithm (T A ) to finish But T A depends on our synchrony assumptions –with weak assumptions, T A might be unbounded For practical systems, eventual completion of the job is not enough!

20. Keidar & Shraer, Technion, Israel PODC 2006 Our Goal Understand the relationship between: –assumptions (1 timely link, failure detectors, etc.) that eventually hold –performance of algorithms that exploit these assumptions, and only them Challenge: How do we understand the performance of asynchronous algorithms that make very different assumptions?

21. Keidar & Shraer, Technion, Israel PODC 2006 Typical Metric: Count “Rounds” Algorithms normally progress in rounds, though rounds are not synchronized among processes at process p i : forever do send messages receive messages while (!some conditions) compute… Previous work: –look at synchronous runs (every message takes exactly  time) –count rounds or “  s” [Keidar, Rajsbaum 01], [Dutta, Guerraoui 02], [Guerraoui, Raynal 04] [Dutta et al. 03], etc.

22. Keidar & Shraer, Technion, Israel PODC 2006 Are All “Rounds” the Same? Algorithm 1 waits for messages from a majority that includes a pre-defined leader in each round –takes 3 rounds Algorithm 2 waits for messages from all (unsuspected) processes in each round –E.g., group membership –takes 2 rounds

23. Keidar & Shraer, Technion, Israel PODC 2006 Do All Rounds Cost the Same? LAN Market Apples $1.00Oranges $1.00

24. Keidar & Shraer, Technion, Israel PODC 2006 Do All “Rounds” Cost the Same? On the Internet, n 2 timely links can be a rarity, [Bakr, Keidar 02] Timely communication –with leader –with majority require timeouts orders of magnitude smaller WAN Market Apples $1.00 Oranges $100.00

25. GIRAF General Round-based Algorithm Framework Inspired by Gafni ’ s RRFD, generalizes it Organize algorithms into rounds Separate algorithm logic from waiting condition Waiting condition defines model Allows reasoning about lower and upper bounds for rounds of different types

26. Keidar & Shraer, Technion, Israel PODC 2006 waiting condition controlled by env. GIRAF – The Generic Algorithm Your pet algorithm here Algorithm for process p i upon receive m add m to M (msg buffer) upon end-of-round FD ← oracle (k) if (k = 0) then out_msg ← initialize(FD) else out_msg ← compute(k, M, FD) k ← k+1 enable sending of out_msg to all

27. Keidar & Shraer, Technion, Israel PODC 2006 GIRAF’s Generality Does not require rounds to be synchronized among processes Can capture any oracle model –in [CHT96] general failure detector model –leader oracle + majority in each round Messages can arrive in any round –allows for untimely albeit reliable links

28. Keidar & Shraer, Technion, Israel PODC 2006 Defining Properties in GIRAF Environment can have –perpetual properties –eventual properties In every run r, there exists a round GSR(r) GSR(r) – the first round from which: –no process fails –all eventual properties hold in each round

29. Keidar & Shraer, Technion, Israel PODC 2006 Defining Properties Timeliness of incoming, outgoing and bidirectional links. Some known failure detector properties Use properties to clearly define models

30. Keidar & Shraer, Technion, Israel PODC 2006 Some Results: Context Consensus problem Global decision time metric –Time until all correct processes decide Message passing Crash failures –t 1 processes

31. Keidar & Shraer, Technion, Israel PODC 2006 ◊LM Model: Leader and Majority Nothing required before GSR In every round k ≥ GSR –Every correct process receives a round k message from a majority of processes, one of which is the Ω-leader. Practically requires much shorter timeouts than Eventual Synchrony [Bakr, Keidar]

32. Keidar & Shraer, Technion, Israel PODC 2006 ◊LM: Previous Work Most Ω-based algorithms wait for majority in each round (not ◊LM) Paxos [Lamport 98] works for ◊LM –Takes constant number of rounds in Eventual Synchrony (ES) –But how many rounds without ES?

33. Keidar & Shraer, Technion, Israel PODC 2006 Paxos Run in ES 21...... (Commit, 21,v 1 ) 21...... 20 21...... (“prepare”,21) yes decide v 1 (Commit, 21, v 1 ) Ω Leader BallotNum number of attempts to decide initiated by leaders 1 2 5 20...... no 5 20...... yes (“prepare”,2)

34. Keidar & Shraer, Technion, Israel PODC 2006 Paxos in ◊LM (w/out ES) 2 (“prepare”,2) 2 5 20 8 13 9 9 9 20 9 13 (“prepare”,9) (“prepare”,14) Ω Leader ok no (5) no (8) ok no (13) 1 5 20 8 13 GSRGSR+1GSR+2GSR+3 BallotNum Commit takes O(n) rounds!

35. Keidar & Shraer, Technion, Israel PODC 2006 What Can We Hope For? Tight lower bound for ES: 3 rounds from GSR [DGK05] ◊LM weaker than ES One might expect it to take a longer time in ◊LM than in ES

36. Keidar & Shraer, Technion, Israel PODC 2006 Result 1: Don't Need ES Leader and majority can give you the same performance! Algorithm that matches lower bound for ES!

37. Keidar & Shraer, Technion, Israel PODC 2006 Our ◊LM Algorithm in a Nutshell Commit with increasing ballot numbers, decide on value committed by majority –like Paxos, etc. Challenge: Don’t know all ballots, how to choose the new one to be highest one? Solution: Choose it to be the round number Challenge: rounds are wasted if a prepare/commit fails. Solution: pipeline prepares and commits: try in each round Challenge: do they really need to say no? Solution: support leader’s prepare even if have a higher ballot number –challenge: higher number may reflect later decision! Won’t agreement be compromised? –solution: new field “trustMe” ensures supported leader doesn't miss real decisions

38. Keidar & Shraer, Technion, Israel PODC 2006 Example Run: GSR=100 1 5 20 8 13 Ω Leader Rounds: GSR+1 GSR+2 101 8 8 20 13 GSR All PREPARE with ! trustMe All COMMIT 101 All DECIDE Did not lead to decision

39. Keidar & Shraer, Technion, Israel PODC 2006 Question 2: ◊S and Ω Equivalent? ◊S and Ω equivalent in the “classical” sense [Chandra, Hadzilacos, Toueg 96] –Weakest for consensus ◊S: eventually (from GSR onward), –all faulty processes are suspected by every correct process –there exists one correct process that is not suspected by any correct process. Can we substitute Ω with ◊S in ◊LM?

40. Keidar & Shraer, Technion, Israel PODC 2006 Result 2: ◊S and Ω not that Equivalent Consensus takes linear time from GSR By reduction to mobile failure model [Santoro, Widmayer 89]

41. Keidar & Shraer, Technion, Israel PODC 2006 Result 3: Do We Need Oracles? Timely communication with majority suffices! ◊AFM (All-From-Majority) simplified: –In every round k ≥ GSR, every correct process p receives round k message from a majority of processes, and p’s message reaches a majority of processes. Decision in 5 rounds from GSR –1 st constant time algorithm w/out oracle or ES –idea: information passes to all nodes in 2 rounds

42. Keidar & Shraer, Technion, Israel PODC 2006 ◊MFM: Majority from Majority –The rest receive a message from a minority Only a little missing for ◊AFM Stronger than models in literature [Aguilera et al. 03, 04], [Malkhi et al. 05] Bounded time from GSR impossible! Result 4: Can We Assume Less?

43. Conclusions Which guarantees should one implement ? –weaker ≠ better some previously suggested assumptions are too weak –sometimes a little stronger = much better worth longer timeouts / better hardware –ES is not essential not worth longer timeouts / better hardware –future: more models, bounds to explore GIRAF