1. 1 Principles of Reliable Distributed Systems Lecture 5: Failure Models, Fault-Tolerant Broadcasts and State-Machine Replication Spring 2005 Dr. Idit Keidar

2. 2 Today’s Material Distributed Systems 2nd edition, Sape Mullender (Editor) –Failure models, from Ch. 2 –State machine replication, from Ch. 7 –Fault-tolerant broadcasts, from Ch. 5 –Vector clocks, Ch. 4 & Ch. 6 of Attiya-Welch

3. 3 Models of Process Failures Crash Receive OmissionSend Omission General Omission Timing Authenticated Byzantine Byzantine Benign Malicious

4. 4 Correct vs. Faulty Processes Look at a complete run (execution) –external observer’s view A process that does not fail in a run is correct in that run Otherwise, the process is faulty in the run –a process that fails any time in the run is faulty throughout the entire run

5. 5 Threshold Failure Model t out of n processes may fail t is usually given as a function of n, e.g., –t < n –2t < n –3t < n

6. 6 Models of Link Failures Failures can be attributed to links instead of processes Reliable links: –every message sent is eventually delivered Failure types: –Crash –Loss (omission) –Timing –Byzantine

7. 7 Synchronous vs. Asynchronous Synchrony: –bounded latency, clock drift, processing time –process crash failures can be accurately detected Asynchrony: non-assumption –process crash failures cannot be accurately detected Failstop –like asynchronous, but crash failures accurately detected Unreliable failure detectors – later in the course

8. 8 Why These Models? Abstractions we can work with Behavior we care about –we don’t care if loss is due to router or network card –we don’t care if Byzantine behavior is due to software bug, hardware bug, or intrusion What happens if we run the algorithm and failures not in the model occur?

9. 9 Providing Abstractions of Models In a synchronous system, we can implement failstop over crash If loss is bounded, we can use retransmission and abstract it away A process that detects that it is slow or incorrect can crash itself, simulating the crash model

10. 10 State Machine (Active) Replication Client sends updates to all servers All servers are identical deterministic state machines –fixed number of replicas –start in same initial state

11. 11 Fault-Tolerant State-Machine Replication How many replicas to mask –t crash failures? –t Byzantine failures?

12. 12 Replica Coordination Requirements Agreement: all correct replicas receive all client requests Order: replicas process requests in the same order

13. 13 Broadcast Service for Replication Primitives: broadcast(m), deliver(m). –For simplicity, assume m is unique. Network Broadcast Algorithm Application deliver broadcast receivesend Broadcast Algorithm Application deliver broadcast receivesend

14. 14 Reliable Broadcast Specification Validity: if a correct process broadcasts m then all correct processes eventually deliver m Agreement: if a correct process delivers m then all correct processes eventually deliver m –Uniform agreement: if any process delivers m then all correct processes eventually deliver m Integrity: m is delivered by a correct process at most once, and only if it was previously broadcast

15. 15 Reliable Broadcast Implementation Model: –asynchronous –process crash failures –reliable links between pairs of correct processes Simple implementation … Does it work with links failures? –under what condition? Does it solve Uniform Reliable Broadcast?

16. 16 Ordering Properties for Broadcasts FIFO: If some process broadcasts message m before message m’, then every correct process that delivers m’ delivers m beforehand. Causal: If message m causally precedes message m’, then every correct process that delivers m’ delivers m beforehand. –causal order: transitive closure of FIFO + some process delivers m before broadcasting m’ Total order: If two correct processes deliver both m and m’, they deliver them in the same order.

17. 17 FIFO Broadcast Reliable Broadcast + FIFO Simple implementation on top of reliable broadcast (using sequence numbers) Network Reliable Broadcast Application receivesend FIFO Broadcast deliverbroadcast deliverbroadcast Reliable Broadcast Application receivesend FIFO Broadcast deliverbroadcast deliver broadcast Reliable Broadcast FIFO Deliver

18. 18 Causal Broadcast Reliable Broadcast + Causal Implementation using Reliable Broadcast Reminder: last week we saw an implementation using logical (Lamport) timestamps (LTS) that enforces total order in addition to Causal Fault-tolerance?

19. 19 Vector Clocks Process pi has a vector clock VC[1…n] –VC[i] is the sequence number of the last message broadcast by pi –for j≠i, VC[j] is the sequence number of the last message pi delivered from pj VC is sent with each message m –for j≠i, m.VC[j] is pj’s latest message that causally precedes m

20. 20 Vector Clocks: Sending At process pi, on broadcast(m) –VC[i] := VC[i]+1 –use reliable broadcast to send m with VC to all –deliver m locally

21. 21 Vector Clocks: Delivery Rule Upon receive m –place in message buffer Deliver m from pj from buffer if –VC[j] = m.VC[j] -1 –forall k≠j : VC[k] ≥ m.VC[k] Upon deliver –VC[j] := VC[j] + 1 VC[j] is the number of messages of pj that causally precede pi’s subsequent messages FIFO

22. 22 Vector Clocks Example

23. 23 Fault-Tolerance? X X

24. 24 Atomic Broadcast Services Atomic Broadcast: – Reliable Broadcast + Total Order FIFO Atomic Broadcast –FIFO + Reliable Broadcast + Total Order Causal Atomic Broadcast –Causal + Reliable Broadcast + Total Order HW question: are FIFO Atomic Broadcast and Causal Atomic Broadcast equivalent?

25. 25 Causal Atomic Broadcast in Failstop Model Order messages by logical timestamp (LTS) –Break ties by process id Use FIFO Broadcast When is a message with LTS t delivered? –Reminder: failstop = failures are accurately detected –Assume further that no message from a faulty process arrives after its failure is detected

26. 26 Example Exam Question 0 0 0  0,2  1 1  1,1   1,3  2 2 3  2,2  3 1 2 2 2

27. 27 Atomic Broadcast in Asynchronous Systems??? Alas, impossible if even one process can crash

28. 28 The Consensus Problem Each process has an input, should decide an output Agreement: correct processes’ decisions are the same Validity: decision is input of one process Termination: eventually all correct processes decide

29. 29 Consensus versus Atomic Broadcast From Atomic Broadcast to Consensus From Consensus to Atomic Broadcast –Homework question From now on, we will focus mainly on consensus, and keep in mind that it suffices for Atomic Broadcast

30. 30 Other Models Asynchronous –Consensus, and hence Atomic Broadcast impossible even with one crash failure –Reliable Broadcast possible Synchronous with Byzantine failures –Consensus, and hence Atomic Broadcast possible with up to t = n-1 failures, assuming authentication Middle ground models will be discussed in future lessons

31. 31 Now, Back to State Machines We can build state machines using Atomic Broadcast A client can forward its request to one of the servers to broadcast to the others –resend on timeout if the server fails –more than one for Byzantine failures What about client failures? –Crash? Byzantine?

32. 32 Multicast Problems Processes organized in groups –groups have names –messages sent to groups –like broadcast, but for group members –processes can join and leave groups –more in upcoming recitations! Processes may care about who else is a member of the group (group membership)