1. CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 15: Broadcast 1

2. Broadcast Specifications CSCE 668Set 15: Broadcast 2  Recall the specification of a broadcast service given in the last set of slides:  Inputs: bc-send i (m)  an input to the broadcast service  p i wants to use the broadcast service to send m to all the procs  Outputs: bc-recv i (m,j)  an output of the broadcast service  broadcast service is delivering msg m, sent by p j, to p i

3. Broadcast Specifications CSCE 668Set 15: Broadcast 3  A sequence of inputs and outputs (bc-sends and bc- recvs) is allowable iff there exists a mapping  from each bc-recv i (m,j) event to an earlier bc-send j (m) event s.t.   is well-defined: every msg bc-recv'ed was previously bc- sent (Integrity)   restricted to bc-recv i events, for each i, is one-to-one: no msg is bc-recv'ed more than once at any single proc. (No Duplicates)   restricted to bc-recv i events, for each i, is onto: every msg bc-sent is received at every proc. (Liveness)

4. Ordering Properties CSCE 668Set 15: Broadcast 4  Sometimes we might want a broadcast service that also provides some kind of guarantee on the order in which messages are delivered.  We can add additional constraints on the mapping  :  single-source FIFO or  totally ordered or  causally ordered

5. Single-Source FIFO Ordering CSCE 668Set 15: Broadcast 5  For all messages m 1 and m 2 and all p i and p j, if p i sends m 1 before it sends m 2, and if p j receives m 1 and m 2, then p j receives m 1 before it receives m 2.  Phrased carefully to avoid requiring that both messages are received.  that is the responsibility of a liveness property

6. Totally Ordered CSCE 668Set 15: Broadcast 6  For all messages m 1 and m 2 and all p i and p j, if both p i and p j receive both messages, then they receive them in the same order.  Phrased carefully to avoid requiring that both messages are received by both procs.  that is the responsibility of a liveness property

7. Happens Before for Broadcast Messages CSCE 668Set 15: Broadcast 7  Earlier we defined "happens before" relation for events.  Now extend this definition to broadcast messages.  Assume all communication is through broadcast sends and receives.  Msg m 1 happens before msg m 2 if  some bc-recv event for m 1 happens before (in the old sense) the bc-send event for m 2, or  m 1 and m 2 are bc-sent by the same proc. and m 1 is bc-sent before m 2 is bc-sent.

8. Example of Happens Before for Broadcast Messages CSCE 668Set 15: Broadcast 8 m1m1 m2m2 m3m3 m4m4 m 1 happens before m 3 and m 4 m 2 happens before m 4 m 3 happens before m 4

9. Causally Ordered CSCE 668Set 15: Broadcast 9  For all messages m 1 and m 2 and all p i, if m 1 happens before m 2, and if p i receives both m 1 and m 2, then p i receives m 1 before it receives m 2.  Phrased carefully to avoid requiring that both messages are received.  that is the responsibility of a liveness property

10. Example CSCE 668Set 15: Broadcast 10 a b single-source FIFO? totally ordered? causally ordered?

11. Example CSCE 668Set 15: Broadcast 11 ab single-source FIFO? totally ordered? causally ordered?

12. Example CSCE 668Set 15: Broadcast 12 a b single-source FIFO? totally ordered? causally ordered?

13. Algorithm BB to Simulate Basic Broadcast on Top of Point-to-Point CSCE 668Set 15: Broadcast 13  When bc-send i (m) occurs:  p i sends a separate copy of m to every processor (including itself) using the underlying point-to-point message passing communication system  When can p i perform bc-recv i (m)?  when it receives m from the underlying point-to- point message passing communication system

14. Basic Broadcast Simulation CSCE 668Set 15: Broadcast 14 … Alg BB BB 0 bc-send i bc-recv i send i recv i asynch pt-to-pt message passing BB n-1 bc-send j bc-recv j send j recv j basic broadcast

15. Correctness of Basic Broadcast Algorithm CSCE 668Set 15: Broadcast 15  Assume the underlying point-to-point message passing system is correct (i.e., conforms to the spec given in previous set of slides).  Check that the simulated broadcast service satisfies:  Integrity  No Duplicates  Liveness

16. Single-Source FIFO Algorithm CSCE 668Set 15: Broadcast 16  Assume the underlying communication system is basic broadcast.  when ssf-bc-send i (m) occurs:  p i uses the underlying basic broadcast service to bcast m together with a sequence number  p i increments sequence number by 1 each time it initiates a bcast  when can p i perform ssf-bc-recv i (m)?  when p i has bc-recv'ed m with sequence number T and has ssf-bc-recv'ed messages from p j (the ssf-bc-sender of m) with all smaller sequence numbers

17. Single-Source FIFO Algorithm CSCE 668Set 15: Broadcast 17 SSF alg (timestamps) basic bcast alg (n copies) point-to-point message passing user of SSF bcast ssf-bc-sendssf-bc-recv bc-send send bc-recv recv basic bcast ssf bcast

18. Asymmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 18  Assume underlying communication service is basic broadcast.  There is a distinguished proc. p c  when to-bcast i (m) occurs:  p i sends m to p c (either assume the basic broadcast service also has a point-to-point mechanism, or have recipients other than p c ignore the msg)  when p c receives m from p i from the basic broadcast service:  append a sequence number to m and bc-send it

19. Asymmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 19  when can p i perform to-bc-recv(m)?  when p i has bc-recv'ed m with sequence number T and has to-bc-recv'ed messages with all smaller sequence numbers

20. Asymmetric Algorithm Discussion CSCE 668Set 15: Broadcast 20  Simple  Only requires basic broadcast  But p c is a bottleneck  Alternative approach next…

21. Symmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 21  Assume the underlying communication service is single-source FIFO broadcast.  Each proc. tags each msg it sends with a timestamp (increasing).  Break ties using proc. ids.  Each proc. keeps a vector of estimates of the other proc's timestamps:  If p i 's estimate for p j is k, then p i will not receive any later msg from p j with timestamp k.  Estimates are updated based on msgs received and "timestamp update" msgs

22. Symmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 22  Each proc. keeps its timestamp to be ≥ all its estimates:  when p i has to increase its timestamp because of the receipt of a message, it sends a timestamp update msg  A proc. can deliver a msg with timestamp T once every entry in the proc's vector of estimates is at least T.

23. Symmetric Algorithm CSCE 668Set 15: Broadcast 23 when to-bc-send i (m) occurs: ts[i]++ add (m,ts[i],i) to pending invoke ssf-bc-send i ((m,ts[i])) when ssf-bc-recv i ((m,T)) from p j occurs: ts[j] := T add (m,T,j) to pending if T > ts[i] then ts[i] := T invoke ssf-bc-send i ("ts-up",T) when ssf-bc-recv i ("ts-up",T) from p j occurs: ts[j] := T invoke to-bc-recv i (m,j) when: (m,T,j) is entry in pending with smallest (T,j) T ≤ ts[k] for all k result: remove (m,T,j) from pending

24. CSCE 668Set 15: Broadcast 24 SSF alg (timestamps) basic bcast alg (n copies) point-to-point message passing symmetric TO alg ssf-bc-sendssf-bc-recv bc-send send bc-recv recv basic bcast user of TO bcast to-bc-sendto-bc-recv ssf bcast TO bcast

25. Correctness of Symmetric Algorithm CSCE 668Set 15: Broadcast 25 Lemma (8.2): Timestamps assigned to msgs form a total order (break ties with id of sender). Theorem (8.3): Symmetric algorithm simulates totally ordered broadcast service. Proof: Must show top-level outputs of symmetric algorithm satisfy 4 properties, in every admissible execution (relies on underlying ssf-bcast service being correct).

26. Correctness of Symmetric Alg. CSCE 668Set 15: Broadcast 26 Integrity: follows from same property for ssf-bcast. No Duplicates: follows from same property for ssf-bcast. Liveness:  Suppose in contradiction some p i has some entry (m,T, j ) stuck in its pending set forever, where (T, j ) is the smallest timestamp of all stuck entries.  Eventually (m,T, j ) has the smallest timestamp of all entries in p i 's pending set.  Why is (m,T, j ) stuck at p i ? Because p i 's estimate of some p k 's timestamp is stuck at some value T' < T.  But that would mean either p k never receives (m,T, j ) or p k 's timestamp-update msg resulting from p k receiving (m,T, j ) is never received at p i, contradicting correctness of the SSF broadcast.

27. Correctness of Symmetric Alg. CSCE 668Set 15: Broadcast 27 Total Ordering: Suppose p i invokes to-bc-recv for msg m with timestamp (T, j ), and later it invokes to-bc-recv for msg m' with timestamp (T', j '). Show (T, j ) < (T', j ').  By the code, if (m',T', j ') is in p i 's pending set when p i invokes the to-bc-recv for m, then (T, j ) < (T', j ').  Suppose (m',T', j ') is not yet in p i 's pending set at that time.  When p i invokes the to-bc-recv for m, precondition ensures that T ≤ ts[ j ']. So p i has received a msg from p j ' with timestamp ≥ T.  By the SSF property, every subsequent msg p i receives from p j ' will have timestamp > T, so T' must be > T.

28. Causal Ordering Algorithms CSCE 668Set 15: Broadcast 28  The symmetric total ordering algorithm ensures causal ordering:  timestamp order extends the happens-before order on messages.  Causal ordering can also be attained without the overhead of total ordering, by using an algorithm based on vector clocks…

29. Causal Order Algorithm CSCE 668Set 15: Broadcast 29 when co-bc-send i (m) occurs: vt[i]++ invoke co-bc-recv i (m) invoke bc-send i ((m,vt)) when bc-recv i ((m,w)) from p j occurs: add (m,w,j) to pending invoke co-bc-recv i (m,j) when: (m,w,j) is in pending w[j] = vt[j] + 1 w[k] ≤ vt[k] for all k ≠ j result: remove (m,w,j) from pending vt[j]++ Note: vt[j] records how many msgs from p j have been co-bc-recv'ed by p i Code for p i :

30. Causal Order Algorithm Discussion CSCE 668Set 15: Broadcast 30  Vector clocks are implemented slightly differently than in the point-to-point case.  In point-to-point case, we exploited indirect (transitive) information about messages received by other procs.  In the broadcast case, we don't need to do that, since very proc will eventually receive every message directly.

31. Causal Order Algorithm Example CSCE 668Set 15: Broadcast 31  Algorithm delays the delivery of the C.O. msgs until causal order property won't be violated. (0,1,0)(0,2,0) (0,3,0) (1,3,0)

32. Correctness of Causal Order Algorithm (Sketch) CSCE 668Set 15: Broadcast 32 Lemma (8.6): The local array variables vt serve as vector clocks. Theorem (8.7): The algorithm simulates causally ordered broadcast, if the underlying communication system satisfies (basic) broadcast. Proof: Integrity and No Duplicates follow from the same properties of the basic broadcast. Liveness requires some arguing. Causal Ordering follows from the lemma.

33. Reliable Broadcast CSCE 668Set 15: Broadcast 33  What do we require of a broadcast service when some of the procs can be faulty?  Specifications differ from those of the corresponding non-fault-tolerant specs in two ways: 1. proc indices are partitioned into "faulty" and "nonfaulty" 2. Liveness property is modified…

34. Reliable Broadcast Specification CSCE 668Set 15: Broadcast 34  Nonfaulty Liveness: Every msg bc-sent by a nonfaulty proc is eventually bc-recv'ed by all nonfaulty procs.  Faulty Liveness: Every msg bc-sent by a faulty proc is bc-recv'ed by either all the nonfaulty procs or none of them.

35. Discussion of Reliable Bcast Spec CSCE 668Set 15: Broadcast 35  Specification is independent of any particular fault model.  We will only consider implementations for crash faults.  No guarantee is given concerning which messages are received by faulty procs.  Can extend this spec to the various ordering variants:  msgs that are received by nonfaulty procs must conform to the relevant ordering property.

36. Spec of Failure-Prone Point-to-Point Message Passing System CSCE 668Set 15: Broadcast 36  Before we can design an algorithm to implement reliable (i.e., fault-tolerant) broadcast, we need to know what we can rely on from the lower layer communication system.  Modify the previous point-to-point spec from the no-fault case in two ways: 1. partition proc indices into "faulty" and "nonfaulty" 2. Liveness property is modified…

37. Spec of Failure-Prone Point-to-Point Message Passing System CSCE 668Set 15: Broadcast 37  Nonfaulty Liveness: every msg sent by a nonfaulty proc to any nonfaulty proc is eventually received. Note that this places no constraints on the eventual delivery of messages to faulty procs.

38. Reliable Broadcast Algorithm CSCE 668Set 15: Broadcast 38 when rel-bc-send i (m) occurs: invoke send i (m) to all procs when recv i (m) from p j occurs: if m has not already been recv'ed then invoke send i (m) to all procs invoke rel-bc-recv i (m)

39. Correctness of Reliable Bcast Alg CSCE 668Set 15: Broadcast 39  Integrity: follows from Integrity property of underlying point-to-point msg system.  No Duplicates: follows from No Duplicates property of underlying point-to-point msg system and the check that this msg was not already received.  Nonfaulty Liveness: follows from Nonfaulty Liveness property of underlying point-to-point msg system.  Faulty Liveness: follows from relaying and underlying Nonfaulty Liveness.