CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 15: Broadcast 1

Broadcast Specifications CSCE 668Set 15: Broadcast 2  Recall the specification of a broadcast service given in the last set of slides:  Inputs: bc-send i (m)  an input to the broadcast service  p i wants to use the broadcast service to send m to all the procs  Outputs: bc-recv i (m,j)  an output of the broadcast service  broadcast service is delivering msg m, sent by p j, to p i

Broadcast Specifications CSCE 668Set 15: Broadcast 3  A sequence of inputs and outputs (bc-sends and bc- recvs) is allowable iff there exists a mapping  from each bc-recv i (m,j) event to an earlier bc-send j (m) event s.t.   is well-defined: every msg bc-recv'ed was previously bc- sent (Integrity)   restricted to bc-recv i events, for each i, is one-to-one: no msg is bc-recv'ed more than once at any single proc. (No Duplicates)   restricted to bc-recv i events, for each i, is onto: every msg bc-sent is received at every proc. (Liveness)

Ordering Properties CSCE 668Set 15: Broadcast 4  Sometimes we might want a broadcast service that also provides some kind of guarantee on the order in which messages are delivered.  We can add additional constraints on the mapping  :  single-source FIFO or  totally ordered or  causally ordered

Single-Source FIFO Ordering CSCE 668Set 15: Broadcast 5  For all messages m 1 and m 2 and all p i and p j, if p i sends m 1 before it sends m 2, and if p j receives m 1 and m 2, then p j receives m 1 before it receives m 2.  Phrased carefully to avoid requiring that both messages are received.  that is the responsibility of a liveness property

Totally Ordered CSCE 668Set 15: Broadcast 6  For all messages m 1 and m 2 and all p i and p j, if both p i and p j receive both messages, then they receive them in the same order.  Phrased carefully to avoid requiring that both messages are received by both procs.  that is the responsibility of a liveness property

Happens Before for Broadcast Messages CSCE 668Set 15: Broadcast 7  Earlier we defined "happens before" relation for events.  Now extend this definition to broadcast messages.  Assume all communication is through broadcast sends and receives.  Msg m 1 happens before msg m 2 if  some bc-recv event for m 1 happens before (in the old sense) the bc-send event for m 2, or  m 1 and m 2 are bc-sent by the same proc. and m 1 is bc-sent before m 2 is bc-sent.

Example of Happens Before for Broadcast Messages CSCE 668Set 15: Broadcast 8 m1m1 m2m2 m3m3 m4m4 m 1 happens before m 3 and m 4 m 2 happens before m 4 m 3 happens before m 4

Causally Ordered CSCE 668Set 15: Broadcast 9  For all messages m 1 and m 2 and all p i, if m 1 happens before m 2, and if p i receives both m 1 and m 2, then p i receives m 1 before it receives m 2.  Phrased carefully to avoid requiring that both messages are received.  that is the responsibility of a liveness property

Example CSCE 668Set 15: Broadcast 10 a b single-source FIFO? totally ordered? causally ordered?

Example CSCE 668Set 15: Broadcast 11 ab single-source FIFO? totally ordered? causally ordered?

Example CSCE 668Set 15: Broadcast 12 a b single-source FIFO? totally ordered? causally ordered?

Algorithm BB to Simulate Basic Broadcast on Top of Point-to-Point CSCE 668Set 15: Broadcast 13  When bc-send i (m) occurs:  p i sends a separate copy of m to every processor (including itself) using the underlying point-to-point message passing communication system  When can p i perform bc-recv i (m)?  when it receives m from the underlying point-to- point message passing communication system

Basic Broadcast Simulation CSCE 668Set 15: Broadcast 14 … Alg BB BB 0 bc-send i bc-recv i send i recv i asynch pt-to-pt message passing BB n-1 bc-send j bc-recv j send j recv j basic broadcast

Correctness of Basic Broadcast Algorithm CSCE 668Set 15: Broadcast 15  Assume the underlying point-to-point message passing system is correct (i.e., conforms to the spec given in previous set of slides).  Check that the simulated broadcast service satisfies:  Integrity  No Duplicates  Liveness

Single-Source FIFO Algorithm CSCE 668Set 15: Broadcast 16  Assume the underlying communication system is basic broadcast.  when ssf-bc-send i (m) occurs:  p i uses the underlying basic broadcast service to bcast m together with a sequence number  p i increments sequence number by 1 each time it initiates a bcast  when can p i perform ssf-bc-recv i (m)?  when p i has bc-recv'ed m with sequence number T and has ssf-bc-recv'ed messages from p j (the ssf-bc-sender of m) with all smaller sequence numbers

Single-Source FIFO Algorithm CSCE 668Set 15: Broadcast 17 SSF alg (timestamps) basic bcast alg (n copies) point-to-point message passing user of SSF bcast ssf-bc-sendssf-bc-recv bc-send send bc-recv recv basic bcast ssf bcast

Asymmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 18  Assume underlying communication service is basic broadcast.  There is a distinguished proc. p c  when to-bcast i (m) occurs:  p i sends m to p c (either assume the basic broadcast service also has a point-to-point mechanism, or have recipients other than p c ignore the msg)  when p c receives m from p i from the basic broadcast service:  append a sequence number to m and bc-send it

Asymmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 19  when can p i perform to-bc-recv(m)?  when p i has bc-recv'ed m with sequence number T and has to-bc-recv'ed messages with all smaller sequence numbers

Asymmetric Algorithm Discussion CSCE 668Set 15: Broadcast 20  Simple  Only requires basic broadcast  But p c is a bottleneck  Alternative approach next…

Symmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 21  Assume the underlying communication service is single-source FIFO broadcast.  Each proc. tags each msg it sends with a timestamp (increasing).  Break ties using proc. ids.  Each proc. keeps a vector of estimates of the other proc's timestamps:  If p i 's estimate for p j is k, then p i will not receive any later msg from p j with timestamp k.  Estimates are updated based on msgs received and "timestamp update" msgs

Symmetric Algorithm for Totally Ordered Broadcast CSCE 668Set 15: Broadcast 22  Each proc. keeps its timestamp to be ≥ all its estimates:  when p i has to increase its timestamp because of the receipt of a message, it sends a timestamp update msg  A proc. can deliver a msg with timestamp T once every entry in the proc's vector of estimates is at least T.

Symmetric Algorithm CSCE 668Set 15: Broadcast 23 when to-bc-send i (m) occurs: ts[i]++ add (m,ts[i],i) to pending invoke ssf-bc-send i ((m,ts[i])) when ssf-bc-recv i ((m,T)) from p j occurs: ts[j] := T add (m,T,j) to pending if T > ts[i] then ts[i] := T invoke ssf-bc-send i ("ts-up",T) when ssf-bc-recv i ("ts-up",T) from p j occurs: ts[j] := T invoke to-bc-recv i (m,j) when: (m,T,j) is entry in pending with smallest (T,j) T ≤ ts[k] for all k result: remove (m,T,j) from pending

CSCE 668Set 15: Broadcast 24 SSF alg (timestamps) basic bcast alg (n copies) point-to-point message passing symmetric TO alg ssf-bc-sendssf-bc-recv bc-send send bc-recv recv basic bcast user of TO bcast to-bc-sendto-bc-recv ssf bcast TO bcast

Correctness of Symmetric Algorithm CSCE 668Set 15: Broadcast 25 Lemma (8.2): Timestamps assigned to msgs form a total order (break ties with id of sender). Theorem (8.3): Symmetric algorithm simulates totally ordered broadcast service. Proof: Must show top-level outputs of symmetric algorithm satisfy 4 properties, in every admissible execution (relies on underlying ssf-bcast service being correct).

Correctness of Symmetric Alg. CSCE 668Set 15: Broadcast 26 Integrity: follows from same property for ssf-bcast. No Duplicates: follows from same property for ssf-bcast. Liveness:  Suppose in contradiction some p i has some entry (m,T, j ) stuck in its pending set forever, where (T, j ) is the smallest timestamp of all stuck entries.  Eventually (m,T, j ) has the smallest timestamp of all entries in p i 's pending set.  Why is (m,T, j ) stuck at p i ? Because p i 's estimate of some p k 's timestamp is stuck at some value T' < T.  But that would mean either p k never receives (m,T, j ) or p k 's timestamp-update msg resulting from p k receiving (m,T, j ) is never received at p i, contradicting correctness of the SSF broadcast.

Correctness of Symmetric Alg. CSCE 668Set 15: Broadcast 27 Total Ordering: Suppose p i invokes to-bc-recv for msg m with timestamp (T, j ), and later it invokes to-bc-recv for msg m' with timestamp (T', j '). Show (T, j ) < (T', j ').  By the code, if (m',T', j ') is in p i 's pending set when p i invokes the to-bc-recv for m, then (T, j ) < (T', j ').  Suppose (m',T', j ') is not yet in p i 's pending set at that time.  When p i invokes the to-bc-recv for m, precondition ensures that T ≤ ts[ j ']. So p i has received a msg from p j ' with timestamp ≥ T.  By the SSF property, every subsequent msg p i receives from p j ' will have timestamp > T, so T' must be > T.

Causal Ordering Algorithms CSCE 668Set 15: Broadcast 28  The symmetric total ordering algorithm ensures causal ordering:  timestamp order extends the happens-before order on messages.  Causal ordering can also be attained without the overhead of total ordering, by using an algorithm based on vector clocks…

Causal Order Algorithm CSCE 668Set 15: Broadcast 29 when co-bc-send i (m) occurs: vt[i]++ invoke co-bc-recv i (m) invoke bc-send i ((m,vt)) when bc-recv i ((m,w)) from p j occurs: add (m,w,j) to pending invoke co-bc-recv i (m,j) when: (m,w,j) is in pending w[j] = vt[j] + 1 w[k] ≤ vt[k] for all k ≠ j result: remove (m,w,j) from pending vt[j]++ Note: vt[j] records how many msgs from p j have been co-bc-recv'ed by p i Code for p i :

Causal Order Algorithm Discussion CSCE 668Set 15: Broadcast 30  Vector clocks are implemented slightly differently than in the point-to-point case.  In point-to-point case, we exploited indirect (transitive) information about messages received by other procs.  In the broadcast case, we don't need to do that, since very proc will eventually receive every message directly.

Causal Order Algorithm Example CSCE 668Set 15: Broadcast 31  Algorithm delays the delivery of the C.O. msgs until causal order property won't be violated. (0,1,0)(0,2,0) (0,3,0) (1,3,0)

Correctness of Causal Order Algorithm (Sketch) CSCE 668Set 15: Broadcast 32 Lemma (8.6): The local array variables vt serve as vector clocks. Theorem (8.7): The algorithm simulates causally ordered broadcast, if the underlying communication system satisfies (basic) broadcast. Proof: Integrity and No Duplicates follow from the same properties of the basic broadcast. Liveness requires some arguing. Causal Ordering follows from the lemma.

Reliable Broadcast CSCE 668Set 15: Broadcast 33  What do we require of a broadcast service when some of the procs can be faulty?  Specifications differ from those of the corresponding non-fault-tolerant specs in two ways: 1. proc indices are partitioned into "faulty" and "nonfaulty" 2. Liveness property is modified…

Reliable Broadcast Specification CSCE 668Set 15: Broadcast 34  Nonfaulty Liveness: Every msg bc-sent by a nonfaulty proc is eventually bc-recv'ed by all nonfaulty procs.  Faulty Liveness: Every msg bc-sent by a faulty proc is bc-recv'ed by either all the nonfaulty procs or none of them.

Discussion of Reliable Bcast Spec CSCE 668Set 15: Broadcast 35  Specification is independent of any particular fault model.  We will only consider implementations for crash faults.  No guarantee is given concerning which messages are received by faulty procs.  Can extend this spec to the various ordering variants:  msgs that are received by nonfaulty procs must conform to the relevant ordering property.

Spec of Failure-Prone Point-to-Point Message Passing System CSCE 668Set 15: Broadcast 36  Before we can design an algorithm to implement reliable (i.e., fault-tolerant) broadcast, we need to know what we can rely on from the lower layer communication system.  Modify the previous point-to-point spec from the no-fault case in two ways: 1. partition proc indices into "faulty" and "nonfaulty" 2. Liveness property is modified…

Spec of Failure-Prone Point-to-Point Message Passing System CSCE 668Set 15: Broadcast 37  Nonfaulty Liveness: every msg sent by a nonfaulty proc to any nonfaulty proc is eventually received. Note that this places no constraints on the eventual delivery of messages to faulty procs.

Reliable Broadcast Algorithm CSCE 668Set 15: Broadcast 38 when rel-bc-send i (m) occurs: invoke send i (m) to all procs when recv i (m) from p j occurs: if m has not already been recv'ed then invoke send i (m) to all procs invoke rel-bc-recv i (m)

Correctness of Reliable Bcast Alg CSCE 668Set 15: Broadcast 39  Integrity: follows from Integrity property of underlying point-to-point msg system.  No Duplicates: follows from No Duplicates property of underlying point-to-point msg system and the check that this msg was not already received.  Nonfaulty Liveness: follows from Nonfaulty Liveness property of underlying point-to-point msg system.  Faulty Liveness: follows from relaying and underlying Nonfaulty Liveness.