1. Ordered Communication

2. Define guarantees about the order of deliveries inside group of processes Type of ordering: Deliveries respect the FIFO ordering of the corresponding sendings Deliveries respect the Causal ordering of the corresponding sendings Delivery respects a total ordering of deliveries (atomic communication)

3. Advantages of ordered communication Orthogonality wrt reliable communication. Reliable broadcast does not have any property on ordering deliveries of messages This can cause anomalies in many applicative contexts “Reliable ordered communication” are obtained adding one or more ordering properties to reliable communication Example: flight booking system. Consider the message pattern depicted in the figure. The server cansel a reservation that has never been done! t client server “reserve”“cancel” “Prices 15% off”

4. FIFO Broadcast\ specification Messages sent by the a process has to be delivered in the sending order. Safety FIFO Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): FIFO Order: if a process sends a broadcast message m before m’, then no correct process delivers m’ if it has not already delivered m. FIFO order can be uniform/non uniform FIFO Broadcast = Reliable Broadcast + FIFO Order

5. Each process q holds: S p a count of messages broadcast by p R p the sequence number of the latest message sent by p and delivered by q For p to FO-multicast a message to g, it piggybacks S p on the message, rbBroadcasts it and increments S p by 1 On receipt of a message from q sent by p with sequence number S, p checks whether S = R p + 1. If so, q FO-delivers it if S > R p + 1 then q places message in hold-back queue until intervening messages have been delivered. (note that rbBroadcast does eventually deliver messages unless the sender crashes) FIFO Broadcast\algorithm

6. Advantages of Ordered Communication (2) FIFO does not prevent all the anomalies due to order of deliveries. Es: newsgroup. Even though the computation satisfies FIFO order, student 2 can be disoriented by the order in which it receives messages m1 and m2. m1 causally precede m2, then m2 can be delivered only if m1 is delivered. Causal Broadcast Prof. Student 2 m 1 : “Fri exam cancelled” Student 1 m 2 : “let’s party on Thu night” m 3 : “but we have an exam on Fri!”

7. Causal Order  FIFO Order, But FIFO Order  Causal Order thus, Causal Order = FIFO Order + ? Causal Broadcast\specification Safety Causal Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): Causal Order: if the sending of a message m causally precedes the sending of a message m’, then every correct process has to deliver m before delivering m’. Causal Broadcast = Reliable Broadcast+Causal Order

8. Causal Broadcast\specification Causal Order = FIFO Order + Local Order. Local Order : if a process delivers a message m before sending a msg m’, then no correct process deliver m’ if it has not already delivered m. Example: p q r t m m’ The delivery of m’ is delaied till the arrival and the delivery of m

9. Causal Broadcast\ implementations Two implementations blocking algorthm using vector clocks (already discussed) non-blocking algorithm using piggybacking of causal past p1p1 p2p2 p3p3 COBcast(m 1 ) CObcast(m 2 ) COBcast(m 3 ) m1m1 m2m2  m 1,m 2, m 3  COdelv(m 1 ) COdelv(m 3 ) m 2 già COdelivered! COdelv(m 1 ) COdelv(m 2 ) COdelv(m 3 ) COdelv(m 2 ) Filter out

10. Advantages of Ordered Communication (3) Causal Order is not enough strong to avoid anomalies Es. banking. Bank account replicated on two sites R1R1 R2R2 A:£100 Deposit £20 Add 10% interest A:£120 A:£110 A:£132 A:£130 Despite the fact that replicas initially share the same state, the state reaches a different value in the two sites at the end of the exexution as shown in Figure. Note that the computation is Causally Ordered To guarantee values of replicas be the same, one has to ensure that the order of delivery be the same at each process. In the above example R1 delivers m1 before m2 and R2 delivers m2 before m1. Note that ensuring the same delivery order at each replicas does not look at the sending order of messages m1 m2

11. Atomic Broadcast\specification Safety Atomic (Total) Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of total order (example): Total Order: if two correct processes p and q deliver m and m’, then p delivers m before m’ if, and only if, q delivers m before m’ Total order is orthogonal with respect to FIFO and Causal Order. Total order would accept indeed a computation in which a process sends n messages to a group, and each of the processes of the group delivers such messages in the reverse order of their sendings. The computation is totally ordered but it is not FIFO.

12. Causal Atomic broadcast Hierarchy of Broadcast Specifications Reliable broadcast FIFO broadcast Causal broadcast FIFO Atomic broadcast Atomic broadcast Total Order Causal Order Total Order FIFO Order Local Order Causal Order Local Order

13. System model Static set of processes Π = {p 1 … p n } Message passing over perfect channels (message exchanging between correct processes is reliable) Asynchronous Crash fault model for processes We characterize the system in terms of its possible runs R R p1p1 p2p2 pnpn TOcast(m) m m m TOdeliver(m) crash r

14. A few notation Property P: predicate on the system, identifying a set of runs R P  R P  P’ iff R P  R P’ Specification S(P 1,…,P m ): logical and of m properties, identifying a set of runs R S =R P 1 ∩ … ∩ R P m  R S → S’ iff R S  R S’ RPRP R P’ RSRS R S’ R P1 R Pn RSRS R R R

15. TO specifications Total order specifications are usually composed by four properties, namely Validity, Integrity,Agreement, and Order. A Validity property guarantees that messages sent by correct processes will eventually be delivered at least by correct processes; An Integrity property guarantees that no spurious or duplicate messages are delivered; An Agreement property ensures that (at least correct) processes deliver the same set of messages; An Order property constrains (at least correct) processes delivering the same messages to deliver them in the same order.

16. TO specifications Total Order Broadcast = S(V,I,A,O) V = Validity I = Integrity A = Agreement O = Order Distinct specifications arise from distinct formulations of each property uniform vs non-uniform A uniform property imposes restrictions on the behavior of (at least) correct processes on the basis of events occurred in some process NUV UI TO(A,O)

17. TO Specifications Crash failure + Perfect channels  NUV. if a correct process TOCAST a message m then some correct process will eventually deliver m UI. For any message m, every process p delivers m at most once and only if m was previously tocast by some (correct or not) process.

18. The Agreement property (Uniform Agreement, UA) If a process (correct or not) todelivers a message m, then all correct processes will eventually todeliver m; (Non-uniform Agreement, NUA) If a correct process todelivers a message m, then all correct processes will eventually todeliver m

19. The Agreement property Constrains the set of delivered messages Correct processes always deliver the same set of messages M Each faulty process p delivers a set M p UA: M p  M NUA: M p can be s.t. M p - M ≠  m2m2 m4m4 p1p1 p2p2 p3p3 m2m2 m4m4 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m1m1 m2m2 UA m4m4 p1p1 p2p2 p3p3 m2m2 m4m4 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m1m1 m2m2 m5m5 NUA

20. The Order property Constrains the order of message deliveries and possibly the set of delivered messages SUTO: if p delivers m<m’, q delivers m’ only after m same order same prefix of the set of delivered messages after an omission, disjoint sets of delivered messages WUTO: if p,q deliver m,m’, they get the same order no restrictions on the set of delivered messages p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 SUTO WUTO

21. The Order property (2) SUTO and WUTO are uniform They both have a non-uniform counterparts: SNUTO and WNUTO (Strong Non-uniform Total Order, SNUTO). If some correct process todelivers some message m before message m', then a correct process todelivers m‘ only after it has todelivered m. (Weak Non-uniform Total Order, WNUTO) If correct processes p and q both todeliver messages m and m', then p todelivers m before m' if and only if q todelivers m before m‘

22. The Order property (2) SUTO  WUTO SNUTO  WNUTO p1p1 p2p2 p3p3 m1m1 m2m2 m2m2 m1m1 m1m1 m2m2 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 SNUTO p1p1 p2p2 p3p3 m1m1 m2m2 m1m1 m1m1 m2m2 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 WNUTO m2m2

23. TO specifications TO(UA,SUTO) The strongest TO spec. p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m6m6 m6m6 p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m6m6 m6m6 m5m5 TO(NUA,SUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO)

24. TO specifications (2) TO(UA,WUTO) m3m3 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m4m4 m4m4 m4m4 m3m3 p1p1 p2p2 p3p3 m2m2 m1m1 m1m1 m1m1 m3m3 m4m4 m4m4 m2m2 m3m3 m4m4 m5m5 m6m6 m6m6 m6m6 m2m2 m2m2 TO(NUA,WUTO) TO(UA,WUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO) TO(NUA,WUTO)

25. TO specifications (3) TO(UA,WNUTO) m4m4 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m4m4 m2m2 TO(NUA,WNUTO) m4m4 p1p1 p2p2 p3p3 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m4m4 m2m2 m3m3 m4m4 m5m5 m6m6 m6m6 m6m6 m2m2 TO(UA,WNUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO) TO(UA,WUTO) TO(NUA,WUTO)