Ordered Communication. Define guarantees about the order of deliveries inside group of processes Type of ordering: Deliveries respect the FIFO ordering.

Slides:



Advertisements
Similar presentations
Chapter 12 Message Ordering. Causal Ordering A single message should not be overtaken by a sequence of messages Stronger than FIFO Example of FIFO but.
Advertisements

CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 14: Simulations 1.
Global States.
Logical Clocks (2).
CS425/CSE424/ECE428 – Distributed Systems – Fall 2011 Material derived from slides by I. Gupta, M. Harandi, J. Hou, S. Mitra, K. Nahrstedt, N. Vaidya.
CS 542: Topics in Distributed Systems Diganta Goswami.
DISTRIBUTED SYSTEMS II FAULT-TOLERANT BROADCAST Prof Philippas Tsigas Distributed Computing and Systems Research Group.
CSE 486/586, Spring 2014 CSE 486/586 Distributed Systems Reliable Multicast Steve Ko Computer Sciences and Engineering University at Buffalo.
CS542 Topics in Distributed Systems Diganta Goswami.
Synchronization Chapter clock synchronization * 5.2 logical clocks * 5.3 global state * 5.4 election algorithm * 5.5 mutual exclusion * 5.6 distributed.
CPSC 668Set 14: Simulations1 CPSC 668 Distributed Algorithms and Systems Spring 2008 Prof. Jennifer Welch.
Lab 2 Group Communication Andreas Larsson
1 Principles of Reliable Distributed Systems Lecture 3: Synchronous Uniform Consensus Spring 2006 Dr. Idit Keidar.
CS 582 / CMPE 481 Distributed Systems Fault Tolerance.
Distributed systems Module 2 -Distributed algorithms Teaching unit 1 – Basic techniques Ernesto Damiani University of Bozen Lesson 3 – Distributed Systems.
Distributed Systems Fall 2009 Logical time, global states, and debugging.
CPSC 668Set 15: Broadcast1 CPSC 668 Distributed Algorithms and Systems Fall 2009 Prof. Jennifer Welch.
Group Communications Group communication: one source process sending a message to a group of processes: Destination is a group rather than a single process.
CPSC 668Set 16: Distributed Shared Memory1 CPSC 668 Distributed Algorithms and Systems Fall 2006 Prof. Jennifer Welch.
Group Communication Phuong Hoai Ha & Yi Zhang Introduction to Lab. assignments March 24 th, 2004.
Copyright © George Coulouris, Jean Dollimore, Tim Kindberg This material is made available for private study and for direct.
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Lecture 5: Synchronous Uniform.
1 Principles of Reliable Distributed Systems Lecture 5: Failure Models, Fault-Tolerant Broadcasts and State-Machine Replication Spring 2005 Dr. Idit Keidar.
CPSC 668Set 15: Broadcast1 CPSC 668 Distributed Algorithms and Systems Fall 2006 Prof. Jennifer Welch.
Distributed Systems Fall 2009 Replication Fall 20095DV0203 Outline Group communication Fault-tolerant services –Passive and active replication Highly.
Lecture 13 Synchronization (cont). EECE 411: Design of Distributed Software Applications Logistics Last quiz Max: 69 / Median: 52 / Min: 24 In a box outside.
Distributed systems Module 2 -Distributed algorithms Teaching unit 1 – Basic techniques Ernesto Damiani University of Bozen Lesson 4 – Consensus and reliable.
Aran Bergman & Eddie Bortnikov & Alex Shraer, Principles of Reliable Distributed Systems, Spring Principles of Reliable Distributed Systems Recitation.
Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Recitation 5: Reliable.
Lecture 12 Synchronization. EECE 411: Design of Distributed Software Applications Summary so far … A distributed system is: a collection of independent.
Composition Model and its code. bound:=bound+1.
State Machines CS 614 Thursday, Feb 21, 2002 Bill McCloskey.
Time, Clocks, and the Ordering of Events in a Distributed System Leslie Lamport (1978) Presented by: Yoav Kantor.
Logical Clocks (2). Topics r Logical clocks r Totally-Ordered Multicasting r Vector timestamps.
Computer Science 425 Distributed Systems (Fall 2009) Lecture 5 Multicast Communication Reading: Section 12.4 Klara Nahrstedt.
1 A Modular Approach to Fault-Tolerant Broadcasts and Related Problems Author: Vassos Hadzilacos and Sam Toueg Distributed Systems: 526 U1580 Professor:
ECE291 Computer Engineering II Lecture 23 Dr. Zbigniew Kalbarczyk University of Illinois at Urbana- Champaign.
Logical Clocks n event ordering, happened-before relation (review) n logical clocks conditions n scalar clocks condition implementation limitation n vector.
CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 15: Broadcast 1.
Total Order Broadcast and Multicast Algorithms: Taxonomy and Survey (Paper by X. Défago, A. Schiper, and P. Urbán) ACM computing Surveys, Vol. 36,No 4,
Reliable Communication in the Presence of Failures Based on the paper by: Kenneth Birman and Thomas A. Joseph Cesar Talledo COEN 317 Fall 05.
Lab 2 Group Communication Farnaz Moradi Based on slides by Andreas Larsson 2012.
Logical Clocks. Topics Logical clocks Totally-Ordered Multicasting Vector timestamps.
Farnaz Moradi Based on slides by Andreas Larsson 2013.
Communication & Synchronization Why do processes communicate in DS? –To exchange messages –To synchronize processes Why do processes synchronize in DS?
Distributed Systems Fall 2010 Logical time, global states, and debugging.
Totally Ordered Broadcast in the face of Network Partitions [Keidar and Dolev,2000] INF5360 Student Presentation 4/3-08 Miran Damjanovic
Distributed systems Consensus Prof R. Guerraoui Distributed Programming Laboratory.
CIS825 Lecture 2. Model Processors Communication medium.
D u k e S y s t e m s Asynchronous Replicated State Machines (Causal Multicast and All That) Jeff Chase Duke University.
SysRép / 2.5A. SchiperEté The consensus problem.
CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS Fall 2011 Prof. Jennifer Welch CSCE 668 Set 16: Distributed Shared Memory 1.
Logical Clocks. Topics r Logical clocks r Totally-Ordered Multicasting.
Fault-Tolerant Broadcast Terminology: broadcast(m) a process broadcasts a message to the others deliver(m) a process delivers a message to itself 1.
Distributed Systems Lecture 9 Leader election 1. Previous lecture Middleware RPC and RMI – Marshalling 2.
Logical Clocks event ordering, happened-before relation (review) logical clocks conditions scalar clocks  condition  implementation  limitation vector.
Distributed Systems Lecture 6 Global states and snapshots 1.
EEC 688/788 Secure and Dependable Computing Lecture 10 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University
Distributed systems Causal Broadcast
CSE 486/586 Distributed Systems Reliable Multicast --- 1
Distributed systems Total Order Broadcast
Computer Science 425 Distributed Systems CS 425 / ECE 428 Fall 2013
CSCE 668 DISTRIBUTED ALGORITHMS AND SYSTEMS
EEC 688/788 Secure and Dependable Computing
EEC 688/788 Secure and Dependable Computing
EEC 688/788 Secure and Dependable Computing
Distributed systems Causal Broadcast
CSE 486/586 Distributed Systems Reliable Multicast --- 1
Presentation transcript:

Ordered Communication

Define guarantees about the order of deliveries inside group of processes Type of ordering: Deliveries respect the FIFO ordering of the corresponding sendings Deliveries respect the Causal ordering of the corresponding sendings Delivery respects a total ordering of deliveries (atomic communication)

Advantages of ordered communication Orthogonality wrt reliable communication. Reliable broadcast does not have any property on ordering deliveries of messages This can cause anomalies in many applicative contexts “Reliable ordered communication” are obtained adding one or more ordering properties to reliable communication Example: flight booking system. Consider the message pattern depicted in the figure. The server cansel a reservation that has never been done! t client server “reserve”“cancel” “Prices 15% off”

FIFO Broadcast\ specification Messages sent by the a process has to be delivered in the sending order. Safety FIFO Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): FIFO Order: if a process sends a broadcast message m before m’, then no correct process delivers m’ if it has not already delivered m. FIFO order can be uniform/non uniform FIFO Broadcast = Reliable Broadcast + FIFO Order

Each process q holds: S p a count of messages broadcast by p R p the sequence number of the latest message sent by p and delivered by q For p to FO-multicast a message to g, it piggybacks S p on the message, rbBroadcasts it and increments S p by 1 On receipt of a message from q sent by p with sequence number S, p checks whether S = R p + 1. If so, q FO-delivers it if S > R p + 1 then q places message in hold-back queue until intervening messages have been delivered. (note that rbBroadcast does eventually deliver messages unless the sender crashes) FIFO Broadcast\algorithm

Advantages of Ordered Communication (2) FIFO does not prevent all the anomalies due to order of deliveries. Es: newsgroup. Even though the computation satisfies FIFO order, student 2 can be disoriented by the order in which it receives messages m1 and m2. m1 causally precede m2, then m2 can be delivered only if m1 is delivered. Causal Broadcast Prof. Student 2 m 1 : “Fri exam cancelled” Student 1 m 2 : “let’s party on Thu night” m 3 : “but we have an exam on Fri!”

Causal Order  FIFO Order, But FIFO Order  Causal Order thus, Causal Order = FIFO Order + ? Causal Broadcast\specification Safety Causal Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): Causal Order: if the sending of a message m causally precedes the sending of a message m’, then every correct process has to deliver m before delivering m’. Causal Broadcast = Reliable Broadcast+Causal Order

Causal Broadcast\specification Causal Order = FIFO Order + Local Order. Local Order : if a process delivers a message m before sending a msg m’, then no correct process deliver m’ if it has not already delivered m. Example: p q r t m m’ The delivery of m’ is delaied till the arrival and the delivery of m

Causal Broadcast\ implementations Two implementations blocking algorthm using vector clocks (already discussed) non-blocking algorithm using piggybacking of causal past p1p1 p2p2 p3p3 COBcast(m 1 ) CObcast(m 2 ) COBcast(m 3 ) m1m1 m2m2  m 1,m 2, m 3  COdelv(m 1 ) COdelv(m 3 ) m 2 già COdelivered! COdelv(m 1 ) COdelv(m 2 ) COdelv(m 3 ) COdelv(m 2 ) Filter out

Advantages of Ordered Communication (3) Causal Order is not enough strong to avoid anomalies Es. banking. Bank account replicated on two sites R1R1 R2R2 A:£100 Deposit £20 Add 10% interest A:£120 A:£110 A:£132 A:£130 Despite the fact that replicas initially share the same state, the state reaches a different value in the two sites at the end of the exexution as shown in Figure. Note that the computation is Causally Ordered To guarantee values of replicas be the same, one has to ensure that the order of delivery be the same at each process. In the above example R1 delivers m1 before m2 and R2 delivers m2 before m1. Note that ensuring the same delivery order at each replicas does not look at the sending order of messages m1 m2

Atomic Broadcast\specification Safety Atomic (Total) Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of total order (example): Total Order: if two correct processes p and q deliver m and m’, then p delivers m before m’ if, and only if, q delivers m before m’ Total order is orthogonal with respect to FIFO and Causal Order. Total order would accept indeed a computation in which a process sends n messages to a group, and each of the processes of the group delivers such messages in the reverse order of their sendings. The computation is totally ordered but it is not FIFO.

Causal Atomic broadcast Hierarchy of Broadcast Specifications Reliable broadcast FIFO broadcast Causal broadcast FIFO Atomic broadcast Atomic broadcast Total Order Causal Order Total Order FIFO Order Local Order Causal Order Local Order

System model Static set of processes Π = {p 1 … p n } Message passing over perfect channels (message exchanging between correct processes is reliable) Asynchronous Crash fault model for processes We characterize the system in terms of its possible runs R R p1p1 p2p2 pnpn TOcast(m) m m m TOdeliver(m) crash r

A few notation Property P: predicate on the system, identifying a set of runs R P  R P  P’ iff R P  R P’ Specification S(P 1,…,P m ): logical and of m properties, identifying a set of runs R S =R P 1 ∩ … ∩ R P m  R S → S’ iff R S  R S’ RPRP R P’ RSRS R S’ R P1 R Pn RSRS R R R

TO specifications Total order specifications are usually composed by four properties, namely Validity, Integrity,Agreement, and Order. A Validity property guarantees that messages sent by correct processes will eventually be delivered at least by correct processes; An Integrity property guarantees that no spurious or duplicate messages are delivered; An Agreement property ensures that (at least correct) processes deliver the same set of messages; An Order property constrains (at least correct) processes delivering the same messages to deliver them in the same order.

TO specifications Total Order Broadcast = S(V,I,A,O) V = Validity I = Integrity A = Agreement O = Order Distinct specifications arise from distinct formulations of each property uniform vs non-uniform A uniform property imposes restrictions on the behavior of (at least) correct processes on the basis of events occurred in some process NUV UI TO(A,O)

TO Specifications Crash failure + Perfect channels  NUV. if a correct process TOCAST a message m then some correct process will eventually deliver m UI. For any message m, every process p delivers m at most once and only if m was previously tocast by some (correct or not) process.

The Agreement property (Uniform Agreement, UA) If a process (correct or not) todelivers a message m, then all correct processes will eventually todeliver m; (Non-uniform Agreement, NUA) If a correct process todelivers a message m, then all correct processes will eventually todeliver m

The Agreement property Constrains the set of delivered messages Correct processes always deliver the same set of messages M Each faulty process p delivers a set M p UA: M p  M NUA: M p can be s.t. M p - M ≠  m2m2 m4m4 p1p1 p2p2 p3p3 m2m2 m4m4 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m1m1 m2m2 UA m4m4 p1p1 p2p2 p3p3 m2m2 m4m4 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m1m1 m2m2 m5m5 NUA

The Order property Constrains the order of message deliveries and possibly the set of delivered messages SUTO: if p delivers m<m’, q delivers m’ only after m same order same prefix of the set of delivered messages after an omission, disjoint sets of delivered messages WUTO: if p,q deliver m,m’, they get the same order no restrictions on the set of delivered messages p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 SUTO WUTO

The Order property (2) SUTO and WUTO are uniform They both have a non-uniform counterparts: SNUTO and WNUTO (Strong Non-uniform Total Order, SNUTO). If some correct process todelivers some message m before message m', then a correct process todelivers m‘ only after it has todelivered m. (Weak Non-uniform Total Order, WNUTO) If correct processes p and q both todeliver messages m and m', then p todelivers m before m' if and only if q todelivers m before m‘

The Order property (2) SUTO  WUTO SNUTO  WNUTO p1p1 p2p2 p3p3 m1m1 m2m2 m2m2 m1m1 m1m1 m2m2 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 SNUTO p1p1 p2p2 p3p3 m1m1 m2m2 m1m1 m1m1 m2m2 m4m4 m3m3 m3m3 m7m7 m6m6 m5m5 WNUTO m2m2

TO specifications TO(UA,SUTO) The strongest TO spec. p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m6m6 m6m6 p1p1 p2p2 p3p3 m2m2 m2m2 m2m2 m1m1 m1m1 m1m1 m4m4 m3m3 m3m3 m6m6 m6m6 m5m5 TO(NUA,SUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO)

TO specifications (2) TO(UA,WUTO) m3m3 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m4m4 m4m4 m4m4 m3m3 p1p1 p2p2 p3p3 m2m2 m1m1 m1m1 m1m1 m3m3 m4m4 m4m4 m2m2 m3m3 m4m4 m5m5 m6m6 m6m6 m6m6 m2m2 m2m2 TO(NUA,WUTO) TO(UA,WUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO) TO(NUA,WUTO)

TO specifications (3) TO(UA,WNUTO) m4m4 p1p1 p2p2 p3p3 m2m2 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m3m3 m4m4 m4m4 m2m2 TO(NUA,WNUTO) m4m4 p1p1 p2p2 p3p3 m2m2 m1m1 m1m1 m1m1 m3m3 m3m3 m4m4 m2m2 m3m3 m4m4 m5m5 m6m6 m6m6 m6m6 m2m2 TO(UA,WNUTO) TO(UA,SUTO) (Strongest total order) TO(NUA,SUTO) TO(UA,WUTO) TO(NUA,WUTO)