1. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Chapter 7 Consistency and Replication (1) DISTRIBUTED SYSTEMS (dDist) 2014

2. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Plan Motivation Consistency models –Data-centric consistency –Client-centric consistency Replica management Consistency protocols

3. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Replication Examples –Many web servers with same contents –Many web clients with copies of web server contents

4. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Reasons for Replication Data is replicated to increase the reliability of a system. Replication for performance –Scaling in numbers –Scaling in geographical area

5. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Caveats Cost of increased bandwidth for maintaining replication –Viability depends on ratio between reads and writes Possible consistency problems –What if Client writes to Server 1… –Client reads from Server 2 before write is propagated? –May not always be a problem…

6. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Tolerable Inconsistency Can keep all replicas identical using totally ordered multicast of updates and reads Very expensive, so we typically try to identify which inconsistencies are tolerable and use this to give more efficient protocols Consistency models are used to phrase which inconsistencies are allowed

7. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Consistency Models Data store provides –Read and write operations –Write operations are propagated to other copies Consistency model –Contract between processes and data store –E.g., read returns value of “latest” write – A need for being more precise here

8. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Continuous Consistency (1/4) Defining inconsistencies –Deviation in numerical values between replicas – numerical error E.g., $0.02 difference between to replicas of stock price –Deviation in staleness between replicas E.g., difference in when stock prize was last updated –Deviation in order of updates E.g., to keep price of roll-back reasonable Consistency units, conits

9. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Continuous Consistency (2/4) Figure 7-2. An example of keeping track of consistency deviations [adapted from (Yu and Vahdat, 2002)]. 1 unseen operation 1 unseen update with amount 5

10. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Continuous Consistency (3/4) Figure 7-3. Choosing the appropriate granularity for a conit. (a) Two updates lead to update propagation. Deviation = 2

11. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Continuous Consistency (4/4) Figure 7-3. Choosing the appropriate granularity for a conit. (b) No update propagation is needed (yet). Deviation = 1

12. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Consistent Ordering When replicas are kept consistent by propagating updates, we need to agree on the order of the updates that get committed Sequential Consistency –All writes are seen in the same, meaningful order at all replicates Causal Consistency –Writes which are causally related are seen in the same, meaningful order at all replicates

13. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Sequential Consistency (1/3) Figure 7-4. Behavior of two processes operating on the same data item. The horizontal axis is time. Write not propagated immediately

14. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Sequential Consistency (2/3) Sequential consistency: –All processes should see (a subset of) the same interleaving of operations –The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential global order –The operations made by each individual process appear in this sequence in the order it made them.

15. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Sequential Consistency (3/3) Figure 7-5. (a) A sequentially consistent data store. (b) A data store that is not sequentially consistent.

16. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Causal Consistency (1/4) Again concurrent operations by multiple processes Causal consistency: –Consider: P writes x, Q reads x, Q writes x The three events are causally related if what P wrote reached Q before the read In that case the events must be seen by all (other) processes in the same order –Writes that are potentially causally related must be seen by all processes in the same order –Concurrent writes may be seen in a different order on different machines!

17. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Causal Consistency (2/4) Figure 7-8. This sequence is allowed with a causally-consistent store, but not with a sequentially consistent store.

18. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Causal Consistency (3/4) Figure 7-9. (a) A violation of a causally- consistent store.

19. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Causal Consistency (4/4) Figure 7-9. (b) A correct sequence of events in a causally-consistent store.

20. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Entry Consistency Guard shared data with locks Updates are performed with respect to shared data when process acquires lock –Matches semantics of parallel programs in which access is undefined when lock is not taken

21. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Eventual Consistency Eventual consistency –Used when there are no write-write conflicts, only read-write conflicts Or when write-write is easy to handle –Propagate writes lazily –E.g., DNS, Web servers, Google Docs What if client is mobile? –We need a guarantee for single client, as it might e.g. be confusing to see time run backwards

22. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client Centric Consistency What if client is mobile? –We need a guarantee for single client, as it might e.g. be confusing to see time run backwards An issue not just with eventual consistency –Even with highly consistent databases an update will be applied / committed at different physical times at different databases –A mobile client or a client logged on at two databases might hence “travel in time”

23. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Centric Consistency Figure 7-11. The principle of a mobile user accessing different replicas of a distributed database.

24. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Centric Models FuturePast Read Write Read Write

25. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Reads (1/3) A data store is said to provide monotonic-read consistency if the following condition holds: If a process reads the value of a data item x, any successive read operation on x by that process will always return that same value or a more recent value, even at other replicas!

26. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Reads (2/3) Figure 7-12. The read operations performed by a single process P at two different local copies of the same data store. (a) A monotonic-read consistent data store. Set of write operations which reached server Write operations from L1 have also been performed at L2

27. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Reads (3/3) Figure 7-12. The read operations performed by a single process P at two different local copies of the same data store. (b) A data store that does not provide monotonic reads.

28. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Writes (1/3) In a monotonic-write consistent store, the following condition holds: A write operation by a process on a data item x is completed before any successive write operation on x by the same process, even at different replicas!

29. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Writes (2/3) Figure 7-13. The write operations performed by a single process P at two different local copies of the same data store. (a) A monotonic-write consistent data store.

30. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Monotonic Writes (3/3) Figure 7-13. The write operations performed by a single process P at two different local copies of the same data store. (b) A data store that does not provide monotonic-write consistency.

31. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Read Your Writes (1/3) A data store is said to provide read- your-writes consistency, if the following condition holds: The effect of a write operation by a process on data item x will always be seen by a successive read operation on x by the same process, even at other replicas!

32. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Read Your Writes (2/3) Figure 7-14. (a) A data store that provides read-your-writes consistency.

33. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Read Your Writes (3/3) Figure 7-14. (b) A data store that does not.

34. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Writes Follow Reads (1/3) A data store is said to provide writes- follow-reads consistency, if the following holds: A write operation by a process on a data item x following a previous read operation on x by the same process is guaranteed to take place on the same or a more recent value of x that was read, even at other replicas!

35. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Writes Follow Reads (2/3) Figure 7-15. (a) A writes-follow-reads consistent data store.

36. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Writes Follow Reads (3/3) Figure 7-15. (b) A data store that does not provide writes-follow-reads consistency.

37. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Replica Management We want to keep many copies of a busy webpage (think Amazon) for scalability and robustness Two questions to ask: –Where should replica servers be placed? –On which replicate servers should copies of a given entity be placed?

38. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Replica-Server Placement (1/2) Minimum K-Median Problem –Input n points Capacity between i and j: c ij Demand of i: d j Number k < n –Output k elements K  N: centers so that they minimize –NP-hard, polynomial-time approximations (PTAS) exists O(K N 2 ): Not good enough for online recalculations – Slashdot effect…

39. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Replica-Server Placement (2/2) Latency-driven replica placement –Select network regions Groups of nodes with low inter-latency, N nodes in total –Place replicas in K most active cells E.g., node with minimal distance to all other nodes in cell –Time O(N max(log N, K)) Includes computing the right cell size

40. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Content Replication and Placement Figure 7-17. The logical organization of different kinds of copies of a data store into three concentric rings.

41. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Permanent Replicas Initial set of replicas that constitute distributed data store –E.g., web server Replicated at a limited number of nodes –Within ownership domain Mirrored at limited number of nodes –Outside ownership domain

42. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server-Initiated Replicas Given a set of replica servers, place copies on the right servers. –For each access, increase access count for the server where it would have been optimal for to have the file –Use access counts to create and delete replicates of files Server closest to C1 and C2

43. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Initiated Replicas (Client) caches –Temporary store for copy of data –Can have several levels: browser, system, proxy, …

44. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Content Distribution (1/2) Propagate only a notification of an update (invalidation) –Good if there are more writes than read Transfer data from one copy to another –Good if there are more reads than writes Propagate the update operation to other copies –Good if updates are “small” compared to data

45. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Content Distribution (2/2) Pull versus Push Protocols –Figure 7-19. A comparison between push-based and pull- based protocols in the case of multiple-client, single-server systems. Lease: Server promises to push in a given period and then you must poll again –Also removes some of the disadvantages of having a statefull server

46. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Summary Consistency is essential in many applications –Performance –Reliability Consistency models –Data-centric consistency –Client-centric consistency Replica management Consistency protocols