1. Consistency and Replication (1)

2. Topics Why Replication? Consistency Models – How do we reason about the consistency of the “global state”? u Data-centric consistency u Client-centric consistency Update propagation – How does an update to one copy of an item get propagated to other copies? Replication protocols – What is the algorithm that takes on update propagation method and enforces a given consistency model.

3. Readings Van Steen and Tanenbaum: 6.1, 6.2 and 6.3 Coulouris: 14

4. Why Replicate? Common approach used in distributed systems Reliability u If one replica is unavailable or crashes, use another u Avoid single points of failure Performance u Placing copies of data close to the processes using them can improve performance through reduction of access time. u If there is only one copy, then the server could become overloaded.

5. Why Not Replicate? Multiple copies may lead to consistency problems. Whenever a copy is modified, that copy becomes different from the rest. Modifications have to be carried out on all copies to ensure consistency. Exactly when and how those modifications need to be carried out determines the price of replication.

6. Why Not Replicate? Example: u To improve performance, web browsers often locally store a copy of a previously fetched web page. This is referred to as caching a web page. u User may perceive access time as excellent. u However, if the user always wants to have the latest version of a page, they may be in for bad luck. u What if the page has been modified and those modification have not been propagated (yet) to cached copies?

7. Why Not Replicate Solutions: u Don’t allow caches – poor access times u Let the web server invalidate or update cached copy – requires the server to keep track of all caches and send them messages. Could degrade performance of server.

8. Performance and Scalability Tradeoff: Keeping copies current may require more network bandwidth. Consider a process P that accesses a local replica N times per second, whereas the replica itself is updated M times per second. If N << M, we have the situation where many updated versions of the local replica will never be accessed by P, rendering the network communication for those versions useless.

9. Performance and Scalability Keeping multiple copies consistent may be subject to scalability problems. A collection of copies is consistent when the copies are always the same. This means that a read operation performed at any copy will always return the same result. When an update propagation is performed on one copy, the update should be propagated to all copies before a subsequent operation takes place. This is referred to as tight consistency.

10. Performance and Scalability To keep replicas tightly consistent, we generally need to ensure that all conflicting operations are done in the same order everywhere. Conflicting operations are defined as follows: u Read-write conflict: A read operation and a write operation act concurrently on the same data item. u Write-write conflicts: Two concurrent write operations on the same data item. Guaranteeing global ordering on conflicting operations is costly in terms of performance. This potentially downgrades the usefulness of replication. Consistency requirements must be weakened. This means that copies may not always be the same everywhere.

11. Data-Centric Consistency Models A consistency model is a contract between a distributed store and processes, in which the data store specifies precisely what the results of read and write operations are in the presence of concurrency. A data store is a distributed collection of storages accessible to clients.

12. Data-Centric Consistency Models

13. Notation To study consistency in detail, we will give numerous examples. Special notation is used. The symbol W i (x)a indicates a write by process P i to data item x with the value a The symbol R i (x)b indicates a read from the item x by P i returning b. It is assumed that each data item is initially NILL.

14. Strict Consistency Strict consistency is defined as follows: Any read to a shared data item X returns the value stored by the most recent write operation on X. You get strict consistency in the normal sequential case, where your program does not interfere with other programs.

15. Strict Consistency Behavior of two processes, operating on the same data item. A strictly consistent store. (a) A store that is not strictly consistent. (b)

16. Strict Consistency Suppose x is a data item stored only on machine B. Imagine that a process on A reads x at time T 1 which means that a message is sent to B to get x. Slightly later, at T 2, a process on B does a write to x. If strict consistency holds, the read should always return the old value regardless of where the machines are and how close T 2 is to T 1

17. Strict Consistency If T 2 -T 1 is 1 nanosecond and the machines are three meters apart, then to propagate the read request from A to B to get there before the write, the signal would have to travel the speed of light. When a data store is strictly consistent, all writes are instantaneously visible to all processes. The laws of physics are not with us on this. Is it reasonable to have strict consistency?

18. Sequential Consistency Two criteria must be satisfied: u The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential order u The operations of each individual process appear in this sequence in the order specified by its program. u Nothing is said about most recent write We are talking about interleaved executions: there is some total ordering for all operations taken together. Read operations are local, but write operations are applied everywhere. Linearizable: Sequential plus operations are ordered according to a global timestamp.

19. Sequential Consistency x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); Prints: 001011 (a) x = 1; y = 1; print (x,z); print(y, z); z = 1; print (x, y); Prints: 101011 (b) y = 1; z = 1; print (x, y); print (x, z); x = 1; print (y, z); Prints: 010111 (c) y = 1; x = 1; z = 1; print (x, z); print (y, z); print (x, y); Prints: 111111 (d) Process P1Process P2Process P3 x = 1; print ( y, z); y = 1; print (x, z); z = 1; print (x, y);

20. Sequential Consistency a) A sequentially consistent data store. b) A data store that is not sequentially consistent.

21. Sequential Consistency One possible implementation u Remember Lamport’s algorithm for total order

22. Causal Consistency Definition: Writes that are potentially causally related must be seen by all processes in the same order. Concurrent writes may be seen in a different order by different processes. Example u Suppose that process P 1 writes a variable x. u P 2 reads x and writes y u The reading of x and the writing of y are potentially causally related if the computation of y may have depended on the value of x read by P 2

23. Casual Consistency This sequence is allowed with a casually-consistent store, but not with sequential or strictly consistent store: W 2 (x)b and W 1 (x)c are concurrent.

24. Casual Consistency a) A violation of a casually-consistent store. b) A correct sequence of events in a casually-consistent store. In (a) we have W 2 (x)b potentially depending on W 1 (x)a because the b may be a result of a computation involving the value read by R 2 (x)a; In (b ) the read has been removed and thus W 1 (x)a and W 2 (x)b are concurrent.

25. Causal Consistency Possible Implementation u Use vector timestamps. u More on this later.

26. FIFO Consistency Definition: Writes done by a single process are received by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes. Basically this is saying that there are no guarantees about the order in which different processes see writes, except that two or more writes from a single source must arrive in order.

27. FIFO Consistency A valid sequence of events of FIFO consistency

28. FIFO Consistency Easy to implement u Tag each write operation with a (process, sequence number). u Perform writes per process in the order of their sequence number.

29. Weak Consistency It is not always necessary that all writes are seen. Consider the case of a process inside a critical section writing records to a replicated database. u The important thing is that the final results are sent everywhere. u There shouldn’t be too much worrying about propagation of intermediate results

30. Weak Consistency Need synchronization variables. Accesses to synchronization variables associated with a data store are sequentially consistent u All processes see them in the same order No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere

31. Eventual Consistency There are examples in which concurrency appears only in a restricted form. Example: u In database systems, most processes hardly ever perform update operations but only read data from the database. u Relatively few processes perform update operations. Question: How fast should updates be made available to only reading processes?

32. Eventual Consistency Example: Consider DNS u DNS name space is partitioned into domains, where each domain is assigned to a naming authority, which acts as owner of that domain. u Only that authority is allowed to update its part of the name space. u Thus, conflicts resulting from two operations that both want to perform an update on the same data never occur. u It is often possible to propagate an update in a lazy fashion which means that a reading process will see an update only after some time has passed since the update took place.

33. Eventual Consistency Example: WWW u Web pages are updated by a single authority, such as a webmaster or the actual owner of the pages. u No write-write conflicts to resolve. u Browsers cache u It is possible that the cached page that is returned to the requesting client is an older version compared to the one available at the actual web server. u Is this really a big deal? The examples provided show that there are applications that can tolerate a relatively high degree of inconsistency. They have in common that if no updates take place for a long time, all replicas will gradually become consistent. This is known as eventual consistency.

34. Eventual Consistency Data stores that are eventually consistent thus have the property that in the absence of updates, all replicas converge toward identical copies of each other. Eventual consistency requires only that updates are guaranteed to propagate to all replicas. However…… u Problems arise when different replicas are accessed. u See example of mobile user.

35. Eventual Consistency The principle of a mobile user accessing different replicas of a distributed database.

36. Eventual Consistency The mobile user accesses the database by connecting to one of the replicas in a transparent way. The application running on the user’s portable computer is unaware (ideally) on which replica it is actually operating. Assume the user performs several update operations and then disconnects again. Later the user accesses the database again, possibly after moving to a different location or by using a different access device. The user may be connected to a different replica. What if the updates have not propagated? Could be confusing to the user.

37. Eventual Consistency The example of the mobile users is typical and is caused by the fact that users may sometimes operate on different replicas. Client-centric consistency is needed. Client-centric consistency provides guarantees for a single client concerning the consistency of accesses to a data store by that client. No guarantees are given concerning concurrent accesses by different clients. Goal: Concentrate on what specific clients want, instead of what should be maintained by servers.

38. Monotonic Reads If a process reads the value of a data item x, any successive read operation on x by that process will always return that same value or a more recent value. Example: Consider a distributed e-mail database In such a database, each user’s mailbox may be distributed and replicated across multiple machines. Mail can be inserted in a mailbox at any location. Updates are propagated in a lazy (i.e., on demand) fashion. Assume that reads don’t change the mailbox. Suppose a user reads their e-mail in Vancouver and then flies to Toronto and reads their e-mail. A monotonic read guarantees that the messages that were in the mailbox in Vancouver will also be in the mailbox in Toronto.

39. Monotonic Writes A write operation by a process on a data item x is completed before any successive write operation on x by the same process. This is especially needed in those cases where a write may affect only part of the state of a data item. Example: Updating a software library u Update may consist of replacing one or more functions resulting in a new version. u Updates performed on a copy of the library should be able to assume that all proceeding updates have been performed first. This is similar to data-centric FIFO consistency.

40. Read Your Writes The effect of a write operation by a process on data item x will always be seen by a successive read operation on x by the same process. The absence of this consistency is seen in the following examples. Example: Updating Web HTML pages u Cached web pages are still read even though that web page has been updated. Example: Password updates for digital library u This may occur at one site, but not immediately propagated to a site where the account/password is actually needed

41. Writes Follow Reads A write operation by a process on a data item x following a previous read operation on x by the same process, is guaranteed to take place on the same or a more recent value of x that was read. In other words, any successive write operation by a process on a data item x will be performed on a copy of x that is up to date with the value most recently read by that process. This can be used to guarantee that a user of a network newsgroup see a posting of a reaction to an article only after they have seen the original article.

42. Implementing Client-Centric Models Globally unique ID per write operation u Assigned by the initiating server u Global IDs can be generated locally. For each client, we keep track of two sets of write identifiers: u Read set n Write IDs relevant to client’s read operations u Write set n IDs of writes performed by client Major performance issue: u Size of read/write sets

43. Implementing Client-Centric Models The global identifier could include the identifier of a server where the operation was initiated. A server is required to log the write operation so that it can be replayed at another server.

44. Implementing Client-Centric Models Monotonic read: u When a client issues a read, the server is given the client’s read set to check whether all the identified writes have taken place locally n If not, the server contacts others to ensure that it is brought up-to-date u After the read, the client’s read set is updated with the server’s “relevant” writes Monotonic write: u When a client issues a write, the server is given the client’s write set n … to ensure that all specified writes have been applied (in-order) u The write operation’s ID is appended to client’s write set

45. Implementing Client-Centric Models Read-your-writes: u Before serving a read request, the server fetches (from other servers) all writes in the client’s write set Writes-follow-reads: u Server is brought up-to-date with the writes in the client’s read set u After write, the new ID is added to the client’s write set, along with the IDs in the read set n … as these have become “relevant” for the write just performed

46. Implementing Client-Centric Models Using vector timestamps for improving efficiency: u When server S i accepts a write operation, it assigns to it a globally unique WID and a timestamp ts(WID) u Each server maintains vector RCVD(i) n RCVD(i)[j] := timestamp of the latest write initiated at server S j that has been received & processed at S i Server returns its current vector timestamp with its responses to read/write requests Client adjusts the timestamp for its own read/write set (how this is done is described next).

47. Implementing Client-Centric Models When a client issues a request to perform a read/write operation at a specific server, the server returns its current timestamp along with the results of the operation. Read and write sets are subsequently represented by vector timestamps. For any such set A, the client constructs VT(A) with VT(A)[i] set equal to the maximum timestamp of all operations in A that were initiated at server S i

48. Implementing Client-Centric Models Efficient representation of read/write set A: u VT(A): vector timestamp n VT(A)[i] := max. timestamp of all operations in A that were initiated at server S i u Union of 2 sets of write IDs: n VT(A+B)[i] := max{ VT(A)[i], VT(B)[i] } u Efficient way to check if A is contained in B: n VT(A)[i] <= VT[B](i)

49. Implementing Client-Centric Models When a server passes its current timestamp back to a client, the client adjusts the vector timestamp of its own read or write set, depending on the operation that was performed.

50. Implementing Client-Centric Models Consider monotonic-read consistency u The client is returned the vector timestamp RCVD(i) from server S i u If the timestamp of the client’s read set is VT(Rset), then for each j, VT(Rset)[j] is set equal to max(VT(Rset)[j],RCVD(i)[j]). The client’s read set will thus reflect the latest write operations it has seen. The corresponding vector timestamp will be sent along with the next read operation that the client issues.