2 Topics Reasons for Replication Models of Consistency –Data-centric consistency models: strict, linearizable, sequential, causal, FIFO, weak, release, entry –Client-centric consistency models: monotonic reads, monotonic writes, read-your-writes Protocols for Achieving Consistency –ROWA –Read and write quorums
3 Replication Reasons: –Reliability: increase availability when servers crash –Performance: load balancing; scale with size of geographical region –Availability: local server likely to be available When one copy is modified, all replicas have to be updated Problem: how to keep the replicas consistent
4 Object Replication Approach 1: application is responsible for replication –Application needs to handle consistency issues Approach 2: system (middleware) handles replication –Consistency handled by the middleware: Simplifies application development but makes object-specific solutions harder (CORBA)
5 Replication and Scaling Replication and caching used for system scalability Multiple copies: –Improves performance by reducing access latency –But higher network overhead of maintaining consistency –Example: object is replicated N times Read frequency R, write frequency W If R <= W, high consistency overhead If R >> W, replication makes sense Consistency maintenance is itself an issue –What semantics to provide? –Tight consistency requires globally synchronized clocks! Solution: loosen consistency requirements –Variety of consistency semantics possible
6 Consistency Models Consistency Model: contract between processes and the data store. If processes follow contract, the data store works correctly.
7 Data-Centric Consistency Data must maintain consistency to a globally-aware standard. “I see all the data servers at once and determine if they are properly consistent”. Data-centric consistency describes how all copies should be updated regardless of whether or not some client sees those updates. In client-centric we are only concerned with what a particular client sees. An intelligent front-end can restrict the servers that certain clients can access so the client sees the desired consistency.
8 Strict Consistency Behavior of two processes, operating on the same data item. (a) A strictly consistent store. (b) A store that is not strictly consistent. Def.: Any read on a data item x returns a value corresponding to the result of the most recent write on x (regardless of which copy was written to).
9 Sequential Consistency (1) a)A sequentially consistent data store. b)A data store that is not sequentially consistent. Def.: The result of any excution is the same as if the operations by all processes on the data store were executed in some sequential order and the operations of each individual process appear in this sequence in the order specified by its program. Sequential consistency is weaker than strict consistency All processes see the same interleaving of operations
10 Sequential Consistency (2) Any valid interleaving is allowed All agree on the same interleaving Each process preserves its program order Nothing is said about “most recent write” Process P1Process P2Process P3 x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); Sequential consistency like FIFO and total order
11 Sequential Consistency Process P1Process P2Process P3 x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); P1 P2 P3 Distributed data store looks like a single copy
12 Sequential Consistency (3) Four valid execution sequences for the processes of the previous slide. The vertical axis is time. x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); Prints: 001011 Signature: 001011 (a) x = 1; y = 1; print (x,z); print(y, z); z = 1; print (x, y); Prints: 101011 Signature: 101011 (b) y = 1; z = 1; print (x, y); print (x, z); x = 1; print (y, z); Prints: 010111 Signature: 010111 (c) y = 1; x = 1; z = 1; print (x, z); print (y, z); print (x, y); Prints: 111111 Signature: 111111 (d)
13 Linearizability Assumption: Operations are timestamped (e.g., Lamport TS) Def.: The result of any execution is the same as if the operations by all processes on the data store were executed in some sequential order and the operations of each individual process appear in this sequence in the order specified by its program. In addition, if ts OP1 (x)
"name": "13 Linearizability Assumption: Operations are timestamped (e.g., Lamport TS) Def.: The result of any execution is the same as if the operations by all processes on the data store were executed in some sequential order and the operations of each individual process appear in this sequence in the order specified by its program.",
"description": "In addition, if ts OP1 (x)
14 Linearizability Neither of these is linearizable since W(x)a on P1 occurs before W(x)b on P2. In (a) P3 and P4 would have to see a before b.
15 Causal Consistency (1) Writes that are potentially causally related must be seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines. Causal consistency is weaker than sequential consistency
16 Causal Consistency P3 P2 P1 Distributed data store starts to look more like multiple copies
17 Causal Consistency (2) This sequence is allowed with a causally-consistent store, but not with sequentially or strictly consistent store. W 2 (x)b may depend on R 2 (x)a and therefore depends on W 1 (x)a. Thus, a must be seen before b at other processes W 2 (x)b and W 1 (x)c are concurrent
18 Causal Consistency (3) a)A violation of a causally-consistent store: W 2 (x)b depends on W 1 (x)a. b)A correct sequence of events in a causally-consistent store.
19 FIFO Consistency (1) Writes done by a single process are seen by all other processes in the order in which they were issued. Writes from different processes may be seen in a different order by different processes. FIFO consistency is weaker than causal consistency. Simple implementation: tag each write by (Proc ID, seq #)
20 FIFO Consistency (2) A valid sequence of events of FIFO consistency
21 FIFO Consistency (3) Statement execution as seen by the three processes. The statements in bold are the write-updates originating from other the processes. Signature 001001 not possible with sequential consistency. In sequential consistency, all processes have the same view. Process P1’s view x = 1; print (y, z); y = 1; z = 1; Prints: 00 Process P2’s view x = 1; y = 1; print (x, z); z = 1; Prints: 10 Process P3’s view y = 1; z = 1; print (x, y); x = 1; Prints: 01 Process P1Process P2Process P3 x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); Signature: 001001
22 FIFO Consistency (4) Sequential consistency: 6 possible statement orderings; none of them kills both processes FIFO consistency: both processes can get killed Process P1Process P2 x = 1; if (y == 0) kill (P2); y = 1; if (x == 0) kill (P1);
23 Models Based on a Sync Operation No consistency is enforced until a synchronization operation is performed. This operation can be done after local reads and writes to propagate the changes to the system. Local copies are ‘caches’. Weak Consistency Release Consistency Entry Consistency
25 Weak Consistency (1) Often not necessary to see all writes done by all processes Weak consistency enforces consistency on a group of operations; not individual read/write statements Synchronization point: –Propagate changes made to local data store to remote data store –Changes made to remote data store are imported Weak consistency is weaker than FIFO consistency
26 Weak Consistency (2) Properties: Accesses to synchronization variables associated with a data store are sequentially consistent (i.e., all processes see all operations on synchronization variables in the same order) No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere (i.e., guarantees all writes have propagated) No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed (i.e., when accessing data items, all previous synchronizations have completed)
27 Weak Consistency (3) a)A valid sequence of events for weak consistency. b)An invalid sequence for weak consistency.
28 Release Consistency (1) More efficient implementation than weak consistency by identifying critical regions –Acquire: ensure that all local copies of the data are brought up to date to be consistent with (released) remote ones –Release: data that has been changed is propagated out to remote data stores Acquire does not guarantee that locally made changes will be sent to other copies immediately Release does not necessarily import changes from other copies
29 Release Consistency (2) A valid event sequence for release consistency. The sync operation is divided into two: acquire and release. In some implementations, ‘acquire’ sets a lock in addition to obtaining the latest value of the data and ‘release’ releases the lock in addition to writing the data.
30 Release Consistency (3) Rules: Before a read or write operation on shared data is performed, all previous acquires done by the process must have completed successfully. Before a release is allowed to be performed, all previous reads and writes by the process must have completed Accesses to synchronization variables are FIFO consistent (sequential consistency is not required).
31 Release Consistency (4) Different implementations: –Eager release consistency: process doing the release pushes out all the modified data immediately to other processes. –Lazy release consistency: no update messages are sent at time of release. When another process does an acquire, it has to obtain the most recent version.
32 Entry Consistency (1) Every data item is associated with synchronization variables. Each data item can be acquired and released as in release consistency. Acquire (entry) gets most recent value. Advantage: increased parallelism Disadvantage: increased overhead
33 Entry Consistency (2) A valid event sequence for entry consistency.
34 Summary of Consistency Models a)Consistency models not using synchronization operations. b)Models with synchronization operations. ConsistencyDescription StrictAbsolute time ordering of all shared accesses matters. Linearizability All processes must see all shared accesses in the same order. Accesses are furthermore ordered according to a (nonunique) global timestamp Sequential All processes see all shared accesses in the same order. Accesses are not ordered in time CausalAll processes see causally-related shared accesses in the same order. FIFO All processes see writes from each other in the order they were used. Writes from different processes may not always be seen in that order (a) ConsistencyDescription WeakShared data can be counted on to be consistent only after a synchronization is done ReleaseTwo sync operations: acquire and release EntryEach data item has an operation for acquire and release. (b)
35 Client Centric Consistency (1) Strong consistency for data store often not necessary Consistency guarantees from a clients perspective Clients often tolerate inconsistencies (e.g., out of date web-pages) Assumptions: –Client may move to a different replica during a single session or may be prevented –Eventual consistent data store: total propagation and consistent ordering Trade-off: consistency vs. availability
36 Client Centric Consistency (2) The principle of a mobile user accessing different replicas of a distributed database.
37 Intuition Assume the application is like a message board, so when P1 reads x, it sees the history of values for x. Also, when P1 writes to x, the new value is appended like a new message would be. 1: X = 2 2: X = 9 3: X = 5.
38 Monotonic Reads (1) If a process P1 reads the value of a data item x, any successive read by P1 will always return that same value or a more recent one. P1 R(x)
39 Monotonic Reads (2) Definition: If Read R1 occurs before R2 in a session and R1 accesses server S1 at time t1 and R2 accesses server S2 at time t2, R2 sees the same as R1 or a more recent value. R1 and R2 are operations by the same client. Example: Calendar updates
40 Monotonic Reads (3) S1W1R1 S2W1R2 Valid S1W1R1 S2R2 Invalid: R2 doesn‘t see W1 If R1 saw W1, then a later R2 should see W1. W0 One client, 2 servers R1 and R2 are 2 reads by the same client.
41 Monotonic Writes (1) A write operation by P1 on data item x is completed before any successive write of x by P1 (even if P1 is now attached to a different server). Assume the application is like a message board, so when P1 reads x, it sees the history of values for x.
42 Monotonic Writes (1) Definition: If Write W1 precedes Write W2 in a session, then, for any server S2, if W2 is on S2 then W1 is on S2 and the order is W1 precedes W2. Like monotonic reads except the writes force consistency. Example: Software Update
43 Monotonic Writes (2) S1W1 S2W1W2 Valid S1W1 S2W2 Invalid W0 Not valid because W1 is not in the history of S2
44 Read Your Writes (1) Definition: If a client’s Read R follows Write W in a session and R is performed at server S at time t, then W is on S at time t Example: Password update propagation: you update your password on one system in the design center then move to another machine. “If I write something, then do a read – what I wrote should be there”.
46 State vs. Operations Design choices of update propagation: 1.Propagate only a notification of an update (e.g., invalidation protocols) 2.Transfer data from one copy to another 3.Propagate the update operation to other copies (a.k.a. active replication)
47 Epidemic Protocols Useful for eventual consistency Propagating updates to all replicas in as few messages as possible Update propagation model: –Infective: node holds update and is willing to spread –Susceptible: node willing to accept update –Removed: updated node not willing to spread Anti-entropy: pick nodes at random Exchanging updates between nodes P and Q: 1.P only pushes its own updates to Q 2.P only pulls in new updates from Q 3.P and Q send updates to each other
48 Consistency Protocol Implementations Primary based (each data item has an associated primary): –Passive replication –Queries (read-only) can be done on backup Replicated write (write operations carried out at multiple replicas, update anywhere): –Active replication –May use quorum based protocols
49 Primary based Replica Management FE C C RM Primary Backup RM Primary-based protocol with a fixed server to which all read and write operations are forwarded. (could be eager or lazy)
50 Primary-Backup Replication There is a single primary RM and one or more secondary (backup, slave) RMs FEs communicate with the primary which executes the operation and sends copies of the updated data to backups If the primary fails, one of the backups is promoted to act as the primary This system implements linearizability, since the primary sequences all the operations on the shared objects
51 Active Replication FEC CRM a FE multicasts each request to the group of RMs Requires totally ordered reliable multicast so that all RMs perfrom the same operations in the same order the RMs process each request identically and reply This is Read One Write All (ROWA)
52 Alternative to ROWA Write quorum: all write quorums must have a non-empty intersection. Read quorum: any read quorum must have a non-empty intersection with all write quorums. Write operation: write new value to all copies in the quorum and update version number. Read operation: read all copies in read quorum and pick the most recent (copies must have write timestamp or version number).
53 Quorum-Based Protocols Three examples of the voting algorithm: a)A correct choice of read and write set b)A choice that may lead to write-write conflicts c)A correct choice, known as ROWA (read one, write all) Constraints on read quorum N R and write quorum N W : 1.N R + N W > N 2.N W > N/2
54 Read and Write Quorums Show what happens when RQ=3 and WQ=3 –P1 writes X = 10 –P2 reads X –P2 writes X = X+5 –P1 reads X Now use RQ=2, WQ=4 Value 3 Version 1 Value 3 Version 1 Value 3 Version 1 Value 3 Version 1 P1 Value 3 Version 1 P2
55 More Info Causally consistent lazy replication (Wuu1984) Efficient Solutions to the Replicated Log and Dictionary Problem, by Wuu G. T. and Bernstein A. J., Readings in client-centric consistency http://www.cs.ubc.ca/grads/resources/thesis/Nov03/Sunny_Ho.pdf The End