1. Shyh-In Hwang in Yuan Ze University1 Chapter 6 Consistency and Replication

2. Shyh-In Hwang in Yuan Ze University2 Reasons for Replication To increase the reliability of a system Survive after one replica crashes Better protection against corrupted data Performance Scalability Numerous clients The size of geographical area

3. Shyh-In Hwang in Yuan Ze University3 Price of Replication Consistency of the replicas When and how to carry out modifications to all copies

4. Shyh-In Hwang in Yuan Ze University4 Object Replication (1) Organization of a distributed remote object shared by two different clients.

5. Shyh-In Hwang in Yuan Ze University5 Object Replication (2) a) A remote object capable of handling concurrent invocations on its own. b) An object adapter is required to handle concurrent invocations How to protect the object against simultaneous access by multiple clients

6. Shyh-In Hwang in Yuan Ze University6 Object Replication (3) a) A distributed system for replication-aware distributed objects. b) A distributed system responsible for replica management (More common. Simplicity for AP developers)

7. Shyh-In Hwang in Yuan Ze University7 Data-Centric Consistency Models The general organization of a logical data store, physically distributed and replicated across multiple processes.

8. Shyh-In Hwang in Yuan Ze University8 A contract between the processes and the data store If the processes agree to obey certain rules, the store promises to work correctly. Consistency Models

9. Shyh-In Hwang in Yuan Ze University9 W i (x) a A write by process P i to data item x with the value a R i (x) b A read from data item x by process P i returning the value b Data items are initially NIL Denotation

10. Shyh-In Hwang in Yuan Ze University10 Strict Consistency The most stringent consistency model Definition Any read on a data item x returns a value corresponding to the result of the most recent write on x.

11. Shyh-In Hwang in Yuan Ze University11 Strict Consistency W(x)a R(x)a Strictly consistent memory R(x)a R(x)NIL Not strictly consistent P1P1 P2P2

12. Shyh-In Hwang in Yuan Ze University12 Strict Consistency Machine AMachine B P2P1 T 2 : R 2 (x) T 1 : W 1 (x), where T 2 = T 1 + 1ns 3 meters C = 3 * 10 8 m/sec Propagation speed: 3 m / 10 -9 s = 3 * 10 9 m/sec In order to meet the strict consistency requirement

13. Shyh-In Hwang in Yuan Ze University13 Strict consistency is an ideal programming model; however, it ’ s nearly impossible to implement in a distributed system. Strict consistency may violate the laws of physics (Einstein ’ s special theory of relativity) It is impossible in a distributed system to assign a unique timestamp to each operation that corresponds to actual global time Strict Consistency

14. Shyh-In Hwang in Yuan Ze University14 Programmers can often live with weaker models. When the order of events is essential, semaphores or other synchronization tools should be used.

15. Shyh-In Hwang in Yuan Ze University15 Definition(Lamport, 1979) The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential order, and the operations of each individual process appear in this sequence in the order specified by its program. Sequential Consistency

16. Shyh-In Hwang in Yuan Ze University16 That is, when processes run concurrently on (possibly) different machines, any valid interleaving of read and write operations is acceptable behavior, but all processes see the same interleaving of operations. Nothing is said about time. There is no reference to the “ most recent ” write operation on an object. A process sees writes from all processes but only its own reads. Sequential Consistency

17. Shyh-In Hwang in Yuan Ze University17 Sequential Consistency P1P1 P2P2 W(x)a R(x) NIL R(x)a P1P1 P2P2 W(x)a R(x)a Two possible results of running the same program

18. Shyh-In Hwang in Yuan Ze University18 Sequential Consistency P1P1 P2P2 W(x)a R(x)bR(x)a P3P3 P4P4 W(x)b R(x)bR(x)a Sequentially consistent data store

19. Shyh-In Hwang in Yuan Ze University19 Sequential Consistency P1P1 P2P2 W(x)a R(x)bR(x)a P1P1 P2P2 W(x)b R(x)aR(x)b Not sequentially consistent

20. Shyh-In Hwang in Yuan Ze University20 Definition Definition of Sequential Consistency, plus If ts op1 (x) < ts op2 (y), then operation OP1(x) should precede OP2(y) in this sequence Ordering according to a set of loosely synchronized clocks with only finite precision Stronger than sequential consistency Linearizability

21. Shyh-In Hwang in Yuan Ze University21 Linearizability Why linearizability? To assist formal verification of concurrent algorithms

22. Shyh-In Hwang in Yuan Ze University22 Sequential Consistency Example: consider the following three parallel processes, assume that all statements are atomic. How many possible execution sequences are there? Process P1Process P2Process P3 x = 1; print ( y, z); y = 1; print (x, z); z = 1; print (x, y); Each statement is assumed to be indivisible

23. Shyh-In Hwang in Yuan Ze University23 Sequential Consistency Process P1Process P2Process P3 x = 1; print ( y, z); y = 1; print (x, z); z = 1; print (x, y); 6! / 2 / 2 / 2 = 90 x, y, and z are initially 0.

24. Shyh-In Hwang in Yuan Ze University24 Four Valid Results Accepted by Sequential Consistency P 1 executes both statements before P 2 or P 3 starts P 3 must complete before P 1 starts x = 1; print (y, z); y = 1; print (x, z); z = 1; print (x, y); Prints: 001011 Signature: 001011 (a) x = 1; y = 1; print (x,z); print(y, z); z = 1; print (x, y); Prints: 101011 Signature: 101011 (b) y = 1; z = 1; print (x, y); print (x, z); x = 1; print (y, z); Prints: 010111 Signature: 110101 (c) y = 1; x = 1; z = 1; print (x, z); print (y, z); print (x, y); Prints: 111111 Signature: 111111 (d)

25. Shyh-In Hwang in Yuan Ze University25 Sequential Consistency Signature: a 6-bit string of the output of P1, P2, P3 in that order, which characterizes a particular interleaving of statements 90 different valid statement orderings less than 64 valid program results under sequential consistency 000000 is not permitted 001001 is not permitted

26. Shyh-In Hwang in Yuan Ze University26 To Express Sequential Consistency Ahamad et all (1993) E1: W 1 (x)a(execution on x) E2: W 2 (x)b E3: R 3 (x)b, R 3 (x)a E4: R 4 (x)b, R 4 (x)a H: history Program order must be maintained Data coherence must be respected R(x) must return the value most recently written to x ( 指 排序非指時間 ) H = W 2 (x)b, R 3 (x)b, R 4 (x)b, W 1 (x)a, R 3 (x)a, R 4 (x)a

27. Shyh-In Hwang in Yuan Ze University27 Problem of sequential consistency: Poor performance (Lipton & Sandberg, 1988) r: read time w: write time t: minimal packet transfer time between nodes r + w  t For any sequentially consistent memory, changing the protocol to improve the read performance makes the write performance worse, and vice versa. Sequential Consistency

28. Shyh-In Hwang in Yuan Ze University28 Causal Consistency – Consider USENET posting example: The “ answer message ” comes before the “ question message, ” thus violate the causal consistency rules. Examples When there is any read followed later by any write, the two events are potentially causally related. A read is causally related to the write that provided the data the read got. If two processes spontaneously and simultaneously write two variables, these are not causally related. – Operations that are not causally related are said to be concurrent.

29. Shyh-In Hwang in Yuan Ze University29 Causal Consistency – Hutto and Ahamad, 1990, a causally consistent memory should obey following condition: Writes that are potentially causally related must be seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines. – Overhead: implementing causal consistency requires keeping track of which processes have seen which writes.

30. Shyh-In Hwang in Yuan Ze University30 Causal Consistency W(x)bR(x)a W(x)c R(x)cR(x)b R(x)c P1P1 P2P2 W(x)a P3P3 P4P4 concurrent This sequence is allowed with causally consistent memory, but not with sequentially consistent memory or strictly consistent memory.

31. Shyh-In Hwang in Yuan Ze University31 A violation of causal memory W(x)bR(x)a R(x)bR(x)a R(x)b P1P1 P2P2 W(x)a P3P3 P4P4 Wrong order Potentially causally related Causal Consistency

32. Shyh-In Hwang in Yuan Ze University32 A correct sequence of events in causal memory(concurrent writes can be seen in a different order on different machines) W(x)b R(x)bR(x)a R(x)b P1P1 P2P2 W(x)a P3P3 P4P4 Concurrent Causal Consistency

33. Shyh-In Hwang in Yuan Ze University33 – PRAM (Pipelined RAM) Consistency Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes. Easy to implement Require that writes originating in a single process be seen everywhere in order. All writes generated by different processes are concurrent FIFO Consistency

34. Shyh-In Hwang in Yuan Ze University34 FIFO Consistency Key difference between sequential & PRAM consistency Sequential consistency: although the order of statement execution and memory references is non-deterministic, at least all processes agree what it is; PRAM consistency: they need not agree

35. Shyh-In Hwang in Yuan Ze University35 A valid sequence of events for PRAM consistency FIFO Consistency R(x)aW(x)b R(x)bR(x)a P1P1 P2P2 W(x)a P3P3 P4P4 W(x)c R(x)c R(x)aR(x)b R(x)c

36. Shyh-In Hwang in Yuan Ze University36 FIFO Consistency 001001 is impossible with sequential consistency x = 1; print (y, z); y = 1; print(x, z); z = 1; print (x, y); Prints: 00 (a) x = 1; y = 1; print(x, z); print ( y, z); z = 1; print (x, y); Prints: 10 (b) y = 1; print (x, z); z = 1; print (x, y); x = 1; print (y, z); Prints: 01 (c) Seen by P1 Seen by P2Seen by P3

37. Shyh-In Hwang in Yuan Ze University37 FIFO Consistency Consider two parallel processes P1 & P2 running under different consistency model Sequential consistent: P1 is killed, P2 is killed, or neither is killed PRAM consistent: Both processes can be killed x = 1; if (y == 0) kill (P2) y = 1; if (x == 0) kill (P1) P1:P2:

38. Shyh-In Hwang in Yuan Ze University38 Not all applications require even seeing all writes, let alone seeing them in order E.g.: a process inside a critical section reading and writing some variables in a tight loop. Other processes are not supposed to touch the variables until the first process has left its critical section The memory has no way of knowing when a process is in a critical section and when it is not, so it has to propagate all writes to all memories in the usual way A better solution is to let the process finish its critical section and then make sure that the final results were sent everywhere. => Synchronization variable Problem of FIFO Consistency

39. Shyh-In Hwang in Yuan Ze University39 – Synchronization variable used to synchronize memory When a synchronization completes, all writes done on that machine are propagated outward and all writes done on other machines are brought in. All of shared memory is synchronized Weak Consistency

40. Shyh-In Hwang in Yuan Ze University40 Weak Consistency Property 1: Accesses to synchronization variables associated with a data store are sequentially consistent All processes see all accesses to synchronization variables in the same order.

41. Shyh-In Hwang in Yuan Ze University41 Weak Consistency Property 2: No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere Accessing a synchronization variable flushes the pipeline (writes).

42. Shyh-In Hwang in Yuan Ze University42 Weak Consistency Property 3: No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed. By doing a synchronization before reading shared data, a process can be sure of getting the most recent values.

43. Shyh-In Hwang in Yuan Ze University43 – Most useful when isolated accesses to shared variables are rare, with most coming in clusters (many accesses in a short period, then none for a long time). – We limit only the time when consistency holds – With Weak Consistency, Sequential consistency is enforced on groups of operations instead of on individual operations. Weak Consistency

44. Shyh-In Hwang in Yuan Ze University44 Weak Consistency Having memory be wrong is acceptable. Only when the function f is called does the compiler have to put the current values of a and b back in memory. int a, b, c, d, e, x, y;/* variables */ int *p, *q;/* pointers */ int f( int *p, int *q);/* function prototype */ a = x * x;/* a stored in register */ b = y * y;/* b as well */ c = a*a*a + b*b + a * b;/* used later */ d = a * a * c;/* used later */ p = &a;/* p gets address of a */ q = &b/* q gets address of b */ e = f(p, q)/* function call */

45. Shyh-In Hwang in Yuan Ze University45 Weak Consistency R(x)b R(x)a P1P1 P2P2 W(x)a P3P3 W(x)b S R(x)aR(x)bS S A valid sequence of events for weak consistency P1P1 P2P2 W(x)a W(x)b S SR(x)a Should be b An invalid sequence of events for weak consistency

46. Shyh-In Hwang in Yuan Ze University46 – Gharachorloo et al., 1990 Two kinds of synchronization variables or operations are provided instead of one in weak consistency. – acquire: a critical region is about to be entered. – release: a critical region has just been exited. Able to tell the difference between entering/leaving a critical region, thus more efficient than weak consistency. – Acquire & release do not have to apply to all of memory. Instead, they may only guard specific shared variables. The shared variables that are kept consistent are said to be protected. Release Consistency

47. Shyh-In Hwang in Yuan Ze University47 A valid event sequence for release consistency Release Consistency Acq(L) W(x)a W(x)bRel(L) Acq(L)R(x)bRel(L) P1P1 P2P2 P3P3 R(x)a

48. Shyh-In Hwang in Yuan Ze University48 – It is also possible to use barriers instead of critical regions with release consistency A barrier is a synchronization mechanism that prevents any process from starting phase n + 1 of a program until all processes have finished phase n. When a process arrives at a barrier, it must wait until all other processes get there too. When the last one arrives, all shared variables are synchronized and then all processes are resumed. Departure from the barrier is done on an acquire and arrival is done on a release. Release Consistency

49. Shyh-In Hwang in Yuan Ze University49 – When the software does an acquire: All the local copies of the protected variables are brought up to date to be consistent with the remote ones if need be. Doing an acquire does not guarantee that locally made changes will be sent to other machines immediately – When the release is done: Protected variables that have been changed are propagated out to other machines. Doing a release does not necessarily import changes from other machines. Release Consistency

50. Shyh-In Hwang in Yuan Ze University50 Release Consistency Rules: Before a read or write operation on shared data is performed, all previous acquires done by the process must have completed successfully. Before a release is allowed to be performed, all previous reads and writes by the process must have completed Accesses to synchronization variables are FIFO consistent (sequential consistency is not required).

51. Shyh-In Hwang in Yuan Ze University51 – Eager release consistency When a release is done, the process doing the release pushes out all the modified data to all other processes that already have a copy. Eager vs. Lazy Release Consistency

52. Shyh-In Hwang in Yuan Ze University52 – Lazy release consistency(Keleher et al., 1992) At the time of release, nothing is sent anywhere. When an acquire is done, the process trying to do the acquire has to get the most recent values of the data from the process or processes holding them. A timestamp protocol can be used to determine which variables have to be transmitted. More efficient: no network traffic until another process does an acquire Repeated acquire-release pairs by the same process in the absence of outside competition are free. E.g. a critical region located inside a loop Eager vs. Lazy Release Consistency

53. Shyh-In Hwang in Yuan Ze University53 Entry Consistency Each synchronization variable has a current owner (process). The owner can enter and exit critical sections repeatedly without having to send any messages on the network. for loop { … Acq(Lx) ; W(x)I ; Rel(Lx) ; … }

54. Shyh-In Hwang in Yuan Ze University54 Entry Consistency Other processes has to ask the owner to change ownership, and the latest values. Effect: all accesses are sequentially consistent Several processes may simultaneously own a synchronization variable in nonexclusive mode (for read, but not write).

55. Shyh-In Hwang in Yuan Ze University55 An acquire access of a synchronization variable is not allowed to perform with respect to a process until all updates to the guarded shared data have been performed with respect to that process. At an acquire, all guarded shared data have been brought up to date. Rules for Entry Consistency (1)

56. Shyh-In Hwang in Yuan Ze University56 Before an exclusive mode access to a synchronization variable by a process is allowed to perform with respect to that process, no other process may hold the synchronization variable, not even in nonexclusive mode. Before updating a shared data item, a process must enter a critical region in exclusive mode so that no other process can update it at the same time. Rules for Entry Consistency (2)

57. Shyh-In Hwang in Yuan Ze University57 After an exclusive mode access to a synchronization variable has been performed, any other process's next nonexclusive mode access to that synchronization variable may not be performed until it has performed with respect to that variable's owner. If a process wants to enter a critical region in nonexclusive mode, it must check with the owner to fetch the most recent copies of the guarded shared data. Rules for Entry Consistency (3)

58. Shyh-In Hwang in Yuan Ze University58 Entry Consistency A valid event sequence for entry consistency.

59. Shyh-In Hwang in Yuan Ze University59 Requires each ordinary shared data items to be associated with some synchronization variable. When an acquire is done on a synchronization variable, only those data guarded by that synchronization variable are made consistent. Lazy release consistency: does not associate shared data items with locks or barriers and at acquire time has to determine empirically which variables it needs. Entry Consistency

60. Shyh-In Hwang in Yuan Ze University60 – Advantages Reduces the overhead associated with acquiring and releasing a synchronization variable, since only a few shared data items have to be synchronized. Allows multiple critical sections involving disjoint shared data to execute simultaneously, increasing the amount of parallelism. – Disadvantages Extra overhead and complexity of associating every shared data item with some synchronization variable. Programming this way is also more complicated and error prone. Entry Consistency

61. Shyh-In Hwang in Yuan Ze University61 Summary of Consistency Models Strict 12345 Interleaving according to absolute time All processes see the same interleaving 12345 12345 P1 P2 12345 P1, P2 sees

62. Shyh-In Hwang in Yuan Ze University62 Summary of Consistency Models Sequential A process sees writes from all processes, but only its own reads. Time does not play a role 12345 Any valid interleaving is OK All processes see the same interleaving 12345 1123452345 P1 P2 ( and : writes) P1, P2 sees

63. Shyh-In Hwang in Yuan Ze University63 Summary of Consistency Models Linearizable It is similar to strict consistency Using synchronized clocks, rather than absolute time. 12345 Interleaving according to synchronized clocks All processes see the same interleaving 12345 12345 P1 P2 12345 Absolute time & after Synchronization Synchronized clock 12345 P2 Absolute time P1, P2 sees

64. Shyh-In Hwang in Yuan Ze University64 Summary of Consistency Models Causal 12 345 Different processes may see different interleaving 12 345 345345 P1 P2 ( and : writes) P1 sees 345345 P2 sees 1212 1212 Potentially causally related Consurrent

65. Shyh-In Hwang in Yuan Ze University65 Summary of Consistency Models FIFO All writes generated by different processes are concurrent 12345 Any valid interleaving is OK Different processes may see different interleaving 12345 1123452345 P1 P2 ( and : writes) P1 sees 1123452345 P2 sees

66. Shyh-In Hwang in Yuan Ze University66 Summary of Consistency Models Consistency models not using synchronization operations. ConsistencyDescription StrictAbsolute time ordering of all shared accesses matters. Linearizability All processes must see all shared accesses in the same order. Accesses are furthermore ordered according to a (nonunique) global timestamp Sequential All processes see all shared accesses in the same order. Accesses are not ordered in time Causal All processes see causally-related shared accesses in the same order. FIFO All processes see writes from a single process in the order they were used. Writes from different processes may not always be seen in that order

67. Shyh-In Hwang in Yuan Ze University67 Summary of Consistency Models Models with synchronization operations. ConsistencyDescription WeakShared data can be counted on to be consistent only after a synchronization is done ReleaseShared data are made consistent when a critical region is exited EntryShared data pertaining to a critical region are made consistent when a critical region is entered.

68. Shyh-In Hwang in Yuan Ze University68 Client-Centric Consistency Models Characteristics Lack of simultaneous updates Most operations are read A very weak consistency model: eventual consistency Can be relatively cheap to hide inconsistencies

69. Shyh-In Hwang in Yuan Ze University69 Eventual Consistency DNS Only one authority is allowed to update its domain No write-write conflicts read-write conflicts: acceptable to propagate updates in a lazy fashion

70. Shyh-In Hwang in Yuan Ze University70 Eventual Consistency WWW Only one authority is allowed to update its site No write-write conflicts Browsers and Web proxies cache pages, which may be out-of-date

71. Shyh-In Hwang in Yuan Ze University71 Eventual Consistency Inconsistency exists and is tolerable in the large-scale distributed and replicated DB Eventual consistency If no updates take place for a long time, all replicas will gradually become consistent.

72. Shyh-In Hwang in Yuan Ze University72 Eventual Consistency The principle of a mobile user accessing different replicas of a distributed database.

73. Shyh-In Hwang in Yuan Ze University73 Eventual Consistency Client-centric consistency Guarantees for a single client concerning the consistency of accesses to a data store by that client No guarantees concerning concurrent accesses by different clients

74. Shyh-In Hwang in Yuan Ze University74 Client-centric consistency How we take care of “ Eventual ” Monotonic reads consistency Monotonic writes consistency Read your writes consistency Writes follow reads consistency

75. Shyh-In Hwang in Yuan Ze University75 Monotonic Reads Definition If a process reads the value of a data item x, any successive read operation on x by the same process will always return that same value or a more recent value. E.g., A user reads emails in San Francisco. Later, he flies to NYC and opens his mailbox again.

76. Shyh-In Hwang in Yuan Ze University76 Monotonic Reads The read operations performed by a single process P at two different local copies of the same data store. a) A monotonic-read consistent data store b) A data store that does not provide monotonic reads. WS(x1;x2): WS(x1) is part of WS(x2) 1:mail from David (2:Mail from Linda) 1:mail from David 2:Mail from Linda

77. Shyh-In Hwang in Yuan Ze University77 Monotonic Writes Definition A write operation by a process on a data item x is completed before any successive write operation on x by the same process. E.g., updates on software library Required when part of the state is updated Not necessary when each write operation completely overwrites the value of x

78. Shyh-In Hwang in Yuan Ze University78 Monotonic Writes Monotonic writes resembles data-centric FIFO consistency Write operations by the same process are performed in the correct order everywhere Difference: we only consider consistency for a single process here rather than concurrent processes in FIFO

79. Shyh-In Hwang in Yuan Ze University79 Monotonic Writes The write operations performed by a single process P at two different local copies of the same data store a) A monotonic-write consistent data store. b) A data store that does not provide monotonic-write consistency. Update to Ver1.01 Update to Ver1.02 Update to Ver1.01

80. Shyh-In Hwang in Yuan Ze University80 Monotonic Writes A weaker form The effects of a write operation are seen only if all preceding writes have been carried out, but perhaps not in the order in which they have been originally initiated. Applicable when write operations are commutative

81. Shyh-In Hwang in Yuan Ze University81 Read Your Writes Definition The effect of a write operation by a process on a data item x will always be seen by a successive read operation on x by the same process. E.g., absence of read-your-writes consistency “ Read your writes ” is not guaranteed when a web browser caches an old copy. Updating passwords may take time

82. Shyh-In Hwang in Yuan Ze University82 Read Your Writes a) A data store that provides read-your-writes consistency. b) A data store that does not.

83. Shyh-In Hwang in Yuan Ze University83 Writes Follow Reads Definition A write operation by a process on a data item x following a previous read operation on x by the same process, is guaranteed to take place on the same or a more recent value of x that was read. Can guarantee that users see a posting of a reaction to an article only after they have seen the original article.

84. Shyh-In Hwang in Yuan Ze University84 Writes Follow Reads a) A writes-follow-reads consistent data store b) A data store that does not provide writes-follow-reads consistency Q: 今天天氣如何？ A: 今天艷陽高照 Q: 今天天氣如何？

85. Shyh-In Hwang in Yuan Ze University85 Distribution Protocols Replica placement Update propagation Epidemic protocols

86. Shyh-In Hwang in Yuan Ze University86 Replica Placement The logical organization of different kinds of copies of a data store into three concentric rings.

87. Shyh-In Hwang in Yuan Ze University87 Permanent Replicas Number of permanent replicas is small Distribution of web servers: more or less statically configured LAN Mirroring: mirror sites

88. Shyh-In Hwang in Yuan Ze University88 Server-Initiated Replicas Push caches: to install temporary replicas in regions where sudden bursts of requests are coming from Web hosting services Dynamically replicating files to servers close to clients

89. Shyh-In Hwang in Yuan Ze University89 Server-Initiated Replicas Counting access requests from different clients. P is the server in the Web Hosting Service near C1 & C2 (by looking up routing database).

90. Shyh-In Hwang in Yuan Ze University90 Server-Initiated Replicas del Q (F) rep Q (F) req Q (F) To remove F To migrate F If cnt Q (P, F) > 0.5 * req Q (F), then migrate F from Q to P If cnt Q (P, F) > Threshold * req Q (F), then replicate F to P Remove F from Q To replicate F req Q (F) > rep Q (F) req Q (F) < del Q (F) & is not last copy repQ(F) > req Q (F) > del Q (F) Condition How

91. Shyh-In Hwang in Yuan Ze University91 Client-Initiated Replicas Client caches To improve access time to data Kept for limited time A nearby client cache, might be shared between clients on the same LAN In the same department In the same region To place cache servers at specific points in WAN

92. Shyh-In Hwang in Yuan Ze University92 Update Propagation Update operations on a distributed and replicated data store are generally Initiated at a client Forwarded to one of the copies The update is propagated to other copies

93. Shyh-In Hwang in Yuan Ze University93 Update Propagation What is propagated A notification of an update Invalidation protocol Little bandwidth Best for Read/Write ratio is low Transfer data from one copy to another Best for Read/Write ratio is high Propagate the update operation to other copies Active replication Perform updates at minimum bandwidth cost Require more processing power

94. Shyh-In Hwang in Yuan Ze University94 Pull versus Push Protocols Push-based approach Server-based protocols Often used between permanent replicas and server-initiated replicas Good when Read/Update ratio is high

95. Shyh-In Hwang in Yuan Ze University95 Pull versus Push Protocols Pull-based approach Client-based protocols Often used by client caches The client polls the server to see whether an update is needed Good when Read/Update ratio is low Drawback: the response time increases in the case of a cache miss

96. Shyh-In Hwang in Yuan Ze University96 Pull versus Push Protocols A comparison between push-based and pull-based protocols in the case of multiple client, single server systems. IssuePush-basedPull-based State of serverList of client replicas and cachesNone Messages sent Update (and possibly fetch update later if invalidation is used) Poll and update Response time at client Immediate (or fetch-update time if invalidation is used) Fetch-update time

97. Shyh-In Hwang in Yuan Ze University97 Pull versus Push Protocols Lease A promise that server will push updates to the client for a specified time Dynamically switching between push-based and pull-based strategy

98. Shyh-In Hwang in Yuan Ze University98 Lease (1) Age-based leases Long-lasting lease to data items that are expected to remain unmodified can reduce number of (lease) update messages (2) Renewal-frequency based leases Long-lasting lease to clients whose cache often needs refreshed Short-term lease for clients who request occasionally Server takes care of clients where its data are popular (3)State-space based leases Lowers the expiration time of new leases when server becomes overloaded gradually Dynamically switch to a more stateless mode

99. Shyh-In Hwang in Yuan Ze University99 Unicasting vs Multicasting Cheaper to use multicasting facilities E.g., all replicas are in the same LAN Multicasting + push-based approach Unicasting + pull-based approach

100. Shyh-In Hwang in Yuan Ze University100 Epidemic Protocols Update propagation in eventual- consistent data stores is often implemented by epidemic protocols Propagating updates to all replicas in as few messages as possible Advantage: scalability Updates are often aggregated into a single message

101. Shyh-In Hwang in Yuan Ze University101 Anti-Entropy Propagation Model A server P picks server Q at random, and exchanges updates with Q P only pushes its own updates to Q P only pulls in new updates from Q P and Q send updates to each other (push- pull approach) More effecitve

102. Shyh-In Hwang in Yuan Ze University102 Rumor Spreading (Gossiping) To ensure that updates are spread quickly => pushing updates to a number of servers Method An infective P contacts an arbitrary other server Q and tries to push the update to Q If Q is already updated, P stops spreading with some probability A really good way to rapidly spread updates

103. Shyh-In Hwang in Yuan Ze University103 Epidemic Protocols Removing data is hard e.g., chained letter ( 幸運信 ) Will easily be infected again: deletion of a data item destroys all information on the item. When receiving an old copy, a server will interpret as updates on something it did not have before. Solution: death certificate Deletion of a data item as an update Old copies are treated as old versions, and will not be spread

104. Shyh-In Hwang in Yuan Ze University104 Epidemic Protocols Problem of death certificate The certificates will be piled up Should be cleaned up eventually Solution: dormant death certificates Death certificates are timestamped with creation time Assumption: updates propagate to all servers within a known finite time. The death certificates are then removed. Keep a few dormant death certificates Spreading death certificates again when needed

105. Shyh-In Hwang in Yuan Ze University105 Consistency Protocols Describes an implementation of a specific consistency model Primary-based protocols Remote-write protocols Primary-based remote-write protocol (single copy) Primary-backup protocol (multiple copies) Local-write protocols: the primary copy migrates Primary-based local-write (single copy) Primary-backup (multiple copies) Replicated-write protocols Active Replication (all copies) Quorum-based protocols (some copies only)

106. Shyh-In Hwang in Yuan Ze University106 Primary-based remote-write protocol with a fixed server to which all read and write operations are forwarded. Remote-Write Protocols (1): Primary-based remote-write protocol

107. Shyh-In Hwang in Yuan Ze University107 Implementation of seqential consistency Process always see the effects of most recent write operations Remote-Write Protocols (2): Primary-backup protocol

108. Shyh-In Hwang in Yuan Ze University108 A single copy is migrated between processes. Local-Write Protocols (1): Primary-based local-write protocol

109. Shyh-In Hwang in Yuan Ze University109 Local-Write Protocols (1): Primary-based local-write protocol A fully distributed non-replicated version of the data store Problems: location of data items Local area: broadcasting Forwarding pointers Home-based approaches Large-scale and widely-distributed data stores: hierarchical location service

110. Shyh-In Hwang in Yuan Ze University110 The primary migrates to the process wanting to perform an update. Local-Write Protocols (2): Primary-backup protocol

111. Shyh-In Hwang in Yuan Ze University111 Local-Write Protocols (2): Primary-backup protocol Advantage Perform write operations locally, while reading processes can still access their local copies Mobile computers Mobile computer becomes the primary server Disconnected: all updates are performed locally Other processes can perform read operations Connected: propagation of updates from the primary to backups

112. Shyh-In Hwang in Yuan Ze University112 Replicated-Write Protocols Write operations can be carried out at multiple replicas, instead of only one Active replication: an operation is forwarded to all replicas Quorum-based: majority voting

113. Shyh-In Hwang in Yuan Ze University113 Active Replication Operations need to be carried out in the same order everywhere Lamport timestamps: does not scale well in large distributed systems Sequencer A central coordinator: assigns a unique sequence number (  primary-based consistency protocols)

114. Shyh-In Hwang in Yuan Ze University114 Active Replication (1) The problem of replicated invocations. Transfer $100,000

115. Shyh-In Hwang in Yuan Ze University115 Active Replication (2) a) Forwarding an invocation request from a replicated object. b) Returning a reply to a replicated object. The invocation request is assigned the same unique identifier by each replica of B

116. Shyh-In Hwang in Yuan Ze University116 Quorum-Based Protocols Three examples of the voting algorithm: a) A correct choice of read and write set b) A choice that may lead to write-write conflicts c) A correct choice, known as ROWA (read one, write all)

117. Shyh-In Hwang in Yuan Ze University117 Quorum-Based Protocols N R : read quorum N W : write quorum All servers in write quorum get the new version of data and the new version number N R + N W > N: to prevent read-write conflicts N W > N/2: to prevent write-write conflicts