1. IM NTU Distributed Information Systems 2004 Replication Management -- 1 Replication Management Yih-Kuen Tsay Dept. of Information Management National Taiwan University

2. IM NTU Distributed Information Systems 2004 Replication Management -- 2 Motivations for Replication Performance enhancement –Client vs. server caching –Server pools –Replication of immutable vs. changing data Increased availability –Server failures –Network partition and disconnected operation Fault tolerance: guarantee correctness in spite of faults

3. IM NTU Distributed Information Systems 2004 Replication Management -- 3 General Requirements Replication transparency –Clients are not aware of multiple physical copies (replicas) of an object. –Clients see one logical copy for each object. Consistency –Servers perform operations in a way that meets the specification of correctness.

4. IM NTU Distributed Information Systems 2004 Replication Management -- 4 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. An Architecture for Replication Management

5. IM NTU Distributed Information Systems 2004 Replication Management -- 5 About the Servers Recoverability State Machines –Consist of state variables and commands –Outputs determined by the sequence of requests processed Static vs. dynamic set of replica managers –Dynamic: servers may crash; new ones may join –Static: crashed servers are considered to cease operating (possibly for an indefinite period)

6. IM NTU Distributed Information Systems 2004 Replication Management -- 6 Phases of Request Processing Issuance –unicast or multicast (from the front end to replica managers) Coordination (to ensure consistency) –FIFO ordering, causal ordering, total ordering, … Execution (maybe tentatively) Agreement (to commit or abort) Response –From one replica manager or several replica managers to the front end * The ordering of the phases varies for different systems.

7. IM NTU Distributed Information Systems 2004 Replication Management -- 7 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Services for Process Groups

8. IM NTU Distributed Information Systems 2004 Replication Management -- 8 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. View-Synchronous Group Communications

9. IM NTU Distributed Information Systems 2004 Replication Management -- 9 Correctness Criteria Linearizability Sequential consistency * Consider individual operations (instead of transactions).

10. IM NTU Distributed Information Systems 2004 Replication Management -- 10 Linearizability The interleaved sequence of operations meets the specification of a single correct copy of the objects. The order of operations in the interleaving is consistent with the real times at which the operations occurred in the actual execution.

11. IM NTU Distributed Information Systems 2004 Replication Management -- 11 Sequential Consistency The one-copy semantics of the replicated objects is respected. The order of operations is preserved for each client, i.e., consistent with the program order for each client. * Every linearizable service is also sequentially consistent.

12. IM NTU Distributed Information Systems 2004 Replication Management -- 12 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Consistency is easily guaranteed if the replica managers are organized as a group and the primary uses view-synchronous group communication to send updates. The Primary-Backup (Passive) Model

13. IM NTU Distributed Information Systems 2004 Replication Management -- 13 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Each front end sends its requests one at a time to all replica managers using a totally ordered multicast primitive, ensuring that all requests are processed in the same order at all replica managers. Active Replication

14. IM NTU Distributed Information Systems 2004 Replication Management -- 14 The Gossip Architecture A framework for providing high availability of service through lazy replication A request normally executed at one replica Replicas updated by lazy exchange of gossip messages (containing most recent updates).

15. IM NTU Distributed Information Systems 2004 Replication Management -- 15 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Operations in a Gossip Service

16. IM NTU Distributed Information Systems 2004 Replication Management -- 16 Timestamps Each front end keeps a vector timestamp reflecting the latest version accessed. The timestamp is attached to every request sent to a replica. Two front ends may exchange messages directly; these messages also carry timestamps. The merging of timestamps is done as usual.

17. IM NTU Distributed Information Systems 2004 Replication Management -- 17 Timestamps (cont.) Each replica keeps a replica timestamp representing those updates it has received. It also keeps a value timestamp, reflecting the updates in the replicated value. The replica timestamp is attached to the reply to an update, while the value timestamp is attached to the reply to a query.

18. IM NTU Distributed Information Systems 2004 Replication Management -- 18 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Timestamp Propagations

19. IM NTU Distributed Information Systems 2004 Replication Management -- 19 The Update Log Every update, when received by a replica, is recorded in the update log of the replica. Two reasons for keeping a log: –The update cannot be applied yet; it is held back. –It is uncertain if the update has been received by all replicas. The entries are sorted by timestamps.

20. IM NTU Distributed Information Systems 2004 Replication Management -- 20 The Executed Operation Table The same update may arrive at a replica from a front end and in a gossip message from another replica. To prevent an update from being applied twice, the replica keeps a list of identifiers of the updates that have been applied so far.

21. IM NTU Distributed Information Systems 2004 Replication Management -- 21 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. A Gossip Replica Manager

22. IM NTU Distributed Information Systems 2004 Replication Management -- 22 Processing Query Requests A query request q carries a timestamp q.prev, reflecting the latest version of the value that the front end has seen. Request q can be applied (i.e., it is stable) if q.prev  valueTS (the value timestamp of the replica that received q). Once q is applied, the replica returns the current valueTS along with the reply.

23. IM NTU Distributed Information Systems 2004 Replication Management -- 23 Processing Update Requests For an update u (not a duplicate), replica i –increments the i-th element of its replica timestamp replicaTS by one, –adds an entry to the log with a timestamp ts derived from u.prev by replacing the i-th element with that of replicaTS, and –return ts to the front end immediately. When the stability condition u.prev  valueTS holds, update u is applied and its ts is merged with valueTS.

24. IM NTU Distributed Information Systems 2004 Replication Management -- 24 Processing Gossip Messages For every gossip message received, a replica does the following: –Merge the arriving log with its own; duplicated updates are discarded. –Apply updates that have become stable. A gossip message need not contain the entire log, if it is certain that some of the updates have been seen by the receiving replica.

25. IM NTU Distributed Information Systems 2004 Replication Management -- 25 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Updates in Bayou

26. IM NTU Distributed Information Systems 2004 Replication Management -- 26 About Bayou Consistency guarantees Merging of updates Dependency checks Merge procedures

27. IM NTU Distributed Information Systems 2004 Replication Management -- 27 Coda vs. AFS More general replication Greater tolerance toward server crashes Allowing disconnected operations

28. IM NTU Distributed Information Systems 2004 Replication Management -- 28 A replicated transactional service should appear the same as one without replicated data. The effects of transactions performed by various clients on replicated data are the same as if they had been performed one at a time on single data items; this property is called one-copy serializability. Transactions with Replicated Data

29. IM NTU Distributed Information Systems 2004 Replication Management -- 29 Failures should be serialized with respect to transactions. Any failure observed by a transaction must appear to have happened before the transaction started. Transactions with Replicated Data (cont.)

30. IM NTU Distributed Information Systems 2004 Replication Management -- 30 Schemes for One-Copy Serializability Read one/write all Available copies replication Schemes that also tolerate network partitioning: –available copies with validation –quorum consensus –virtual partition

31. IM NTU Distributed Information Systems 2004 Replication Management -- 31 Source: Instructor’s guide for G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. B A Client + front end BBB A A getBalance(A) Client + front end Replica managers deposit(B,3); U T Transactions on Replicated Data

32. IM NTU Distributed Information Systems 2004 Replication Management -- 32 Available Copies Replication A client's read request on a logical data item may be performed by any available replica, but a client's update request must be performed by all available replicas. A local validation procedure is required to ensure that any failure or recovery does not appear to happen during the progress of a transaction.

33. IM NTU Distributed Information Systems 2004 Replication Management -- 33 Source: Instructor’s guide for G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. A X Client + front end P B Replica managers deposit(A,3); UT deposit(B,3); getBalance(B) getBalance(A) Replica managers Y M B N A B Available Copies Replication (cont.)

34. IM NTU Distributed Information Systems 2004 Replication Management -- 34 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Network Partition

35. IM NTU Distributed Information Systems 2004 Replication Management -- 35 Available Copies with Validation The available copies algorithm is applied within each partition. When a partition is repaired, the possibly conflicting transactions that took place in the separate partitions are validated. If the validation fails, some of the transactions have to be aborted.

36. IM NTU Distributed Information Systems 2004 Replication Management -- 36 Quorum Consensus Methods One way to ensure consistency across different partitions is to make a rule that operations can only be carried out within one of the partitions. A quorum is a subgroup of replicas whose size gives it the right to execute operations. Version numbers or timestamps may be used to determine whether copies of the data item are up to date.

37. IM NTU Distributed Information Systems 2004 Replication Management -- 37 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. An Example for Quorum Consensus

38. IM NTU Distributed Information Systems 2004 Replication Management -- 38 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Two Network Partitions

39. IM NTU Distributed Information Systems 2004 Replication Management -- 39 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Virtual Partition

40. IM NTU Distributed Information Systems 2004 Replication Management -- 40 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Overlapping Virtual Partitions

41. IM NTU Distributed Information Systems 2004 Replication Management -- 41 Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition. Creating Virtual Partitions