Replication: optimistic approaches Marc Shapiro with Yasushi Saito (HP Labs) Cambridge Distributed Systems Group

Bases de Données Avancées Replication: optimistic approaches 1 Motivations for this work Peer-to-peer, decentralised write sharing Lessons and commonalities Understand limitations Different solutions: spectrum or discrete points? Simple formal model

Bases de Données Avancées Replication: optimistic approaches 2 Optimistic replication Replicas of shared objects on sites Without synchronisation: peer-to-peer read and update! Consistency: a posteriori, offline Merge independent updates Applications: high latency networks disconnected operation cooperative work Improves availability & performance

Bases de Données Avancées Replication: optimistic approaches 3 Example: cooperative engineering with CVS CVS: developing shared code Local, disconnected replica: no interference Conflicts: Write same file = syntactic Overlap in file = violates edit semantics Doesn’t compile, test = violates application semantics Both sides of a conflict are excluded Manual repair

Bases de Données Avancées Replication: optimistic approaches 4 Example: Bayou General-purpose database Any replica can update, log actions action = { dependency check, operation, merge-procedure } Optimistic replication: epidemic exchange logs { roll-back, replay }*; commit dep-check: semantic check for conflict merge-proc: semantic repair

Bases de Données Avancées Replication: optimistic approaches 5 Basic vocabulary While isolated: tentative updates When connected, reconcile: Propagate & collect updates (Conceptually) Restart from initial state Replay updates (if possible) Overriding goal: consistency

1. Consistency Study component issues of consistency

Bases de Données Avancées Replication: optimistic approaches 7 What is consistency? Consistent with user intents apply operations according to user scenario Consistent with data invariants dependent actions pre- and post-conditions conflict resolution Replicas consistent with each other converge towards same values

Bases de Données Avancées Replication: optimistic approaches 8 Consistency: problem taxonomy 1. Objects & updates Internal vs. external consistency Value / value log / operation log Single master / multi-master 2. Detecting dependence vs. concurrency 3. Concurrency control 4. Laziness of concurrency control Pessimistic / advanced concurrency / optimistic 5. Convergence

Bases de Données Avancées Replication: optimistic approaches 9 Operation-based reconciliation Updates: concurrent, unsynchronised Local log of actions = operation descriptions object identifier, method, arguments Multi-log collects local + remote logs Reconciliation schedule: merge multi-log & run sequentially Scheduling issues: Include vs. exclude Execution order

Bases de Données Avancées Replication: optimistic approaches 10 Operation-based model

Bases de Données Avancées Replication: optimistic approaches 11 Dependence vs. concurrency Two actions are either have a dependency or commutative / concurrent Dependent actions: do not conflict must be scheduled in dependence order Concurrent actions potentially conflict Dependence / concurrency detection is a fundental mechanism

Bases de Données Avancées Replication: optimistic approaches 12 Concurrency control Concurrent & no conflict  commute: execute both, arbitrary order Conflict detection options Conflict resolution options

Bases de Données Avancées Replication: optimistic approaches 13 Convergence Liveness: sites receive same/all actions Safety: given same actions, sites compute the same value Stability: actions eventually not undone

2. Dependency & Concurrency Mechanisms to detect if actions are dependent or concurrent

Bases de Données Avancées Replication: optimistic approaches 15 Scalar clocks and timestamps Wall clock, Lamport clock Total order Total order, consistent with causal dependence Schedule in timestamp order Can’t detect concurrency

Bases de Données Avancées Replication: optimistic approaches 16 Happens-before e 1 precedes e 2 in process e 1 sends, e 2 receives  e 1  e 2  (e 1  e 2 )   (e 2  e 1 )  e 1 || e 2 e 1 || e 2 : e 1 does not cause e 2 e 1  e 2 : e 1 might cause e 2 Partial order, consistent with causal dependence Schedule consistent with 

Bases de Données Avancées Replication: optimistic approaches 17 Syntactic vs. semantic mechanisms Scalar timestamps no concurrency detection very conservative approx. of causality Vector timestamps detect concurrency conservative approx. of causality Alternative: explicit semantic constraints

Bases de Données Avancées Replication: optimistic approaches 18 Locks as semantic constraints Read(x) depends on previous Write(x) in same process, or previously-received Write(x), whichever is later Write(x) depends on previous Read(*) in same process More semantic information than Happens- Before Step in the right direction, but still too coarse

Bases de Données Avancées Replication: optimistic approaches 19 IceCube: Primitive constraints Declarative (“static”): MustHave: a  b if a  s and a  b then b  s (not necessarily contiguous nor in order) Order: a  b if a, b  s and a  b then a before b in s (not necessarily both nor contiguous) Within log, across logs Imperative (dynamic): preCondition (State)

Bases de Données Avancées Replication: optimistic approaches 20 Log constraints parcel predecessor- successor alternatives Express user intents: Predecessor/successor: a  b  b  a b uses effect of a; “a causes b” Parcel: a  b  b  a  transaction Alternatives: a  b  b  a

3. Concurrency control & scheduling Policies for dealing with concurrent actions

Bases de Données Avancées Replication: optimistic approaches 22 Optimistic concurrency control & scheduling Two actions are either: dependent  schedule in dependence order concurrent and non-conflicting or commutative  schedule in any order concurrent and conflicting  schedule in non-conflicting order  or exclude one, the other, or both

Bases de Données Avancées Replication: optimistic approaches 23 Concurrency control Concurrent & no conflict  commute: execute both, arbitrary order Conflict detection options: 2 concurrent actions conflict only if operate on same object only if both write only if violate semantic invariant Conflict resolution options: exclude both exclude 1st, include 2nd (or vice-versa) execute both in favorable order (rewrite and execute both)

Bases de Données Avancées Replication: optimistic approaches 24 What is a conflict? 1 site executes code + pre/post-conditions Pre/post-conditions often unknown Dependency between successive actions Schedule execution must satisfy pre/post- conditions Violation  conflict pre(x 0 )post(x 0, f(x 0 )) x 1 := f(x 0 ) pre(x 0 )post(x’ 1, g(x 0 )) x’ 1 := g(x 0 ) pre(x 1 )post(x 1, g(x 1 )) x 2 := g(x 1 )

Bases de Données Avancées Replication: optimistic approaches 25 Thomas’ Write Rule Pre- / post-conditions unknown Scalar clocks no concurrency detect implicit concurrency control schedule in clock order a later action excludes earlier ones Lost updates Delete ambiguity: “tombstone” state

Bases de Données Avancées Replication: optimistic approaches 26 Value-based Version Vector concurrency control Pre- / post-conditions unknown Independent objects actions to different objects commute VV = per-object vector timestamp any concurrent writes to object conflict Resolution: Manual Values: “Resolver” per data type

Bases de Données Avancées Replication: optimistic approaches 27 Bayou scheduling Disjoint databases; 1 primary / database Transaction: single database Action = { dependency check, operation, merge-procedure } Optimistic replication: epidemic exchange logs { roll-back, replay }*; commit Conflict  dependency check fails  merge-procedure

Bases de Données Avancées Replication: optimistic approaches 28 Bayou dependency checks Write-write conflicts: on replay check that data unchanged Read-write conflicts: check input data can detect concurrent updates semantic: only relevant changes Application-specific checks bank account balance > £100 fine grain

Bases de Données Avancées Replication: optimistic approaches 29 IceCube: Object constraints Shared data type advertises static semantics mutually exclusive a  b  b  a best order (e.g. bank: credits before debits) a  b Only between concurrent actions Also: dynamic constraints commute best order mutually exclusive

Bases de Données Avancées Replication: optimistic approaches 30 IceCube scheduling Insight: conflict: choice of which action to exclude maximise value

Bases de Données Avancées Replication: optimistic approaches 31 IceCube execution model log constraints object constraints dynamic constraints

Bases de Données Avancées Replication: optimistic approaches 32 Search vs. syntactic order

Bases de Données Avancées Replication: optimistic approaches 33 Performance of IceCube heuristics

4. Convergence Can a peer-to-peer system converge? Hard in the general case Formalise to understand limitations, trade-offs

Bases de Données Avancées Replication: optimistic approaches 35 Convergence Liveness: sites receive same/all operations epidemic multicast quickly Safety: sites compute the same value equivalent schedules Stability: actions eventually not undone stable schedules Users, external world dependency Garbage collection

Bases de Données Avancées Replication: optimistic approaches 36 Schedule soundness & equivalence s sound: Closed for MustHave a  s  a  b  b  s Consistent with Order (a,b  s  a  b)  a before b in s Equivalence: s  t s, t sound a  s  a  t ordering is irrelevant!

Bases de Données Avancées Replication: optimistic approaches 37 Stability Peer-to-peer, indefinite tentative update + advisory reconciliation OK But stability needed: Users, external world depend on it Garbage collect multilog Stable: eventually decisions not changed committed: definitely included in all schedules aborted: definitely excluded

Bases de Données Avancées Replication: optimistic approaches 38 Correctness of stability Actions known to be stable at site i: stable i = committed i  aborted i Live:  action a, site i:  a  stable i Safe:  site i, schedule s i : s i sound  committed i  s i  site i,k: committed i  aborted k =  Safety invariant: strong, global!

Bases de Données Avancées Replication: optimistic approaches 39 Maintaining disjointness  site i,k: committed i  aborted k =  Different possibilities Unilateral abort TWR, Holliday 2000 Unilateral commit Deterministic abort / commit rule TWR Primary (only one) site decides Bayou, CVS Consensus before deciding Deno, Holliday

Bases de Données Avancées Replication: optimistic approaches 40 Maintaining soundness  site i, schedule s i : s i sound  committed i  s i When aborting a, also abort actions that MustHave a When committing a, also abort uncommitted actions that are ‘Order’ed before a Maintain both soundness and disjointness. Peer-to-peer commitment is hard!

Bases de Données Avancées Replication: optimistic approaches 41 Stability with TWR Independent objects Independent writes (no MustHave nor Order) All sites take same decision: Given two writes to same object, abort the earlier Whether concurrent or not Write stable when seen by all sites Disjointness: committed i =  Soundness: no MustHave (no transactions)

Bases de Données Avancées Replication: optimistic approaches 42 Stability in Bayou Databases: Disjoint Independent: no multi-DB transaction 1 primary / database Log constraints: transactions, time order Disjointness: Only 1 site decides about a: the primary for the database that a updates Soundness: whole transaction commits or aborts

Bases de Données Avancées Replication: optimistic approaches 43 Holliday’s pre-commit protocol Log constraints: multi-object transactions happens-before order Read transactions commit locally Read-Write transactions: consensus to commit convert locks to intentions pre-commit, vote commit if quorum ‘yes’ abort if anti-quorum ‘no’ or conflict with committed

Bases de Données Avancées Replication: optimistic approaches 44 Trade-offs Deterministic rule fast, inflexible Partition + primary single point of failure no MustHave across partition boundaries Consensus slow scalability impossibility of consensus in asynchronous systems with failure

5. Conclusions

Bases de Données Avancées Replication: optimistic approaches 46 Need for OR not going away “Network technology improving: keep everything consistent pessimistically.” True, but: Constant latency; unavailable bandwidth Mobile access  unbounded latency Increasing numbers of replicas “Conflicts are rare.” True, but: Do occur Very high cost

Bases de Données Avancées Replication: optimistic approaches 47 OR pros & cons Peer-to-peer read/write sharing OR accepts more updates: Performance despite latency Availability despite failures Increased complexity Semantic information Not transparent Bottleneck moved to commit Hard to make peer-to-peer Unless (unacceptable?) restrictions Unavoidable

Bases de Données Avancées Replication: optimistic approaches 48 The end