Telex/IceCube: a distributed memory with tunable consistency Marc Shapiro, Pierre Sutra, Pierpaolo Cincilla INRIA & LIP6, Regal group Nuno Preguic ̧ a Universidade Nova de Lisboa

1 Telex/IceCube It is: A distributed shared memory Object-based (high-level operations) Transactional (ACID… for some definition of ID) Persistent Top level design goal: availability ∞-optimistic ⇒ (reconcile ⇒ cascading aborts) Minimize aborts ⇒ non-sequential schedules (Only) consistent enough to enforce invariants Partial replication Distributed TMs — 22-Feb-2012

Scheduling (∞-optimistic execution) Sequential execution In (semantic) dependence order Dynamic checks for preconditions: if OK, application approves schedule Conflict ⇒ fork Latest checkpoint + replay Independent actions not rolled back Semantics: Conflict = non-commuting, antagonistic Independent 2Distributed TMs — 22-Feb-2012

3 Shared Calendar Application Telex lifecycle Application operations Telex appt +action(appt) User requests > application: actions, dependence > Telex: add to ACG, transmit

4Distributed TMs — 22-Feb-2012 Telex lifecycle Schedule Telex sched Shared Calendar Application ACG > Telex: valid path = sound schedule > application: tentative execute > application: if OK, approve approve

5Distributed TMs — 22-Feb-2012 Telex lifecycle Conflict ⇒ multiple schedules Telex !!! sched1 sched2 Shared Calendar Application ACG > Telex: valid path = sound schedule > application: tentative execute > application: if OK, approve approve2

6Distributed TMs — 22-Feb-2012 Constraints (Semantics-based conflict detection) Action: reified operation Constraint: scheduling invariant Binary relations: NotAfter    Enables (implication)    NonCommuting    Combinations: Antagonistic    Atomic    Causal dependence        Action-constraint graph ACG

7Distributed TMs — 22-Feb-2012 Single site scheduling Sound schedule: Path in the ACG that satisfies constraints Fork: Antagonism    NonCommuting + Dynamic checks    Several possible schedules Penalise lost work Optimal schedule: NP-hard IceCube heuristics

8Distributed TMs — 22-Feb-2012 Minimizing aborts (IceCube heuristics) Iteratively pick a sound schedule Application executes, checks, approves Check invariants Reject if violation Request alternative schedule Restart from previous Approved schedule Preferred for future Proposed to agreement

9Distributed TMs — 22-Feb-2012 /users/shapiro/mydoc Marc Pierre Ruby Logging and transmission = multilog Avoid write contention: per-doc, per-writer logs Append to my document log Transmit sequentially Read all document logs Ruby

10Distributed TMs — 22-Feb-2012 getCstrt Telex lifecycle Receive remote operations Telex ACG +cstrt(antg) Shared Calendar Application User requests > application: actions, dependence > Telex: add to ACG, transmit, merge

11Distributed TMs — 22-Feb-2012 Eventual consistency Common stable prefix Diverge beyond Combine approved schedules ⇒ Consensus on next extension of prefix Equivalence, not equality 0 0 0

12Distributed TMs — 22-Feb-2012 Telex lifecycle Convergence protocol Telex agreement Shared Calendar Application approved schedules > Telex: exchange, agree > commit/abort, serialise

13 Fast Gen. Genuine Consensus Throughput on LAN (1024 regs.) [Sutra 2010] Generalised Paxos Fast Paxos Paxos FGGC ☺ Distributed TMs — 22-Feb-2012

14 Special-case commutative commands ⇒ collision recovery Improvements higher on WAN. Generalised Paxos Fast Paxos Paxos FGGC WAN typical ☺ Distributed TMs — 22-Feb-2012 FGGC

FGGC: Varying commutativity Each command reads or writes a randomly-chosen register; WAN Paxos FGGC 1 register FGGC registers ☺ Distributed TMs — 22-Feb-2012

16Distributed TMs — 22-Feb-2012 Concurrency control: application safety assertions Dynamic test by application, ex.: bank account > 0 file system is a proper tree "(…(…(…)…(…))…)" Application approves/disapproves schedule at run time

17Distributed TMs — 22-Feb-2012 Example: Sakura Shared Calendar over Telex Private calendar + common meetings Example proposals: M1: Marc & Lamia & JeanMi, Monday | Tuesday | Friday M2: Lamia & Marc & Pierre, Tuesday | Wed. | Friday M3: JeanMi & Pierre & Marc, Monday | Tues. | Thurs. Change M3 to Friday Telex: (Local) Explore solutions, approve (Global) Combine, Commit Marc  Lamia Tues.  Fri. M1  M2 Run-time check

18Distributed TMs — 22-Feb-2012 Example ACG: calendar application Schedule: sound cut in the graph

19Distributed TMs — 22-Feb-2012 Status Open source 35,000 Java (well-commented) lines of code Available at gforge.inria.fr, BSD licensegforge.inria.fr, Applications: Sakura co-operative calendar (INRIA) Decentralised Collaborative Environment (UPC) Database, STMBench7 (UOC) Co-operative text editor (UNL, INRIA, UOC) Co-operative ontology editor (U. Aegean) Co-operative UML editor (LIP6)

20Distributed TMs — 22-Feb-2012 Lessons learned Separation of concerns: Application: business logic – Semantics: constraints, run-time checks System: Persistence, replication, consistency – Optimistic: availability, WAN operation – Adapts to application requirements – Constraints + run-time machine learning Modular ⇒ efficiency is an issue High latency, availability ⇒ reduce messages Commutative operations ⇒ CRDTs Partial replication is hard!

21Distributed TMs — 22-Feb-2012

22Distributed TMs — 22-Feb

23Distributed TMs — 22-Feb-2012 Application design MVC-like execution loop + rollback Designing for replication is an effort: Commute (no constraints) >> Antagonistic >> Non-Commuting Weaken invariants Make invariants explicit

24Distributed TMs — 22-Feb-2012 IceCube heuristic superior

25Distributed TMs — 22-Feb-2012 Multilog avoids write contention

26Distributed TMs — 22-Feb-2012 Decomposing ACG improves performance

27Distributed TMs — 22-Feb-2012 Designing applications for replication Replication is beneficial in many information sharing scenarios Understand your application needs Design for replication: Aim for commutativity Weaken invariants Make invariants explicit Telex: platform for decentralised, mutable, persistent shared data

28Distributed TMs — 22-Feb-2012 P2P collaborative work Shared documents Multiple masters Different locations, times Reconciliation: Isolated operation, diverge Detect & resolve conflicts Converge Conflict = invariant violated Telex: platform to ease application design & implementation Lamia Marc JeanMi shared document

29Distributed TMs — 22-Feb-2012 Goals Separation of concerns Support multiple interaction styles On-line WYSIWIS = pessimistic vs. Isolation, disconnected = optimistic Operate on local replica, replay remote ops Detect conflicts, choose, roll back Multi-object operations Safe: enforce application invariants Converge, commit Synchronise: only as required by application invariants (no more, no less)

30Distributed TMs — 22-Feb-2012 Competition Co-operative text editing: All operations assumed to commute (no invariants) Single object Total order: Database, atomic broadcast Safe, slow, doesn't scale Peer-to-peer: File sharing, database replication: scalable Single object Last Writer Wins (LWW): unsafe! Telex: principled, general-purpose, multi-object, adapts to application requirements, P2P