1. Logic and Lattices for Distributed Programming Neil Conway UC Berkeley Joint work with: Peter Alvaro, Peter Bailis, David Maier, Bill Marczak, Joe Hellerstein, Sriram Srinivasan Basho Chats #004 June 27, 2012

2. Programming

3. Distributed Programming

4. Dealing with Disorder Introduce order –Paxos, Zookeeper, Two-Phase Commit, … –“Strong Consistency” Tolerate disorder –Correct behavior in the face of many possible network orders –Typical goal: replicas converge to same final state “Eventual Consistency”

5. Eventual Consistency PopularHard to program

6. Help developers build reliable programs on top of eventual consistency

7. This Talk 1. Theory –CRDTs, Lattices, and CALM 2. Practice –Programming with Lattices –Case Study: KVS

8. Read: {Alice, Bob} Write: {Alice, Bob, Dave} Write: {Alice, Bob, Carol} Students {Alice, Bob, Dave} Students {Alice, Bob, Carol} Client 0 Client 1 Read: {Alice, Bob} Students {Alice, Bob} How to resolve? Students {Alice, Bob}

9. Proble m Replicas perceive different event orders GoalSame final state at all replicas Solutio n Commutative operations (“merge functions”)

10. Students {Alice, Bob, Carol, Dave} Client 0 Client 1 Merge = Set Union

11. Commutative Operations Used by Dynamo, Riak, Bayou, etc. Formalized as CRDTs: Convergent and Commutative Replicated Data Types –Shapiro et al., INRIA (2009-2012) –Based on join semilattices –Commutative, associative, idempotent Practical libraries: Statebox, Knockbox

12. Time Set (Union) Integer (Max) Boolean (Or) “Growth”: Larger Sets “Growth”: Larger Numbers “Growth”: false  true

13. Client 0 Client 1 Students {Alice, Bob, Carol, Dave} Teams { } Read: {Alice, Bob, Carol, Dave} Read: { } Write: {, } Teams {, } Remove: {Dave} Students {Alice, Bob, Carol} Replica Synchronization Students {Alice, Bob, Carol} Teams {, }

14. Client 0 Client 1 Students {Alice, Bob, Carol, Dave} Teams { } Read: {Alice, Bob, Carol} Read: { } Teams { } Remove: {Dave} Students {Alice, Bob, Carol} Replica Synchronization Students {Alice, Bob, Carol} Nondeterministic Outcome! Teams { }

15. Possible Solution: Wrap both replicated values in a single complex CRDT

16. Goal: Compose larger application using “safe” mappings between simple lattices

17. Time Set (merge = Union) Integer (merge = Max) Boolean (merge = Or) size() >= 5 Monotone function from set  max Monotone function from max  boolean

18. Monotonicity in Practice “The more you know, the more you know” Never retract previous outputs (“mistake-free”) Typical patterns: immutable data accumulate knowledge over time threshold tests (“if” w/o “else”) Typical patterns: immutable data accumulate knowledge over time threshold tests (“if” w/o “else”)

19. Monotonicity and Determinism Agents strictly learn more knowledge over time Monotone: different learning order, same final outcome Result: Program is deterministic!

20. A program is confluent if it produces the same results regardless of network nondeterminism 20

21. A program is confluent if it produces the same results regardless of network nondeterminism 21

22. Consistency As Logical Monotonicity CALM Analysis 1.All monotone programs are confluent 2.Simple syntactic test for monotonicity Result: Simple static analysis for eventual consistency

23. Handling Non-Monotonicity … is not the focus of this talk Basic choices: 1.Nodes agree on an event order using a coordination protocol (e.g., Paxos) 2.Allow non-deterministic outcomes If needed, compensate and apologize

24. Putting It Into Practice What we’d like: Collection of agents No shared state (  message passing) Computation over arbitrary lattices

25. Bloom OrganizationCollection of agents CommunicationMessage passing StateRelations (sets) ComputationRelational rules over sets (Datalog, SQL)

26. BloomBloom L OrganizationCollection of agents CommunicationMessage passing StateRelations (sets)Lattices ComputationRelational rules over sets (Datalog, SQL) Functions over lattices

27. Quorum Vote in Bloom L QUORUM_SIZE = 5 RESULT_ADDR = "example.org" class QuorumVote include Bud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] lset :votes lmax :vote_cnt lbool :got_quorum end bloom do votes <= vote_chn {|v| v.voter_id} vote_cnt <= votes.size got_quorum <= vote_cnt.gt_eq(QUORUM_SIZE) result_chn <~ got_quorum.when_true { [RESULT_ADDR] } end Map set ! max Map max ! bool Threshold test on bool Lattice state declarations 27 Communication interfaces Accumulate votes into set Annotated Ruby class Program state Program logic Merge function for set lattice

28. Builtin Lattices NameDescription?a t bSample Monotone Functions lboolThreshold testfalse a ∨ b when_true() ! v lmaxIncreasing number 1max(a,b ) gt(n) ! lbool +(n) ! lmax -(n) ! lmax lminDecreasing number −1−1min(a,b)lt(n) ! lbool lsetSet of values;a [ bintersect(lset) ! lset product(lset) ! lset contains?(v) ! lbool size() ! lmax lpsetNon-negative set;a [ bsum() ! lmax lbagMultiset of values;a [ bmult(v) ! lmax +(lbag) ! lbag lmapMap from keys to lattice values empty map at(v) ! any-lat intersect(lmap) ! lmap 28

29. Case Study

30. Goal: Provably eventually consistent key-value store (KVS) Assumption: Map keys to lattice values (i.e., values do not decrease) Assumption: Map keys to lattice values (i.e., values do not decrease) Solution: Use a map lattice

31. Time Replica 1 Replica 2 Nested lattice value

32. Time Replica 1 Replica 2 Add new K/V pair

33. Time Replica 1 Replica 2 “Grow” value in extant K/V pair

34. Time Replica 1 Replica 2 Replica Synchronization

35. Goal: Provably eventually consistent KVS that stores arbitrary values Solution: Assign a version to each key-value pair Each replica stores increasing versions, not increasing values

36. Object Versions in Dynamo/Riak 1.Each KV pair has a vector clock version 2.Given two versions of a KV pair, prefer the one with the strictly greater version 3.If versions are incomparable, invoke user- defined merge function

37. Vector Clock: Map from node IDs  logical clocks Logical Clock: Increasing counter Solution: Use a map lattice Solution: Use an increasing-int lattice

38. Version-Value Pairs Pair = Pair merge(Pair o) { if self.fst > o.fst: self elsif self.fst < o.fst: o else new Pair(self.fst.merge(o.fst), self.snd.merge(o.snd)) }

39. Time Replica 1 Replica 2

40. Time Replica 1 Replica 2 Version increase; NOT value increase

41. Time Replica 1 Replica 2 R1’s version replaces R2’s version

42. Time Replica 1 Replica 2 New version @ R2

43. Time Replica 1 Replica 2 Concurrent writes!

44. Time Replica 1 Replica 2 Merge VC (automatically), value merge via user’s lattice (as in Dynamo)

45. Lattice Composition in KVS

46. Conclusion Dealing with EC Many event orders  order- independent (disorderly) programs LatticesDisorderly state Monotone Functions Disorderly computation Monotone Bloom Lattices + monotone functions for safe distributed programming

47. Questions Welcome Please try Bloom! http://www.bloom-lang.org Or: gem install bud

48. Backup Slides

49. Lattices hS,t,?i is a bounded join semi-lattice iff: –S is a partially ordered set –t is a binary operator (“least upper bound”) For all x,y 2 S, x t y = z where x · S z, y · S z, and there is no z’  z 2 S such that z’ · S z. Associative, commutative, and idempotent –? is the “least” element in S (8x 2 S: ? t x = x) 49 Example: increasing integers –S = Z, t = max, ? = -∞

50. Monotone Functions f : S  T is a monotone function iff 8a,b 2 S : a · S b ) f(a) · T f(b) 50 Example: size(Set) ! Increasing-Int size({A, B}) = 2 size({A, B, C}) = 3

51. From Datalog ! Lattices Datalog (Bloom)Bloom L StateRelationsLattices Example Values[[“red”, 1], [“green”, 2]]set: [“red”, “green”] map: {“red” => 1, “green” => 2} counter: 5 condition: false ComputationRules over relationsFunctions over lattices Monotone Computation Monotone rulesMonotone functions Program SemanticsFixpoint of rules (stratified semantics) Fixpoint of functions (stratified semantics) 51

52. Bloom Operational Model 52

53. QUORUM_SIZE = 5 RESULT_ADDR = "example.org" class QuorumVote include Bud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] table :votes, [:voter_id] scratch :cnt, [] => [:cnt] end bloom do votes <= vote_chn {|v| [v.voter_id]} cnt <= votes.group(nil, count(:voter_id)) result_chn = QUORUM_SIZE} end Quorum Vote in Bloom Communication Persistent Storage Transient Storage Accumulate votes Send message when quorum reached Not (set) monotonic! 53

54. Current Status WriteupsBloom L : UCB Tech ReportUCB Tech Report Bloom/CALM: CIDR’11, websiteCIDR’11website Lattice Runtime Available as a git branch To be merged soon-ish Examples, Case Studies KVS Shopping carts Causal delivery Under development: MDCC, concurrent editingMDCC