1. Logic and Lattices for Distributed Programming Neil Conway, William R. Marczak, Peter Alvaro, Joseph M. Hellerstein UC Berkeley David Maier Portland State University

2. Distributed Programming: Key Challenges Asynchrony Partial Failure

4. Dealing with Disorder Enforce global order –Paxos, Two-Phase Commit, GCS, … –“Strong Consistency” Tolerate disorder –Programmer must ensure correct behavior for many possible network orders –“Eventual Consistency” Typical goal: replicas converge to same final state

5. Dealing with Disorder Enforce global order –Paxos, Two-Phase Commit, GCS, … –“Strong Consistency” Tolerate disorder –Programmer must ensure correct behavior for many possible network orders –“Eventual Consistency” Typical goal: replicas converge to same final state

6. Goal: Make it easier to write programs on top of eventual consistency

7. This Talk 1.Prior Work –Convergent Modules (CRDTs) –Monotonic Logic (CALM) 2.Bloom L 3.Case Study

8. Read: {Alice, Bob} Write: {Alice, Bob, Dave} Write: {Alice, Bob, Carol} Students {Alice, Bob, Dave} Students {Alice, Bob, Carol} Client 0 Client 1 Read: {Alice, Bob} Students {Alice, Bob} How to resolve? Students {Alice, Bob}

9. Proble m Replicas perceive different event orders GoalSame final state at all replicas Solutio n Use commutative operations (“merge functions”)

10. Students {Alice, Bob, Carol, Dave} Client 0 Client 1 Merge = Set Union

11. Commutative Operations Common design pattern Formalized as CRDTs: Convergent and Commutative Replicated Data Types –Shapiro et al., INRIA (2009- 2012) –Based on join semilattices

12. Lattices hS,t,?i is a bounded join semilattice iff: –S is a set –t is a binary operator (“least upper bound”) Associative, commutative, and idempotent Induces a partial order on S: x · S y if x t y = y Informally, “merge function” for elements of S –? is the “least” element in S 8x 2 S: ? t x = x 12

13. Time Set (LUB = Union) Increasing Integer (LUB = Max) Boolean (LUB = Or)

14. Client 0 Client 1 Students {Alice, Bob, Carol, Dave} Teams { } Read: {Alice, Bob, Carol, Dave} Read: { } Write: {, } Teams {, } Remove: {Dave} Students {Alice, Bob, Carol} Replica Synchronization Students {Alice, Bob, Carol} Teams {, }

15. Client 0 Client 1 Students {Alice, Bob, Carol, Dave} Teams { } Read: {Alice, Bob, Carol} Read: { } Teams { } Remove: {Dave} Students {Alice, Bob, Carol} Replica Synchronization Students {Alice, Bob, Carol} Nondeterministic Outcome! Teams { }

16. Problem: Composition of CRDTs can result in non-determinism

17. Possible Solution: Encapsulate all distributed state in a single CRDT Hard to design, verify, and test Doesn’t scale with application size

18. Goal: Design a language that allows safe composition of CRDTs

19. Solution: … Datalog? Concurrent work: distributed programming using Datalog –P2 (2006-2010) –Bloom (2010-2012) Monotonic logic: building block for convergent distributed programs

20. Monotonic Logic As input set grows, output set does not shrink –“Retraction-free” Order independent e.g., map, filter, join, union, intersection Non-Monotonic Logic New inputs might retract previous outputs Order sensitive e.g., aggregation, negation

21. Monotonicity and Determinism Agents learn strictly more knowledge over time Different learning order, same final outcome Result: Program is deterministic!

22. Consistency As Logical Monotonicity CALM Analysis 1.All monotone programs are deterministic 2.Simple syntactic test for monotonicity Result: Whole-program static analysis for eventual consistency

23. Problem: CALM only applies to programs over growing sets Version NumbersTimestampsThreshold Tests

24. Quorum Vote A coordinator accepts votes from agents Count # of votes –When Count(Votes) > k, send “success” message

25. Quorum Vote A coordinator accepts votes from agents Count # of votes –When Count(Votes) > k, send “success” message Aggregation is non-monotonic!

26. CRDTs Limited scope (single object) Flexible types (any lattice) CALM Whole program analysis Limited types (only sets) Bloom L Whole program analysis Flexible types (any lattice)

27. Bloom L Constructs OrganizationCollection of agents CommunicationMessage passing StateLattices ComputationFunctions over lattices

28. Monotone Functions f : S  T is a monotone function iff 8a,b 2 S : a · S b ) f(a) · T f(b) 28

29. Time Set (LUB = Union) Increasing Integer (LUB = Max) Boolean (LUB = Or) size() >= 5 Monotone function from set  increase-int Monotone function from increase-int  boolean

30. Quorum Vote in Bloom L QUORUM_SIZE = 5 RESULT_ADDR = "example.org" class QuorumVote include Bud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] lset :votes lmax :vote_cnt lbool :got_quorum end bloom do votes <= vote_chn {|v| v.voter_id} vote_cnt <= votes.size got_quorum <= vote_cnt.gt_eq(QUORUM_SIZE) result_chn <~ got_quorum.when_true { [RESULT_ADDR] } end Monotone function: set ! max Monotone function: max ! bool Threshold test on bool (monotone) Lattice state declarations 30 Communication interfaces Accumulate votes into set Annotated Ruby class Program state Program logic Merge function for set lattice Monotonic  CALM

31. Bloom L Features Generalizes logic programming to lattices –Integration of relational-style queries and functions over lattices –Efficient incremental evaluation scheme Library of built-in lattices –Booleans, increasing/decreasing integers, sets, multisets, maps, … API for defining custom lattices

32. Case Studies Key-Value Store –Object versioning via vector clocks –Quorum replication Replicated Shopping Cart –Using custom lattice types to encode domain-specific knowledge

33. Case Studies Key-Value Store –Object versioning via vector clocks –Quorum replication Replicated Shopping Cart –Using custom lattice types to encode domain-specific knowledge

34. Case Study: Shopping Carts 34

35. Case Study: Shopping Carts 35

36. Case Study: Shopping Carts 36

37. Case Study: Shopping Carts 37

38. Perspectives on Shopping CRDTs –Individual server replicas converge Bloom –Checkout is non-monotonic  requires distributed coordination Built-in Bloom L lattice types –Checkout is not a monotone function of any of the built-in lattices

39. Observation: Once a checkout occurs, no more shopping actions can be performed

40. Observation: Each client knows when a checkout can be processed “safely”

41. Monotone Checkout OPS = [1] Incomplete OPS = [2] Incomplete OPS = [3] Incomplete OPS = [1,2] Incomplete OPS = [2,3] Incomplete OPS = [1,2,3] Complete 41

42. Monotone Checkout 42

43. Monotone Checkout 43

44. Monotone Checkout 44

45. Monotone Checkout 45

46. Shopping Takeaways Checkout summary is a monotone function of client’s activities Custom lattice type captures application- specific notion of “forward progress” –“Unsafe” state hidden behind ADT interface

47. Recap 1.How to build eventually consistent systems –Write disorderly programs 2.Disorderly state –Lattices 3.Disorderly computation –Monotone functions over lattices 4.Bloom L –Type system for deterministic behavior –Support for custom lattice types

48. Thank You! http://www.bloom-lang.net http://www.bloom-lang.net

49. Backup Slides

50. Strong Consistency in Industry “… there was a single overarching theme within the keynote talks… strong synchronization of the sort provided by a locking service must be avoided like the plague… [the key] challenge is to find ways of transforming services that might seem to need locking into versions that … can operate correctly without locking.” -- Birman et al., “Toward a Cloud Computing Research Agenda” (LADIS, 2009) 50

51. Bloom Operational Model 51

52. QUORUM_SIZE = 5 RESULT_ADDR = "example.org" class QuorumVote include Bud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] table :votes, [:voter_id] scratch :cnt, [] => [:cnt] end bloom do votes <= vote_chn {|v| [v.voter_id]} cnt <= votes.group(nil, count(:voter_id)) result_chn = QUORUM_SIZE} end Quorum Vote in Bloom Communication Persistent Storage Transient Storage Accumulate votes Send message when quorum reached Not (set) monotonic! 52 Count votes Annotated Ruby class Program state Program logic

53. Built-in Lattices NameDescription?a t bSample Monotone Functions lboolThreshold testfalse a ∨ b when_true() ! v lmaxIncreasing number 1max(a,b ) gt(n) ! lbool +(n) ! lmax -(n) ! lmax lminDecreasing number −1−1min(a,b)lt(n) ! lbool lsetSet of values;a [ bintersect(lset) ! lset product(lset) ! lset contains?(v) ! lbool size() ! lmax lpsetNon-negative set;a [ bsum() ! lmax lbagMultiset of values;a [ bmult(v) ! lmax +(lbag) ! lbag lmapMap from keys to lattice values empty map at(v) ! any-lat intersect(lmap) ! lmap 53

54. Failure Handling Great question! 1.Monotone programs handle transient faults very well –Deterministic  simple logging –Commutative, idempotent  simple recovery 2.Future work: “controlled non-determinism” –Timeout code is fundamentally non-deterministic –But we still want mostly deterministic programs

55. Handling Non-Monotonicity … is not the focus of this talk Basic alternatives: 1.Nodes agree on an event order using distributed coordination (e.g., Paxos) 2.Allow non-deterministic outcomes If needed, compensate and apologize