1. Phase Reconciliation for Contended In-Memory Transactions Neha Narula, Cody Cutler, Eddie Kohler, Robert Morris MIT CSAIL and Harvard

2. Cloud Computing and Databases Two trends: multicore and in-memory databases Multicore databases face increased contention due to skewed workloads and rising core counts

3. BEGIN Transaction ADD(x, 1) ADD(y, 2) END Transaction

4. Throughput on a Contentious Transactional Workload

6. Concurrency Control Forces Serialization core 0 core 1 core 2 ADD(x)ADD(y) time

7. BEGIN Transaction ADD(x, 1) ADD(y, 2) END Transaction

8. Multicore OS Scalable Counters core 0 core 1 core 2 x 0 = x 0 +1; x 1 = x 1 +1; x 2 = x 2 +1; Kernel can apply increments in parallel using per-core counters time

9. Multicore OS Scalable Counters core 0 core 1 core 2 x 0 = x 0 +1; x 1 = x 1 +1; x 2 = x 2 +1; print(x); x=x 0 +x 1 +x 2 ; print(x) To read per-core data, system has to stop all writes and reconcile x time

10. Can we use per-core data in complex database transactions?

11. BEGIN Transaction ADD(x, 1) ADD(y, 2) END Transaction BEGIN Transaction GET(x) GET(y) END Transaction

12. Challenges Deciding what records should be split among cores Transactions operating on some contentious and some normal data Different kinds of operations on the same records

13. Phase Reconciliation Database automatically detects contention to split a record among cores Database cycles through phases: split and joined OCC serializes access to non-split data Split records have assigned operations for split phases Doppel, an in-memory transactional database

14. Outline 1.Phases 2.Splittable operations 3.Doppel optimizations 4.Performance evaluation

15. Split Phase A transaction can operate on split and non-split data Split records are written to per-core (x) Rest of the records use OCC (u, v, w) OCC ensures serializability for the non-split parts of the transaction core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 split phase

16. Reconciliation Cannot correctly process a read of x in current state Abort read transaction Reconcile per-core data to global store core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x=x+x 0 x 0 =0 x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 x=x+x 1 x 1 =0 x=x+x 2 x 2 =0 split phase reconciliation

17. Joined Phase Wait until all processes have finished reconciliation Resume aborted read transactions using OCC Process all new transactions using OCC No split data core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x=x+x 0 x 0 =0 GET(x) GET(v) x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 x=x+x 1 x 1 =0 x=x+x 2 x 2 =0 split phase joined phase reconciliation

18. Resume Split Phase When done, transition to split phase again Wait for all processes to acknowledge they have completed joined phase before writing to per-core data core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x=x+x 0 x 0 =0 GET(x) GET(v) x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 ADD(x) ADD(v) x 1 =x 1 +1 x=x+x 1 x 1 =0 x=x+x 2 x 2 =0 ADD(x) ADD(v) x 2 =x 2 +1 split phase joined phase split phase reconciliation

19. Outline 1.Phases 2.Splittable operations 3.Doppel Optimizations 4.Performance evaluation

20. Transactions and Operations BEGIN Transaction (y) ADD(x,1) ADD(y,2) END Transaction Transactions are composed of one or more operations Only some operations are amenable to splitting

21. Supported Operations Splittable ADD(k,n) MAX(k,n) MULT(k,n) OPUT(k,v,o) TOPKINSERT(k,v,o ) Not Splittable GET(k) PUT(k,v)

22. MAX Example core 0 core 1 core 2 MAX(x,55) MAX(x,10) MAX(x,21) x 0 = MAX(x 0,55) x 1 = MAX(x 1,10) x 2 = MAX(x 2,21) x 0 = 55 x 1 = 10 x 2 = 21 Each core keeps one piece of summarized state x i MAX is commutative so results can be determined out of order

23. MAX Example core 0 core 1 core 2 MAX(x,55) MAX(x,10) MAX(x,21) x 0 = MAX(x 0,55) x 1 = MAX(x 1,10) x 2 = MAX(x 2,21) x 0 = 55 x 1 = 27 x 2 = 21 Each core keeps one piece of summarized state x i MAX is commutative so results can be determined out of order MAX(x,27) x 1 = MAX(x 1,27) MAX(x,2) x 1 = MAX(x 1,2) x = 55

24. MAX Example core 0 core 1 core 2 MAX(x,55) MAX(x,10) MAX(x,21) x 0 = MAX(x 0,55) x 1 = MAX(x 1,10) x 2 = MAX(x 2,21) x 0 = 55 x 1 = 27 x 2 = 21 Each core keeps one piece of summarized state x i MAX is commutative so results can be determined out of order MAX(x,27) x 1 = MAX(x 1,27) MAX(x,2) x 1 = MAX(x 1,2) x = 55

25. What Can Be Split? Operations that can be split must be: –Commutative –Pre-reconcilable –On a single key

26. What Can’t Be Split? Operations that return a value –ADD_AND_GET(x,4) Operations on multiple keys –ADD(x,y) Different operations in the same phase, even if arguments make them commutative –ADD(x,0) and MULT(x,1)

27. Outline 1.Phases 2.Splittable operations 3.Doppel Optimizations 4.Performance evaluation

28. Batching Transactions core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x=x+x 0 x 0 =0 GET(x) GET(v) x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 x=x+x 1 x 1 =0 x=x+x 2 x 2 =0 split phase joined phase reconciliation

29. Batching Transactions Don’t switch phases immediately; stash reads Wait to accumulate stashed transactions Batch for joined phase core 0 core 1 core 2 ADD(x) ADD(u) ADD(x) ADD(v) ADD(x) ADD(w) GET(x)GET( v) x=x+x 0 x 0 =0 GET(x) GET(v) x 0 =x 0 +1 x 1 =x 1 +1 x 2 =x 2 +1 x=x+x 1 x 1 =0 x=x+x 2 x 2 =0 split phase joined phase reconciliation GET(x) ADD(x) ADD(v) x 1 =x 1 +1

30. Ideal World Transactions with contentious operations happen in the split phase Transactions with incompatible operations happen correctly in the joined phase

31. How To Decide What Should Be Split Data Database starts out with no split data Count conflicts on records –Make key split if #conflicts > conflictThreshold Count stashes on records in the split phase –Move key back to non-split if #stashes too high

32. Outline 1.Phases 2.Splittable operations 3.Data classification 4.Performance evaluation

33. Performance Contention vs. parallelism What kinds of workloads benefit? A realistic application: RUBiS Doppel implementation: Multithreaded Go server; worker thread per core All experiments run on an 80 core intel server running 64 bit Linux 3.12 Transactions are one-shot procedures written in Go

34. Parallel Performance on Conflicting Workloads Throughput (millions txns/sec) 20 cores, 1M 16 byte keys, transaction: ADD(x,1)

35. Increasing Performance with More Cores 1M 16 byte keys, transaction: ADD(x,1) all writing same key

36. Varying Number of Hot Keys 20 cores, transaction: ADD(x,1)

37. How Much Stashing Is Too Much? 20 cores, transactions: LIKE read, LIKE write

38. RUBiS Auction application modeled after eBay –Users bid on auctions, comment, list new items, search 1M users and 33K auctions 7 tables, 26 interactions 85% read only transactions More contentious workload –Skewed distribution of bids –More writes

39. StoreBid Transaction BEGIN Transaction (bidder, amount, item) num_bids_item = num_bids_item + 1 if amount > max_bid_item: max_bid_item = amount max_bidder_item = bidder bidder_item_ts = Bid{bidder, amount, item, ts} END Transaction

40. StoreBid Transaction BEGIN Transaction (bidder, amount, item) ADD(num_bids_item,1) MAX(max_bid_item, amount) OPUT(max_bidder_item, bidder, amount) PUT(new_bid_key(), Bid{bidder, amount, item, ts}) END Transaction All commutative operations on potentially conflicting auction metadata

41. RUBiS Throughput Throughput (millions txns/sec) 20 cores, 1M users 33K auctions

42. Related Work Split counters in multicore OSes –Linux kernel Commutativity in distributed systems and concurrency control –[Shapiro ‘11] –[Li ‘12] –[Lloyd ‘12] –[Weihl ‘88] Optimistic concurrency control –[Kung ’81] –[Tu ‘13]

43. Summary Contention is rising with increased core counts Commutative operations are amenable to being applied in parallel, per-core We can get good parallel performance on contentious transactional workloads by combining per-core split data with concurrency control Neha Narula http://nehanaru.la @neha