1. 1 Mobisnap: a database system for mobile computing Nuno Preguiça DI – FCT Universidade Nova de Lisboa Project partially supported by Praxis XXI/FCT/MCT LSDCS,CITI,UNL + GSD,UMinho

2. 2 Motivation Widespread use of mobile computers/devices and wireless communications Periods of disconnection (from server) No connectivity (no network or network failure) Limited energy (batteries) Economic factors Unacceptable latency (physics) Service overload (during demand peaks) Many applications use relational databases

3. 3 Goal System to support independent operation Users should be able to read/write data independently (i.e. without coordination) Data structured according to the relational data model Based on standards – SQL and PL/SQL

4. 4 Basic approach Optimistic replication Clients partially replicate data Users may submit transactions in disconnected clients

5. 5 Optimistic replication Database initial state in server

6. 6 Optimistic replication Client 1 caches objects

7. 7 Optimistic replication Client 2 caches objects

8. 8 Optimistic replication User 2 submits an operation

9. 9

10. 10 Optimistic replication User 1 submits operation

11. 11 Optimistic replication User 1 submits operation

12. 12 Optimistic replication Client 1 uploads changes

13. 13 Optimistic replication Client 1 uploads changes Operation can be replayed without problems because the object has the same value as seen in the client

14. 14 Optimistic replication Client 1 uploads changes Operation can be replayed without problems because the object has the same value as seen in the client

15. 15 Optimistic replication Client 2 uploads changes

16. 16 Optimistic replication Client 2 uploads changes Conflict !!! The object has been modified concurrently

17. 17 Optimistic replication Client 2 uploads changes Conflict !!! The object has been modified concurrently Problem 1 Detect conflicts

18. 18 Optimistic replication Client 2 uploads changes Conflict !!! The object has been modified concurrently Problem 1 Detect conflicts Problem 2 Solve conflicts

19. 19 Optimistic replication Client 2 uploads changes Conflict !!! The object has been modified concurrently Problem 1 Detect conflicts Problem 2 Solve conflicts

20. 20 Optimistic replication Client 2 uploads changes Conflict !!! The object has been modified concurrently Problem 1 Detect conflicts Problem 2 Solve conflicts Problem 3 Results in clients must be considered tentative

21. 21 Outline Introduction Optimistic replication and problems Middleware architecture Mobile transactions Reservations Reconciliation Final remarks

22. 22

23. 23 Single server Maintains official data version

24. 24 Multiple clients Maintain database snapshots Two versions: committed and tentative Can read/write the database

25. 25 Legacy clients Access database directly

26. 26 Middleware architecture

27. 27 Mobile transactions Small PL/SQL programs Subset of PL/SQL with some extensions/changes Major restriction: no loops, no cursors Programs are tentatively executed in the mobile devices Programs are executed (again) in the server to obtain the final result Transactions are not integrated using the read/write sets of client execution

28. 28 DECLARE l_stock INTEGER; l_price FLOAT; BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); NOTIFY( ‘SMS’, ‘351927435456’,‘Order added for product red…’); COMMIT (80,l_price); ENDIF; SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 47; IF l_price = 10 THEN -- update stock, add order record, send notification COMMIT (47,l_price); ENDIF; NOTIFY( ‘SMS’, ‘351927435456’,‘Order failed…’); ROLLBACK; END;

29. 29 DECLARE l_stock INTEGER; l_price FLOAT; BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); NOTIFY( ‘SMS’, ‘351927435456’,‘Order added for product red…’); COMMIT (80,l_price); ENDIF; SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 47; IF l_price = 10 THEN -- update stock, add order record, send notification COMMIT (47,l_price); ENDIF; NOTIFY( ‘SMS’, ‘351927435456’,‘Order failed…’); ROLLBACK; END; Precise conflict detection rules Encode users’ intents Reduce false conflicts

30. 30 DECLARE l_stock INTEGER; l_price FLOAT; BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); NOTIFY( ‘SMS’, ‘351927435456’,‘Order added for product red…’); COMMIT (80,l_price); ENDIF; SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 47; IF l_price = 10 THEN -- update stock, add order record, send notification COMMIT (47,l_price); ENDIF; NOTIFY( ‘SMS’, ‘351927435456’,‘Order failed…’); ROLLBACK; END; Conflict resolution rules Explore semantic information and users’ intents E.g. alternative actions

31. 31 DECLARE l_stock INTEGER; l_price FLOAT; BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); NOTIFY(‘SMS’,‘35127435456’,‘Order added for product red…’); COMMIT (80,l_price); ENDIF; SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 47; IF l_price = 10 THEN -- update stock, add order record, send notification COMMIT (47,l_price); ENDIF; NOTIFY( ‘SMS’, ‘351927435456’,‘Order failed…’); ROLLBACK; END; Notify function Sends the final result to the user (that may not be using the system when the final result is determined)

32. 32 Reservations Goal: guarantee the final result of mobile transactions independently A reservation provides some guarantee about the future state of the database

33. 33 Escrow reservation Exclusive right to use a share of a partitionable resource represented by a numerical item E.g. the stock of some product may be split among several users Implements escrow model in SQL with some changes x: x>min => x i : x i >min  x=  (x i -min i ) Traditional: x: x>min => x i : x i >min i  x=  x i  min =  min i

34. 34 Value-use reservation Right to use some value for a given data item (despite its value when the transaction is executed) E.g. a salesperson may use a reserved price for some product Select statements will return the reserved value

35. 35 Lock-like reservations Value-change Exclusive right to modify some data item Traditional fine-grain write lock E.g. a user may reserve the right to change the passenger’s name for a given seat in a train Slot Exclusive right to modify records that conform some given condition Similar to predicate lock (locks records that exist and that do not exist) E.g. a user may reserve the right to schedule a meeting in a room in a given period of time

36. 36 Shared lock-like reservations Shared value-change reservation Guarantees that it is possible to modify some data item E.g. used to guarantee that a counter may be incremented Shared slot reservation Guarantees that it is possible to modify data items with some given values E.g. used to guarantee insert operations in an append-only table

37. 37 Reservation model Clients obtain (leased) reservation before disconnecting The server enforces the granted reservations updating the database and setting appropriate triggers System transparently verifies if transactions can be guaranteed by the available reservations no special operations are used to access reserved data items all reservations are used If a transaction is guaranteed in the client, its execution in the server will follow the expected path (if reservations do not expire)

38. 38 Example: product request BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END;

39. 39 Example: product request Client gets reservations BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=8015>=0--- 45-2value-useproductspriceid=8096.5---45-1

40. 40 Example: product request select statement BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=8015>=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2

41. 41 Example: product request IF instruction BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=8015>=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2 Condition is TRUE All involved variables are guaranteed by reservations

42. 42 Example: product request update statement BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=80 15 >=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2 Update can be guaranteed by the escrow reservation Updates reservation and database snapshots

43. 43 Example: product request update statement BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=80 5 >=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2 Update can be guaranteed by the escrow reservation Updates reservation and database snapshots

44. 44 Example: product request insert statement BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=80 5 >=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2 Insert can not be guaranteed Execution proceeds

45. 45 Example: product request commit statement BEGIN SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT (80,l_price); ENDIF; ROLLBACK; END; idtypetablecolumnconditionvalueinfolinked 45-1escrowproductsstockid=80 5 >=0--- 45-2value-useproductspriceid=8096.5---45-1 l_stock=15; reserved; 45-1 l_price=96.5; reserved; 45-2 Transactions ends Result: reservation commit; level=read Meaning: the execution in the server will follow the same execution path but write statements may be blocked (the insert statement was not guaranteed)

46. 46 Transaction execution in the client 1 st – checks if it can be guaranteed If the execution ends in a commit statement the result is reservation commit, level: Full All statements could be guaranteed Read Pre-condition It was possible to guaranteed the result of IF statements in the depth-first execution path Alternative pre-condition It was possible to guarantee an alternative execution path (some IF statement could not be guaranteed)

47. 47 Transaction execution in the client 2 nd – when it can not be guaranteed If the execution requires some uncached data item, the result is unknown If the first execution ends in a rollback statement, the transaction is re-executed against the tentative data version If execution ends in a commit statement, the result is tentative commit If execution ends in a rollback statement, the result is tentative commit

48. 48 Transaction execution in the server Re-executes the transactions For reserved transactions Unlocks data items of used reservations Guaranteed reads need special handling (the same value must be returned as the result of a select statement) “Select” may read multiple columns – the same must be returned

49. 49 Remarks Reservations may lead to unnecessary aborts Re-execution mechanism + reconciliation Clients have to obtain the right reservations before disconnecting Application-dependent

50. 50 Mobile sales application

51. 51 Mobile sales application Good prediction Local guarantee: > 90% maximum Tend to 100% as usage rate deviates from 100% Immediate guarantee: > 95% maximum No prediction Local guarantee: > 80% maximum Immediate guarantee: > 95% maximum

52. 52 Contributions Multiple types of reservations Necessary to guarantee most transactions Implemented in a SQL-based system Client transparently verifies if reservations can be used Transactions written as usual Integrated with mobile transactions Middleware approach Legacy clients can continue to access database as usual

53. 53 Outline Introduction Optimistic replication and problems Middleware architecture Mobile transactions Reservations Reconciliation Final remarks

54. 54 Reconciliation The execution order may influence the the result and the number of transactions that can be executed successfully Transactions waiting re-execution Client may upload transactions from other clients

55. 55 Reconciliation example 9am room A or 9am room B Log 1 9am room A Log 2 9am room B or 9am room C

56. 56 Reconciliation example Possible schedules 9am room A or 9am room B 9am room A 9am room B or 9am room C 9am room A or 9am room B 9am room A 9am room B or 9am room C Log 1Log 2

57. 57 Reconciliation example Possible schedules 9am room A or 9am room B 9am room A 9am room B or 9am room C 9am room A or 9am room B 9am room A 9am room B or 9am room C 9am room A or 9am room B 9am room A 9am room B or 9am room C Log 1Log 2

58. 58 Reconciliation approach Based on IceCube (MSR,Cambridge) Goal: maximize the set of transactions executed with success Reconciliation process seen as an optimization problem Set of (semantic) static relations defined among transactions Heuristic search explores static information

59. 59 Log relations (constraints) Applications may explicitly set the following relations among transactions Predecessor-successor Alternatives May be extracted from single transaction code Parcel Original IceCube

60. 60 Object relations (constraints) From the transaction code, the following relations are inferred Overlaps Commutes Makes impossible T1 write make T2 preconditions false Helps T1 writes helps T2 preconditions to be true Prejudices

61. 61 Reconciliation algorithm New schedule created iteratively. In each step: Selects a transaction (using an heuristic based on the static relations) Executes the transaction Reconciliation creates new schedules until a good solution is found

62. 62 How to infer relations? 1 st – extract information Analyze each program statically, extracting from paths leading to commits: (semantic) read set select statements used in writes/preconditions (semantic) write set Update, insert, delete statements (semantic) preconditions Conditions in if statements leading to a commit

63. 63 SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT; ELSE ROLLBACK; ENDIF;

64. 64 SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; -- l_stock = read(products,id=80,stock) -- l_price = read(products,id=80,price) IF l_price = 10 THEN UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT; ELSE ROLLBACK; ENDIF; ReadSet = {}

65. 65 SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; -- l_stock = read(products,id=80,stock) -- l_price = read(products,id=80,price) IF l_price = 10 THEN -- as the contrary leads to a rollback -- precond(read(products,id=80,stock)>=10) -- precond(read(products,id=80,price)<=100) UPDATE products SET stock = stock - 10 WHERE id = 80; INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT; ELSE ROLLBACK; ENDIF; Read set only includes read used in pre-conditions and writes ReadSet={read(products,id=80,stock), read(products,id=80,price)}

66. 66 SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; -- l_stock = read(products,id=80,stock) -- l_price = read(products,id=80,price) IF l_price = 10 THEN -- as the contrary leads to a rollback -- precond(read(products,id=80,stock)>=10) -- precond(read(products,id=80,price)<=100) UPDATE products SET stock = stock - 10 WHERE id = 80; -- update(products,id=80,stock,stock-10) INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); COMMIT; ELSE ROLLBACK; ENDIF; ReadSet={read(products,id=80,stock), read(products,id=80,price)}

67. 67 SELECT stock,price INTO l_stock,l_price FROM products WHERE id = 80; -- l_stock = read(products,id=80,stock) -- l_price = read(products,id=80,price) IF l_price = 10 THEN -- as the contrary leads to a rollback -- precond(read(products,id=80,stock)>=10) -- precond(read(products,id=80,price)<=100) UPDATE products SET stock = l_stock - 10 WHERE id = 80; -- update(products,id=80,stock,stock-10) INSERT INTO orders VALUES (newid,8785,80,10,l_price,'processing'); -- insert(orders,(id,client,product,qty,price,status), -- (newid,80,10,l_price,'processing')) COMMIT; ELSE ROLLBACK; ENDIF; ReadSet={read(products,id=80,stock), read(products,id=80,price)}

68. 68 Infer constraints/relations Use the information extracted from transactions Compare semantic definitions E.g. read(products,id=80,stock) update(products,id=80,stock,stock-10) Intersect because: Same table, same column, conditions are compatible

69. 69 Example: overlaps T1 overlaps with T2 if any of the following is true: T1 reads a data item written (insert,update,delete) by T2 (or vice-versa) T1 writes a data item written by T2 (or vice-versa) Unless blind deletes More generally, we should also consider reads/writes that access different data items related by some constraint

70. 70 Remarks IceCube may improve the reconciliation result The automatic extraction of relations allows the use of the IceCube approach with generic transactions

71. 71 Final remarks Mobile transactions Define conflict detection and resolution rules Reservations Used to guarantee the result of transactions in mobile (disconnected) clients Reconciliation Optimizes the set of transactions that can be executed with success

72. 72 Questions? http://asc.di.fct.unl.pt/mobisnap nmp@di.fct.unl.pt