1. © Tieto Corporation Public Shared Database Concurrency Aivars Kalvāns Lead Software Architect Tieto aivars.kalvans@tieto.com

2. © Tieto Corporation Public Who am I Working at “Tieto Cards” > 11 years Junior developer → Lead Software Architect C and C++, Python Oracle database C++ library for working with Oracle OCI MongoDB for Developers certificate 2013-09-092

3. © Tieto Corporation Public About Tieto Cards Payment card systems Both on-line and batch processing Linux, AIX, HP-UX, Solaris, Windows Oracle Tuxedo middleware (C++) Oracle database (PL/SQL, PRO*C, OCI) 2013-09-093

4. © Tieto Corporation Public Largest customers 31,000,000 cards 1,000,000,000 financial transactions/year 160,000 POS terminals 3,700 ATM 250 financial transactions/second on-line More for batch processing: 24h data within 8h Thousands in case of spherical cards in a vacuum 2013-09-094

5. © Tieto Corporation Public Terms Concurrency Consistency Locks Blocking 2013-09-095

6. © Tieto Corporation Public Shared memory concurrency Concurrent components communicate by altering the contents of shared memory locations Threads (C++, Java, C#, …) Processes (Oracle on Unix) Popular topic at developer conferences Lock-free Actor model, … 2013-09-096

7. © Tieto Corporation Public Shared database concurrency Concurrent components communicate by altering the contents of shared memory locations Concurrent components communicate by altering shared data in a database 2013-09-097

8. © Tieto Corporation Public Why is it important Concurrency in application server → Concurrency in database server Database is the bottleneck Mixing functionality of batch and online processing Design for concurrency Hard to fix afterwards 2013-09-098

9. © Tieto Corporation Public Concurrently shared data Accounts and limits Card Merchant Bank Real-time statistics for fraud detection Country Merchant category (hotel, casino, escort service) 2013-09-099

10. © Tieto Corporation Public Locking in a database Pessimistic locking and Optimistic locking http://www.orafaq.com/papers/locking.pdf More challenging than the technology is overcoming resistance from seasoned development professionals who have been using the trusted SELECT… FOR UPDATE for all of their Oracle careers. These individuals may need to be convinced of the benefits of using optimistic and on large development projects their support will be crucial. 2013-09-0910

11. © Tieto Corporation Public What has been tried Use the best practices, architecture, patterns for the application Leave database concurrency to optimistic locking 2013-09-0911

12. © Tieto Corporation Public Does not work as expected* Acceptable until x transactions per second System gets slower when concurrency increases Optimistic = hope nobody modifies the same data * for our use case, your experience may differ 2013-09-0912

13. © Tieto Corporation Public Oracle 101 Row-level (TX) locks LOCK TABLE … Locks are held until COMMIT or ROLLBACK Not in case of ROLLBACK TO SAVEPOINT Writes don’t block reads Reads don’t block writes 2013-09-0913

14. © Tieto Corporation Public Placing a row-level lock UPDATE, DELETE, SELECT … FOR UDATE [NOWAIT] INSERT Primary key or unique constraint violation Every modification implies locking To guarantee ACID properties Let’s call it “implicit locking” 2013-09-0914

15. © Tieto Corporation Public Few years later… 2013-09-0915

16. © Tieto Corporation Public Pessimistic vs. Optimistic TimePessimisticOptimistic t1SELECT … FOR UPDATE SELECT version, … t2Modify data t3UPDATE …UPDATE SET version = :next_version WHERE version = :known_version t4Retry if ROWCOUNT=0 t5COMMIT 2013-09-0916 LOCK

17. © Tieto Corporation Public “Implicit” vs. Optimistic Time“Implicit”Optimistic t1SELECT …SELECT version, … t2Modify data t3UPDATE …UPDATE SET version = :next_version WHERE version = :known_version t4Retry if ROWCOUNT=0 t5COMMIT 17 2013-09-09 LOCK Additional consistency control!

18. © Tieto Corporation Public When to control consistency? There are 10 kinds of updates 2013-09-0918 Don’t need itNeed consistency Relative update balance = balance + :depositbalance = balance - :withdraw Absolute update name = :new_namebalance = :new_balance

19. © Tieto Corporation Public Conclusions about locking If you don’t have concurrent updates on the same data Does not matter which locking you use Every modification places row-level locks If you do have concurrent updates on the same data There is nothing “optimistic” about optimistic locking in Oracle database Pessimistic is better Optimistic locking burns more cycles due to retries 2013-09-0919

20. © Tieto Corporation Public How to choose locking “Implicit locking” by default Relative updates Last update wins for absolute updates Pessimistic locking To prevent concurrency Optimistic locking Stateless applications “Conditional consistency” 2013-09-0920

21. © Tieto Corporation Public How we have chosen locking Cards – pessimistic locking Historic reasons, unlikely concurrent updates Statistics – “implicit locking” Accounts Optimistic locking for “conditional consistency” Ensure consistency sometimes for withdrawals and complex interest calculations “Implicit locking” otherwise …and few more tricks 2013-09-0921

22. © Tieto Corporation Public How to increase concurrency Same as with shared memory concurrency Reduce the time locks are held Reduce lock granularity Not an option Lock-less … NoSQL 2013-09-0922

23. © Tieto Corporation Public Reduce (b)locking time Do as little work as possible between locking DML and COMMIT Reorder to perform locking DML last UPDATE, INSERT, COMMIT vs. INSERT, UPDATE, COMMIT Up to 50% improvement Not all DML have equal possibility of blocking Update card account – 10 times / month Update bank account – 10 times / second 2013-09-0923

24. © Tieto Corporation Public Increase performance Bulk operations Multiple rows with one DML SELECT … WHERE id IN (:id1, :id2, :id3) ROWID within transaction boundaries Physical address of the row Index maps values to ROWIDs Except: Index-Organized Tables Table management (shrink, flashback, partition key change) SELECT ROWID, … UPDATE / DELETE … WHERE ROWID = :rowid 2013-09-0924

25. © Tieto Corporation Public Increase performance Faster servers Faster CPU (memory, disk) → shorter locking time Few fast CPUs are better than many slow CPUs Locking time can be reduced by faster not more processes 2013-09-0925

26. © Tieto Corporation Public Do more with less DML INSERT … RETURNING … INTO … UPDATE and DELETE as well INSERT ALL … / INSERT WHEN … MERGE … USING (SELECT … FROM DUAL) 2013-09-0926

27. © Tieto Corporation Public Release locks faster Avoid distributed transactions Often too easy to use Two-phase commit: prepare and commit Extra step and coordination overhead before locks are released COMMIT on DML success DML+COMMIT in single round-trip INSERT, COMMIT vs. INSERT+COMMIT – 10% improvement Up to 50% improvement in case of blocking 2013-09-0927 ProcessesTPS / DistributedTPS / Local 48361070 (+28%) 89721260 (+29%)

28. © Tieto Corporation Public Reduce lock granularity “Partition” rows Multiple rows instead of one Update row based on random or hash Aggregate rows when reading 2013-09-0928 idbalance 1342 idbalancepartition 13150 1371 112 1393

29. © Tieto Corporation Public Reduce lock granularity INSERT instead of UPDATE Create a change journal INSERT offsets Aggregate and UPDATE once in a while 2013-09-0929

30. © Tieto Corporation Public Thank you and a happy programmers’ day! 2013-09-0930

31. © Tieto Corporation Public Aivars Kalvāns Lead Software Architect Tieto aivars.kalvans@tieto.com