1. OPS-12: A New Spin on Some Old Latches Richard Banville Fellow

2. © 2008 Progress Software Corporation2 Agenda  Monitoring  Latch usage  Performance

3. © 2008 Progress Software Corporation3 Some Definitions First  What is a “latch” Synchronization mechanism Latches vs “locks”  What is a critical section Code access to shared resource Types of resources  How is a latch obtained test, test-and-set, spin loop –Performed on “shared” value Single threading access

4. © 2008 Progress Software Corporation4 Interactions  Cache coherency  Multi CPU issues  False sharing  Avoiding cache line ping pong Size Alignment Increase # of latches Reduced latch usage – need & time Playing nicely with the hardware

5. © 2008 Progress Software Corporation5 Latch Info VST access to latch information _Latch-Type Latch type _Latch-Name Name _Latch-Lock # times locked _Latch-Wait Nap count _Latch-HoldUser # of last holder _Latch-QholdUser # of last holder of queue latch for each _latch:

6. © 2008 Progress Software Corporation6 _Latch-Type: MT_LT_QUEUE Queue Latch governor LatchSemaphore Multi level latch Wakeup order guaranteed -spin 0 forces all latches Test, test-and-set spin loop

7. © 2008 Progress Software Corporation7 _Latch-Type: MT_LT_SPIN Queue Latch governor LatchSemaphore Multi level latch Wakeup order guaranteed -spin 0 forces all latches Spin Latch Latch Short Nap Single level latch Most performant “backoff” Test, test-and-set spin loop -nap to -napmax

8. © 2008 Progress Software Corporation8 Latch Implementations Spin or queue latch acts as governor of “mux” latch table governor Resources protected Mux latch table (128) Multi level latch with one governor Associates critical section w/resource Underlying one to many relationship Turns 1 latch into 128 latches “Mux” latch request

9. © 2008 Progress Software Corporation9 Latch Implementations Spin or queue latch acts as governor of object latch Resources protected Multi level latch with one governor Associates critical section w/resource Underlying one to one relationship Turns 1 latch into many latches governor Enhanced mux latch Object latch - type 1 request

10. © 2008 Progress Software Corporation10 Latch Implementations Spin or queue latch maintained within the resource itself Resources protected Single level latch. No governor Associates critical section w/resource One to one relationship Greatly improves concurrency No single access funneling Statistics maintained as if 1 latch exists Object latch - type 2 request

11. © 2008 Progress Software Corporation11 Latch Implementations Table of spin or queue latchs Resources protected Latch table Single level latch with no governor Associates critical section w/resource Underlying one to many relationship Latch families request Statistics typically make this look like one latch Modified “mux” latch mechanism

12. © 2008 Progress Software Corporation12 Latch Holder  _Latch-QHolder Last owner of queue type latch Not last queued user  _Latch-Holder Last owner of spin latch Not honored for “true” object latches –Type 2 Who am I waiting on?

13. © 2008 Progress Software Corporation13 Latch Counts  _Latch-Lock # times latch locked/acquired High numbers are OK  _Latch-Wait # times user had to wait (after spin) High numbers may not be good –…but is not wrong –Performance issue potential Balance with –spin value (see tuning) Activity statistics (or lack there of)

14. © 2008 Progress Software Corporation14 Latch Names _Latch-Name  LKP  LKF  LHT  LHT2  LHT3  LHT4  SEQ  PWQ  AIW  BIW  BFP  BHT  BF1  BF2  BF3  BF4  BF5  BF6  BF7  CPQ  LRU  LR2  LR3  LR4  L27  L28  L30  L31  MTX  USR  OM  AIB  BIB  SCC  GST  TXT  TXQ

15. © 2008 Progress Software Corporation15 Agenda  Monitoring  Latch usage  Performance

16. © 2008 Progress Software Corporation16 Miscellaneous Latches  USR* Protects login (user) control structures  SCC* Protects schema locking operation  GST Protects shared memory allocation *Indicates queue latch

17. © 2008 Progress Software Corporation17 Obsolete Latches  Removed AIW, LR2, LR3, LR4  Not in use BIW  Renamed & reserved for future use BF5 L27 BF6 L28 BF7 L29 BF8 L30 Latches removed or reserved

18. © 2008 Progress Software Corporation18 Latches Protecting Database Update Actions

19. © 2008 Progress Software Corporation19 Update Latches - MTX DB Update Action Record BI note Record AI note Perform Action DB Update Action Record BI note Record AI note Perform Action DB Update Action Record BI note Record AI note Perform Action Top of the food chain – busiest/longest update latch Protects txn allocation & bi/ai note order Quiet point maintenance Online backup startup Latches protecting recovery logging procedure High activity High nap rate MT X Not allowed User 1 User 2 User 3

20. © 2008 Progress Software Corporation20 Rollback Processing BI Buffer Pool – Recording a change -bibufs 10 NF - a NF - b NF - c NF - d NF - e 3231 30 29 Modified Queue Free List 15 Current Input Buffer 9 Backout Buffer 12 Backout Buffer BI Current Output Buffer New Notes (Actions) Forward Processing

21. © 2008 Progress Software Corporation21 BIB and AIB latch usage -bibufs 10 Free(a) Free(b) Free(c) Free(d) Free(e) 3231 30 29 Modified Queue Free List BI Current Output Buffer New Notes (Actions) Forward Processing Bi (ai) buffer life cycle B I W Very little contention until BIW active OLTP latch order: MTX, BIB, AIB Page writer latch order: BIB or AIB BI B User

22. © 2008 Progress Software Corporation22 Latches – LRU & PWQ 32 2048 1024 -B least recently used (LRU) chain maintenance... 8192 1056 LRU MRU High activity High nap rate LRU

23. © 2008 Progress Software Corporation23 Latches – LRU & PWQ 32 2048 1024 -B least recently used (LRU) chain maintenance... 8192 1056 LRU MRU 1088 2080 3200 Page writer queue (PWQ) myDB Page Writer Page Writer Dirty buffer Very little contention unless too many APWs LRU PWQ LRU User

24. © 2008 Progress Software Corporation24 Latches – LRU & CPQ 32 2048 1024 -B least recently used (LRU) chain maintenance... 8192 1056 LRU MRU 1088 2080 3200 Page writer queue (PWQ) myDB Page Writer Page Writer 1024 7104 6080 Check point queue (CPQ) Check- point Check- point Very little contention CPQ

25. © 2008 Progress Software Corporation25 Online backup queue - BFP Online “point in time” backup User 1 BI Area 6 Area 7Area 8 … Online Backup.bak Backup queue Modify request Very little contention Only used w/online backup Back it up and mark it BFP TXE MTX BFP

26. © 2008 Progress Software Corporation26 Update Latches  TXT Protects transaction table modifications  TXQ Protects acquire/release of transaction end lock (TXE)  SEQ Protects sequence control structures  Moderate activity  Little contention Latches for write operation structures

27. © 2008 Progress Software Corporation27 Latches Protecting Data Access Actions and Housekeeping

28. © 2008 Progress Software Corporation28 Disk Storage Files Storage Areas Data Area Extent Table A Table BCust TblTable D Index A-1 Index B-1Name Idx Cust# IdxSRep IdxIndex D-1Index D-2 Logical Physical

29. © 2008 Progress Software Corporation29 Disk Storage Files Storage Areas Schema AreaArea 7Area 8 Extent Table A Table BCust TblTable D Index A-1 Index B-1Name Idx Cust# IdxSRep IdxIndex D-1Index D-2 Logical Physical Languages Storage Engine

30. © 2008 Progress Software Corporation30 Disk Storage Files Storage Areas and Object Mapping Schema AreaArea 7Area 8 Extent Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Physical Mapping Languages Storage Engine

31. © 2008 Progress Software Corporation31 Object Cache – OM Latch Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Mapping Languages -omsize OM cache loaded on demand Very high activity High nap rate OM latch needed for paging maintenance Get _StorageObject record Schema OMOM

32. © 2008 Progress Software Corporation32 Object Cache - OM Latch Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Mapping Languages OM cache loaded at startup No latching, no paging Zero contention! Primary Cache Little contention 2 nd cache for over flow & OM latch needed for paging maintenance Get _StorageObject record Schema Secondary Cache new objects OMOM Schema -omsize

33. © 2008 Progress Software Corporation33 Buffer Pool Hash Table - BHT Hash Table (-hash) Shared memory block lookup List of (–B) buffer pool entries (unordered) Buffer pool location lookup single threaded User 1 User 2 User 3 User 4 Block ID BH T High activity, few naps Ptr to buffer

34. © 2008 Progress Software Corporation34 Buffer Pool Hash Table - BHT Hash Table (-hash) BHT now latch family of 256 List of (–B) buffer pool entries (unordered) Buffer pool location lookup multi-threaded Block ID User 1 User 2 User 3 User 4 Block ID High activity, few naps Ptr to buffer BHT... BHT

35. © 2008 Progress Software Corporation35 -B buffer pool entry info – BF[1-4] Database access latches List of (–B) buffer pool entries (unordered) Ptr to buffer Hash Table (-hash) BHT... BHT

36. © 2008 Progress Software Corporation36 -B buffer pool entry info – BF[1-4] List of (–B) buffer pool entries (unordered) Buffer pool info structure supports Hash Table (-hash) Database access latches max of 4 concurrent threads. BF 1 BF 2 BF 3 BF 4 Latch family of 4 Ptr to buffer BHT... BHT

37. © 2008 Progress Software Corporation37 -B buffer pool entry info – BF[1-4] List of (–B) buffer pool entries (unordered) Block ID User 1 User 2 User 3 User 4 Block ID BF latch family of 4 Object latch type 2 Ptr to buffer Hash Table (-hash) BHT... BHT

38. © 2008 Progress Software Corporation38 -B buffer pool entry info – BF[1-4] Hash Table (-hash) List of (–B) buffer pool entries (unordered) Block ID User 1 User 2 User 3 User 4 Block ID -B info structure supports –B threads 4 latch slots used for statistics only Ptr to buffer BF latch family of 4 Object latch type 2 BHT... BHT

39. © 2008 Progress Software Corporation39 Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Acquire a lock LK F Plain latch Single threaded access High activity Low nap rate

40. © 2008 Progress Software Corporation40 Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Protected by LHT (LKT) Record locks Record “get” locks Table locks Purged record locks and LKP latches Type 1 object latch Single threaded latch access Multi threaded chain access LHT LKP LH T LK P High activity/waits Low activity/waits

41. © 2008 Progress Software Corporation41 LHT1 LHT2 LHT3 LHT4 Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Protected by LHT[1-4] Record locks Record “get” locks Type 1 object latch family Multi threaded latch access Multi threaded chain access - Added 4 governors Enhanced concurrency Low contention/waits LK P High activity, few waits

42. © 2008 Progress Software Corporation42 Agenda  Monitoring  Latch usage  Performance

43. © 2008 Progress Software Corporation43 Tuning parameters  -spin # retries before nap –0 or 1 or 10,000 or 6,000 * # cpus or … Can be changed online  -mux Use multiplexing/governor (1)  -nap Initial amount to nap (10 ms)  -napmax* Max amount to nap (5 sec)  -napinc and -napstep Both obsolete and ignored Nothing new here

44. © 2008 Progress Software Corporation44 Foresight and insight  CPU usage -spin ~6,000 * # CPUs CPU thrashes, decrease –spin > 16 CPUs, avoid cache line ping pong Activity matters  Activity # latches acquired # latch waits Application throughput What to monitor

45. © 2008 Progress Software Corporation45 Performance – 10.1C vs 10.1B House KeepingData Access

46. © 2008 Progress Software Corporation46 Performance – 10.1C vs 10.1B HousekeepingData Access Latches/sec LRU 31%Buf 43%

47. © 2008 Progress Software Corporation47 Performance – wait for it 4,294 (~9%) fewer latch waits/sec in 10.1c

48. © 2008 Progress Software Corporation48 Performance – you got it! ~40% ~56% 0 4 8 12 16 24 32 256 512 768 992 # Users

49. © 2008 Progress Software Corporation49 In Summary  Better insight  We’ve done some things  You can too Tuning Move to 10.1C

50. © 2008 Progress Software Corporation50 Questions ?

51. © 2008 Progress Software Corporation51 Thank You

52. © 2008 Progress Software Corporation52