Presentation is loading. Please wait.

Presentation is loading. Please wait.

Some More Database Performance Knobs North American PUG Challenge Richard Banville Software Fellow OpenEdge Development.

Similar presentations


Presentation on theme: "Some More Database Performance Knobs North American PUG Challenge Richard Banville Software Fellow OpenEdge Development."— Presentation transcript:

1 Some More Database Performance Knobs North American PUG Challenge Richard Banville Software Fellow OpenEdge Development

2 © 2012 Progress Software Corporation. All rights reserved. 2 Agenda 1 LRU (again) Networking: Message Capacity Networking: Resource Usage Index Rebuild Summary 5

3 © 2012 Progress Software Corporation. All rights reserved. 3 Agenda 1 LRU (again) Networking: Message Capacity Networking: Resource Usage Index Rebuild Summary 5

4 © 2012 Progress Software Corporation. All rights reserved. 4 LRU (again) RM Block T1 IX Block I1 RM Block T3 IX Block I3 Least Recent Most Recent  Replacement policy of database buffer pool Maintains working set of data buffers Just a linked list – a shared data structure Changes made orderly by LRU Latch  Replace buffer at LRU end with newly read block from disk

5 © 2012 Progress Software Corporation. All rights reserved. 5 LRU (again) RM Block T1 IX Block I1 RM Block T3 IX Block I3 Least Recent Most Recent  Pros – proficient block usage predictor Maintains high buffer pool hit ratio  Cons – housekeeping costs Single threads access to buffer pool (even if for an instant) High activity, relatively high nap rate  Managing LRU: Private read only buffers: -Bp –BpMax (not w/-lruskips until 10.2b07) Alternate buffer pool: –B2 New: -lruskips -lru2skips

6 © 2012 Progress Software Corporation. All rights reserved. 6 LRU (again) RM Block T1 IX Block I1 RM Block T3 IX Block I3 Least Recent Most Recent Find first T1.

7 © 2012 Progress Software Corporation. All rights reserved. 7 LRU (again) RM Block T1 IX Block I1 RM Block T3 IX Block I3 Least Recent Most Recent Find first T1. RM Block T1 RM Block T3 IX Block I3 IX Block I1 RM Block T3 IX Block I3 IX Block I1 RM Block T1

8 © 2012 Progress Software Corporation. All rights reserved. 8 LRU (again) Least Recent Most Recent Find first T1. (again) RM Block T3 IX Block I3 IX Block I1 RM Block T1 For each T1: end. For each w/many tables. What about … RM Block T3 IX Block I3 RM Block T1 IX Block I1 For each w/many tables, many users. RM Block T3 IX Block I3 IX Block I1 RM Block T1

9 © 2012 Progress Software Corporation. All rights reserved. 9 Location, location, location Least Recent Most Recent  With –B 1,000,000 What does it take to evict from the buffer pool? What does it take to go from MRU to LRU?  Do we need MRU on EACH access then? I think not.

10 © 2012 Progress Software Corporation. All rights reserved. 10 Improving Concurrency Least Recent Most Recent  -lruskips LRU and LFU combined Small numbers make a BIG difference Monitor OS Read I/Os and LRU latch contention Adjust online via _Startup. _Startup-LRU-Skips VST field Adjust online via promon –R&D -> 4. Administrative Functions... -> 4. Adjust LRU force skips

11 © 2012 Progress Software Corporation. All rights reserved. 11 Performance – 10.2b06 & -lruskips ~39% # Users

12 © 2012 Progress Software Corporation. All rights reserved. 12 Performance – 10.2b06 & -lruskips (250 users) Note change in LRU latch waits vs buffer latch waits

13 © 2012 Progress Software Corporation. All rights reserved. 13 Performance – 10.2b06 & -lruskips (250 users, big db) Note focus now is on LRU and BHT (not buf)

14 © 2012 Progress Software Corporation. All rights reserved. 14 Performance – 10.2b06 & -lruskips (big db) ~44% ~52% # Users ~15%

15 © 2012 Progress Software Corporation. All rights reserved. 15 Conclusions  -lruskips can eliminate the LRU bottleneck  LRU isn’t the last bottleneck  Overall improvement relative to other contention Data access limited by buffer level contention Table scans over small tables have more buffer contention than large tables –Application changes can improve performance too!

16 © 2012 Progress Software Corporation. All rights reserved. 16 Agenda 1 LRU (again) Networking: Message Capacity Networking: Resource Usage Index Rebuild Summary 5

17 © 2012 Progress Software Corporation. All rights reserved. 17 Networking Control  Process based control -Ma, -Mn, -Mi –Controls the order users are assigned to servers -PendCondTime  Resource based control -Mm –Maximum size of network message –Client & server startup  New tuning knobs – resource based control Alleviate excessive system CPU usage by network layer Control record data stuffed in a network message –Applicable for “prefetch” queries Philosophy: Throughput by keeping server busy without remote client waits!

18 © 2012 Progress Software Corporation. All rights reserved. 18 Networking – Prefetch Query  No-lock query with guaranteed forward motion or scrolling Multiple records stuffed into single network message Browsed static and preselected queries scrolling by default define query cust-q for customer SCROLLING. open query cust-q FOR EACH customer NO-LOCK. repeat: get next cust-q. end. define query cust-q for customer SCROLLING. open query cust-q FOR EACH customer NO-LOCK. repeat: get next cust-q. end. FOR EACH customer NO-LOCK: …. end. FOR EACH customer NO-LOCK: …. end. DO PRESELECT EACH customer NO-LOCK: …. end. DO PRESELECT EACH customer NO-LOCK: …. end.

19 © 2012 Progress Software Corporation. All rights reserved. 19 Server Network Message Processing Loop

20 © 2012 Progress Software Corporation. All rights reserved. 20 Server Network Message Processing Loop

21 © 2012 Progress Software Corporation. All rights reserved. 21 -NmsgWait Server Network Message Processing Loop What’s new:

22 © 2012 Progress Software Corporation. All rights reserved. 22 Server Network Message Processing Loop

23 © 2012 Progress Software Corporation. All rights reserved. 23 Server Network Message Processing Loop

24 © 2012 Progress Software Corporation. All rights reserved. 24 Server Network Message Processing Loop 10 milliseconds to poll(0)! Poll() is system CPU intensive 10 microseconds to copy 1 record

25 © 2012 Progress Software Corporation. All rights reserved. 25 Potential side effects Server Network Message Processing Loop -prefetchPriority What’s new:

26 © 2012 Progress Software Corporation. All rights reserved. 26 Server Network Message Processing Loop

27 © 2012 Progress Software Corporation. All rights reserved. 27 Process Waiting Network Message

28 © 2012 Progress Software Corporation. All rights reserved. 28 Process Waiting Network Message Non-prefetch query request

29 © 2012 Progress Software Corporation. All rights reserved. 29 Process Waiting Network Message 1 st record of a prefetch query request

30 © 2012 Progress Software Corporation. All rights reserved. 30 Process Waiting Network Message Secondary records of a prefetch query request: - threshold not met - default threshold is 16 records

31 © 2012 Progress Software Corporation. All rights reserved. 31 Process Waiting Network Message Secondary records of a prefetch query request: - Client waiting - Threshold met - Send message

32 © 2012 Progress Software Corporation. All rights reserved. 32 Process Waiting Network Message  Increase network message fill rate: - Improve TCP throughput - Improve overall server performance  Defaults have not changed  Provides control for you  Every deployment is different What’s new:

33 © 2012 Progress Software Corporation. All rights reserved. 33 NOTE: - -Mm size determines max -Mm 4096 / 16 rec = 256 bytes Process Waiting Network Message -prefetchDelay -prefetchNumRecs -prefetchFactor Potential side effects: – Improved TCP/system performance – Choppy behavior on remote client? Threshold control: # recs vs % full Disregard 1 st record request check

34 © 2012 Progress Software Corporation. All rights reserved. 34 Altering Network Message Behavior  Promon Support (_Startup VST too!) Alter online –R&D … –4. Administrative Functions … –7. Server Options … Server Options: 1. Server network message wait time: 2 seconds 2. Delay first prefetch message: Enabled 3. Prefetch message fill percentage: 90 % 4. Minimum records in prefetch message: Suspension queue poll priority: 0 7. Terminate a server Server Options: 1. Server network message wait time: 2 seconds 2. Delay first prefetch message: Enabled 3. Prefetch message fill percentage: 90 % 4. Minimum records in prefetch message: Suspension queue poll priority: 0 7. Terminate a server

35 © 2012 Progress Software Corporation. All rights reserved. 35 Performance – 10.2b06 & Networking changes ~32% ~212% # Users

36 © 2012 Progress Software Corporation. All rights reserved. 36 Agenda 1 LRU (again) Networking: Message Capacity Networking: Resource Usage Index Rebuild Summary 5

37 © 2012 Progress Software Corporation. All rights reserved. 37 Assumptions for best performance  Index data is segregated from table data Indexes & tables are in different storage areas  You have enough disk space for sorting  You understand the impact of CPU and memory consumption  Process allowed to use available system resources

38 © 2012 Progress Software Corporation. All rights reserved. 38 Index Rebuild Parameters - Overview sort block size (8K – 64K, note new limit) # threads for data scan phase merge block size ( default -TB) merge pool fraction of system memory (in %) # threads per concurrent sort group merging -mergethreads # concurrent sort group merging -threadnum # merge buffers to merge each merge pass -TM report system usage statistics a bit quieter than before -rusage -silent -TB -datascanthreads -TMB -TF

39 © 2012 Progress Software Corporation. All rights reserved. 39 Phases of Index Rebuild (“non-recoverable”) Index Scan Data Scan/ Key Build Sort-Merge Index Key Insertion Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Read –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity

40 © 2012 Progress Software Corporation. All rights reserved. 40 Phases of Index Rebuild Index Scan Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Area 9: Index scan (Type II) complete. Index area is scanned start to finish (single threaded) Block at a time with cluster hops Index blocks are put on free chain for the index Index Object is not deleted (to fix corrupt cluster or block chains) Order of operation: Blocks are read from disk, Blocks are re-formatted in memory Blocks are written to disk as –B is exhausted Causes I/O in other phases for block re-format Can be eliminated with manual area truncate where possible

41 © 2012 Progress Software Corporation. All rights reserved. 41 Phases of Index Rebuild Index Scan Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Data Scan/ Key Build Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Table data area is scanned start to finish (multi-threaded if –datascanthreads) Each thread processes next block in area (with cluster hops) Database re-opened by each thread in R/O mode Ensure file handle ulimits set high enough Processing area 8 : (11463) Start 4 threads for the area. (14536) Area 8: Multi-threaded record scan (Type II) complete.

42 © 2012 Progress Software Corporation. All rights reserved. 42 Data Scan/Key Build Record b) Extract next record from data block and build index key (sort order) Key -TF c) Insert key into sort block (-TB 8K thru 64K) Sort Block d) Sort/merge full sort block into merge block. (-TMB -TB thru 64K) Sort Block a) Thread reads next data block in data area RM Block e) Write merge block to –TF, overflow to temp (-TMB sized I/O).srt1.srt2 DB … Merge Block

43 © 2012 Progress Software Corporation. All rights reserved. 43 Sort Groups: -SG 3 (note 8 is minimum) Record Index 1 SG 1 SG 2 SG 3 Index 2 Index 3 Index 4 1) -T /usr1/richb/temp/ 3).srt /usr1/richb/temp/ 0 /usr1/richb/temp/ 4).srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/.srt1.srt2.srt3 2).srt 0 /usr1/richb/temp/  Each group has its own sort file  Sort file location 1. Sort files in same directory (I/O contention) 4. Sort files in different location  Ensure enough space Each index assigned a particular sort group (hashed index #)

44 © 2012 Progress Software Corporation. All rights reserved. 44 Phases of Index Rebuild Index Scan Data Scan/ Key Build Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Sort-Merge Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sorting index group 3 Spawning 4 threads for merging of group 3. Sorting index group 3 complete.

45 © 2012 Progress Software Corporation. All rights reserved. 45 Sort-Merge Phase  Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks stored in –TF and temp files.  These merge blocks are further merged –TM# at a time to form new larger “runs” of sorted merge blocks.  -TM# of these new “runs” are then merged to form even larger “runs” of sorted merge blocks.  When there is only one very large “run” left, all the key entries in the sort group are in sorted order. Sorted!

46 © 2012 Progress Software Corporation. All rights reserved. 46 -threadnum vs -mergethreads  -threadnum 2 -TF.srt1 -TF.srt2 -TF.srt3 Thread 1 Thread 2 Merge phase group 2 Merge phase group 1

47 © 2012 Progress Software Corporation. All rights reserved. 47 -threadnum vs -mergethreads  -threadnum 2 -TF.srt1 -TF.srt2 -TF.srt3 Thread 2 Thread 1 Merge phase group 2 Merge phase group 3 Thread 0 B-tree insertion occurs as soon as a sort group’s merge is completed. Thread 0 begins b-tree insertion concurrently.

48 © 2012 Progress Software Corporation. All rights reserved. 48 -threadnum vs -mergethreads  -threadnum 2 –mergethreads 3 Thread 1 Thread 2 Note: 8 actively running threads Thread 3 Thread 4 Thread 5 Thread 6 Thread 7 Thread 8 -TF.srt1 -TF.srt2 -TF.srt3 Merge phase group 1 Merge phase group 2  Merge threads merge successive “runs” of merge blocks concurrently.

49 © 2012 Progress Software Corporation. All rights reserved. 49 -threadnum vs -mergethreads  -threadnum 2 –mergethreads 3 Thread 2 Thread 6 Thread 7 Thread 8 Thread 1 Thread 3 Thread 4 Thread 5 -TF.srt2 -TF.srt3 -TF.srt1 Merge phase group 3 Merge phase group 2

50 © 2012 Progress Software Corporation. All rights reserved. 50 -threadnum vs -mergethreads  -threadnum 2 –mergethreads 3 Thread 0 Thread 2 Thread 6 Thread 7 Thread 8 Thread 1 Thread 3 Thread 4 Thread 5 -TF.srt1 -TF.srt2 -TF.srt3 Thread 0 begins b-tree insertion concurrently. Merge phase group 2 Merge phase group 3 Note: 9 actively running threads B-tree insertion occurs as soon as a sort group’s merge is completed.

51 © 2012 Progress Software Corporation. All rights reserved. 51 Phases of Index Rebuild Index Scan Data Scan/ Key Build Sort-Merge Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Index Key Insertion Read –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity

52 © 2012 Progress Software Corporation. All rights reserved. 52 Index Key Insertion Phase Building index 11 (cust-num) of group 3 … Building of indexes in group 3 completed. Multi-threaded index sorting and building complete.  Key entries from sorted merge blocks are inserted into b-tree  Performed sequentially entry at a time, index at a time  Leaf level insertion optimization (avoids b-tree scan)  Leaf level written to disk as soon as full (since never revisited) Root Leaf Write leaf when full DB Index B-tree

53 © 2012 Progress Software Corporation. All rights reserved Indexes were rebuilt. (11465) Index rebuild complete. 0 error(s) encountered.

54 © 2012 Progress Software Corporation. All rights reserved. 54 Index Rebuild - Tuning  Truncate index only area if possible .srt file  Parameters -mergethreads: 2 or 4 and –threadnum 2 or 1 -datascanthreads: 1.5 * # CPUs -B 1024 –TF 80 (monitor physical memory paging) –TMB 64 –TB 64 –TM 32 –T: separate disk, RAM disk if not using -TF (no change) -rusage & -silent

55 © 2012 Progress Software Corporation. All rights reserved. 55 Performance Numbers Elapsed Time Cost of each phase (in secs)

56 © 2012 Progress Software Corporation. All rights reserved. 56 Agenda 1 LRU (again) Networking: Message Capacity Networking: Resource Usage Index Rebuild Summary 5

57 © 2012 Progress Software Corporation. All rights reserved. 57 Summary  LRU Potential for a big win Always room for improvement –Us and you!  Networking You now have more control With power comes responsibility  Index Rebuild Big improvements if –Your database is setup properly –You provide system resources to index rebuild Hopefully you’ll never need it

58 © 2012 Progress Software Corporation. All rights reserved. 58 Questions ?

59

60 © 2012 Progress Software Corporation. All rights reserved. 60 Performance – 10.2b06 & -lruskips (250 users) Latches/sec

61 © 2012 Progress Software Corporation. All rights reserved. 61 Performance – 10.2b06 & -lruskips (250 users, big db) Latches/sec

62 © 2012 Progress Software Corporation. All rights reserved. 62 Networking – Promon Support (VSTs too!) Activity: Servers Total Messages received 1183 Messages sent 1181 Bytes received Bytes sent Records received 0 Records sent 1180 Queries received 1181 Time slices 0 Activity: Servers Total Messages received 1183 Messages sent 1181 Bytes received Bytes sent Records received 0 Records sent 1180 Queries received 1181 Time slices 0 Activity: Servers Total Messages received 98 Messages sent 96 Bytes received 7500 Bytes sent Records received 0 Records sent 1180 Queries received 96 Time slices 1149 Activity: Servers Total Messages received 98 Messages sent 96 Bytes received 7500 Bytes sent Records received 0 Records sent 1180 Queries received 96 Time slices 1149 Activity: Servers Total Messages received 81 Messages sent 79 Bytes received 6208 Bytes sent Records received 0 Records sent 1180 Queries received 79 Time slices 1148 Activity: Servers Total Messages received 81 Messages sent 79 Bytes received 6208 Bytes sent Records received 0 Records sent 1180 Queries received 79 Time slices 1148 Activity: Servers Total Messages received 80 Messages sent 78 Bytes received 6132 Bytes sent Records received 0 Records sent 1180 Queries received 78 Time slices 1148 Activity: Servers Total Messages received 80 Messages sent 78 Bytes received 6132 Bytes sent Records received 0 Records sent 1180 Queries received 78 Time slices 1148 No Prefetch Prefetch, 90% capacity Prefetch, default settings Prefetch, 90% cap. & delay


Download ppt "Some More Database Performance Knobs North American PUG Challenge Richard Banville Software Fellow OpenEdge Development."

Similar presentations


Ads by Google