1. Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc

2. Agenda 1 Overview Processing Details Tuning Suggestions Some Statistics 2 3 4 Summary 5

3. Phases of Index Rebuild (non-recoverable) Index Scan Data Scan/ Key Build Sort-Merge Index Key Insertion Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Read sorted list in –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity

4. Index Rebuild Parameters - Overview sort block size (8K – 64K, note new limit) # threads for data scan phase merge block size ( default -TB) merge pool fraction of system memory (in %) # threads per concurrent sort group merging -mergethreads # concurrent sort group merging -threadnum # merge buffers to merge each merge pass -TM report system usage statistics a bit quieter than before -rusage -silent -TB -datascanthreads -TMB -TF

5. Agenda 1 Processing Details Overview Tuning Suggestions Some Statistics 2 3 4 Summary 5

6. Phases of Index Rebuild Index Scan Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Area 9: Index scan (Type II) complete. Index area is scanned start to finish (single threaded) Block at a time with cluster hops Index blocks are put on free chain for the index Index Object is not deleted (to fix corrupt cluster or block chains) Order of operation: Blocks are read from disk, Blocks are re-formatted in memory Blocks are written to disk as –B is exhausted Causes I/O in other phases for block re-format Can be eliminated with manual area truncate where possible

7. Phases of Index Rebuild Data Scan/ Key Build Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Index Scan Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Table data area is scanned start to finish (multi-threaded if –datascanthreads) Each thread processes next block in area (with cluster hops) Database re-opened by each thread in R/O mode Ensure file handle ulimits set high enough Processing area 8 : (11463) Start 4 threads for the area. (14536) Area 8: Multi-threaded record scan (Type II) complete.

8. Rules for -datascanthreads Index rebuild run with sort option No index being rebuilt exists in table area being scanned No word indexes are being rebuilt for table data being scanned Data area being scanned is type TII storage area. If any true: use single threaded data scan for that area Violation for one area does not preclude use on another area

9. Data Scan/Key Build Record b) Extract next record from data block and build index keys (sort order) Key -TF c) Insert key into sort block for sort group (-TB 8K thru 64K) d) Sort/merge full sort block into merge block. (-TMB -TB thru 64K) Sort Block a) Thread reads next data block in data area RM Block e) Write merge block to –TF, overflow to temp (-TMB sized I/O).srt1.srt2 DB … Merge Block What about mixed areas? Process index blocks as in index scan phase No longer read only! Sort Block

10. Sort Groups: -SG 8 (note 8 is minimum) Each sort group has its own sort file Sort file location 1 & 2. Sort files in same directory (I/O contention) 4. Sort files in different location Ensure enough space Record Index 1 SG 1 SG 2 SG 3 Index 2 Index 3 Index 9 1) -T /usr1/richb/temp/ 4).srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/.srt1.srt2.srt3 2).srt 0 /usr1/richb/temp/ Each index assigned a particular sort group (hashed index #) 3).srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/

11. More on Sort Groups Sort file size is in 1K units 0 indicates unlimited space Sort file max size with –TMB 8: 16 TB Increase –TMB (-TMB 64: 128 TB) 3).srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/ Record Index 1 SG 1 SG 2 SG 3 Index 2 Index 3 Index 9 1) -T /usr1/richb/temp/ 4).srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/.srt1.srt2.srt3 2).srt 0 /usr1/richb/temp/ Each index assigned a particular sort group (hashed index #)

12. Last one on Sort Groups, I promise Max –SG 64 per area. What if more than 64 indexes Sort groups can contain more than one indexs entries: MOD(idx-num, -SG) Data scan will add key entry to appropriate sort group Sort/merge orders indexes in sort group by index number Key insertion phase inserts all entries for one index followed by all index key entries for the other within same sort group. Record Index 1 SG 1 SG 2 SG 3 Index 2 Index 3 Index 9.srt1.srt2.srt3 Each index assigned a particular sort group (hashed index #) Index 1 Index 9 Index 2 Index 3

13. Phases of Index Rebuild Index Scan Data Scan/ Key Build Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Sort-Merge Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sorting index group 3 Spawning 4 threads for merging of group 3. Sorting index group 3 complete.

14. Sort-Merge Phase Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks stored in –TF and temp files. Merge blocks are further merged –TM# at a time to form new larger runs of sorted merge blocks. -TM# of these new runs are then merged forming even larger runs of sorted merge blocks. When only one very large run left, all the key entries in the sort group are in sorted order. Sorted! Up to 7 concurrent merge threads Up to 3 concurrent merge threads Only 1 merge thread on last run

15. -threadnum vs -mergethreads -threadnum 2 -TF.srt1 -TF.srt2 -TF.srt3 Thread 1 Thread 2 Merge phase group 2 Merge phase group 1

16. -threadnum vs -mergethreads -threadnum 2 -TF.srt1 -TF.srt2 -TF.srt3 Thread 2 Thread 1 Merge phase group 2 Merge phase group 3 Thread 0 B-tree insertion occurs as soon as a sort groups merge is completed. Thread 0 begins b-tree insertion concurrently.

17. -threadnum vs -mergethreads -threadnum 2 –mergethreads 3 Thread 1 Thread 2 Note: 8 actively running threads Thread 3 Thread 4 Thread 5 Thread 6 Thread 7 Thread 8 -TF.srt1 -TF.srt2 -TF.srt3 Merge phase group 1 Merge phase group 2 Merge threads merge successive runs of merge blocks concurrently.

18. -threadnum vs -mergethreads -threadnum 2 –mergethreads 3 Thread 2 Thread 6 Thread 7 Thread 8 Thread 1 Thread 3 Thread 4 Thread 5 -TF.srt2 -TF.srt3 -TF.srt1 Merge phase group 3 Merge phase group 2

19. -threadnum vs -mergethreads -threadnum 2 –mergethreads 3 Thread 0 Thread 2 Thread 6 Thread 7 Thread 8 Thread 1 Thread 3 Thread 4 Thread 5 -TF.srt1 -TF.srt2 -TF.srt3 Thread 0 begins b-tree insertion concurrently. Best performance with low –threadnum & high -mergethreads Merge phase group 2 Merge phase group 3 Note: 9 actively running threads B-tree insertion occurs as soon as a sort groups merge is completed.

20. Phases of Index Rebuild Index Scan Data Scan/ Key Build Sort-Merge Scan index data area start to finish I/O Bound with little CPU activity Eliminated with area truncate Scan table data area start to finish (area at a time) Read records, build keys, insert to temp sort buffer Sort full temp file buffer blocks (write if > -TF) I/O Bound with CPU Activity Sort-merge –TF and/or temp sort file CPU Bound with I/O Activity I/O eliminated if –TF large enough Index Key Insertion Read sorted list in –TF or temp sort file Insert keys into index Formats new clusters; May raise HWM I/O Bound with little CPU Activity

21. Index Key Insertion Phase Key entries from sorted merge blocks are inserted into b-tree Performed sequentially entry at a time, index at a time Leaf level insertion optimization (avoids b-tree scan) Leaf level written to disk as soon as full (since never revisited) Building index 11 (cust-num) of group 3 … Building of indexes in group 3 completed. Multi-threaded index sorting and building complete. Root Leaf Write leaf when full DB Index B-tree

22. 2085 Indexes were rebuilt. (11465) Index rebuild complete. 0 error(s) encountered.

23. Agenda 1 Tuning Suggestions Overview Some Statistics 2 3 4 Summary 5 Processing Details

24. Assumptions for best performance Index data is segregated from table data Indexes & tables are in different storage areas User data areas are TII storage areas You have enough memory/disk space for sorting If not, go home. Youre done. You understand the impact of CPU and memory consumption Process allowed to use available system resources

25. Index Rebuild - Tuning Truncate index only area if possible.srt file – set up properly Spread I/O across disks Avoid with –TF settings (or RAM disk) The contents of table customer" will be deleted. The contents of index "cust-order" will be deleted. Are you sure you want to truncate storage area "Customer/Order area" (y/n) The contents of table customer" will be deleted. The contents of index "cust-order" will be deleted. Are you sure you want to truncate storage area "Customer/Order area" (y/n) proutil -C truncate area Customer/Order Area This could be a life changing decision…

26. Index Rebuild - Tuning Parameters -datascanthreads = 1.5 * # CPUs -mergethreads * -threadnum = 1.5 * #CPUs -threadnum 2 to 4 -B 1024 –TF 80 monitor physical memory paging –TMB 64 –TB 64 –TM 32 could be lower with large –mergethreads (> 16)

27. Memory usage approximation Total memory used (in KB): totalMemory = dataScanOverhead + tfMemory + allSortGroups + 1MB Total memory used (in KB): totalMemory = dataScanOverhead + tfMemory + allSortGroups + 1MB

28. Arough approximation: 64 GB Ram, 16 CPUs, 8 indexes proutil -C idxbuild area Customer/Order Index Area -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8

29. Arough approximation: 64 GB Ram, 16 CPUs, 8 indexes proutil -C idxbuild area Customer/Order Index Area -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8

30. Arough approximation: 64 GB Ram, 16 CPUs, 8 indexes proutil -C idxbuild area Customer/Order Index Area -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8 Total memory used (in KB): totalMemory = 76,800+ 53,687,091 + 166,320 + 1MB = 54,909,667 K = 52.4 GB Total memory used (in KB): totalMemory = 76,800+ 53,687,091 + 166,320 + 1MB = 54,909,667 K = 52.4 GB

31. Agenda 1 Some Statistics Overview Tuning Suggestions Processing Details 2 3 4 Summary 5

32. Performance Numbers Elapsed Time Cost of each phase (in secs)

33. Agenda 1 Summary Overview Tuning Suggestions Some Statistics 2 3 4 Processing Details 5

34. Summary Index Rebuild Big improvements if Your database is setup properly You provide system resources to index rebuild You use the new settings in 10.2b06 One bug fix in 10.2b07 More efficient memory allocation coming soon! (10.2b08) Hopefully youll never need to index rebuild

35. Questions ?

36. October 6–9, 2013 Boston #PRGS www.progress.com/exchange-pug Special low rate of $495 for PUG Challenge attendees with the code PUGAM And visit the Progress booth to learn more about the Progress App Dev Challenge!