1. Towards Self-Tuning Databases Sang-Won Lee (swlee@acm.org)swlee@acm.org SKKU VLDB Lab. ( http://oralab.skku.ac.kr/~oralab )http://oralab.skku.ac.kr/~oralab Nov. 1, 2002

2. 2 SKKU VLDB Lab.Toward Self-Tuning Databases Contents Introduction -Why DB tuning? Current Database Tuning Paradigm Toward Self-Tuning Databases

3. 3 SKKU VLDB Lab.Toward Self-Tuning Databases What is DB Tuning? Car tuning -Goals? -What? -How?

4. 4 SKKU VLDB Lab.Toward Self-Tuning Databases DB = IT Core Platform in the 21 st Century? ERP/CRM/SEM/BSC DW/OLAP/Data Mining Web Log Analysis GIS/XML/Mobile Bio-informatics.... Ubiquitous Databases

5. 5 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Tuning in the 21st Century? DB performance influences all aspects of your business -DB performance management is one of the most important success factors in almost every IT projects -(Scott McNealy, Sun CEO, ) Database tuning is not a technical issue! Database tuning is a business issue! DB tuning can improve your productivity DB tuning can increase your revenue/profit DB tuning can increase your ROI(Return on Investment)

6. 6 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Tuning in the 21st Century?(2) More data -doubling every 9 month More users -from CEO to sales representatives More complex query -e.g. OLAP/data mining query Faster responses -e.g. web response time impacts revenue! -e.g. real time personalization Without DB tuning, you need double-up investment on DB server and software every 6 month!

7. 7 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Tuning in the 21 st Century?(3) Data growth vs. computer speedup ** Moores Law -- # of transistors/chip doubles every 18 months (1965)

8. 8 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Tuning in the 21 st Century?(4) Disk sales doubling every 9 months -Greg Papadopoulos, Sun CTO -similar results from Winter VLDB survey (http://www.wintercorp.com) Time to process all your data doubles every 18 months! -You need DPA, not DBA! -Database Performance Administrators -BUT! they are extremely expensive!! Maximize your DB performance!! (DBMax) Maximize your PRODUCTIVITY Maximize your REVENUE/PROFIT Maximize your ROI

9. 9 SKKU VLDB Lab.Toward Self-Tuning Databases Tuning Activities for Databases Systems (H/W, O/S) Oracle Server Database Applications SQL Oracle Database Architecture Components for TuningCheck Lists for TuningCriterial for Tuning Applications Logic / Server Connection / APPS Config etcResponse Time, CPU usage SQL Access Path / Join Order / Join Methods / Parallelization / Efficient Parsing etc. SQL Tuning Elapsed Time, Plan, Trace, Performance Statistics Oracle Database Server SGA Size / Server Configuration / Initial Paramaters SettingPerformance Statistics, Waiting Event, CPU usage Logical & Physical Database Design Physical DB Design / IO Balancing / De-Fragmentation / Removing Chaining etc. Performance Statistics, Waiting Event, Disk I/O Systems (H/W, O/S) Addign CPUs / Kernel Parameter Setting / RAID Config. etcCPU Usage, Memory Usage, Disk I/O Traffic

10. 10 SKKU VLDB Lab.Toward Self-Tuning Databases Current DB Tuning Paradigm 1.(Logical) DB design DB Modeller 2.H/W set-up and configuration HW experts 3.Application development and test Developers 4.System open DPA 5.System operation 6.Data increases 7.Unpredicted performance problems DPA Monitoring Drill-down to Bottleneck Root Cause Analysis Tuning

11. 11 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Monitoring in the 21 st Century? DB tuning vs. DB monitoring -DB Monitoring DB Tuning -E.g.: Tuning 10 hour SQL into 10 minute SQL is not any more special skill Why DB monitoring is so difficult in the 21 st century? -e.g. ERP, DW/OLAP, CRM etc -packaged and tool-based -DB schema is too complex -SQL is unknown -data size is unpredictable -.....

12. 12 SKKU VLDB Lab.Toward Self-Tuning Databases Why DB Monitoring in the 21 st Century?(2) Usually, thousands of SQL statements are running concurrently in a database Cho & Lees rule -When database performance is low, less than 5% SQLs are using more than 95% database resource A key issue in DB monitoring -To rapidly identify the problematic SQL statements! -Without tool, this task is extremely difficult

13. 13 SKKU VLDB Lab.Toward Self-Tuning Databases Why Self-Tuning Databases? War of TCO(Total Cost of Ownership) in 21 st century -Major costs of IT infra. ownership -H/W, S/W, Human Cost -Moores law: H/W, S/W cost -BUT! Human cost -cf. 1950s: telephone industry -manual switching by operators automatic switching system

14. 14 SKKU VLDB Lab.Toward Self-Tuning Databases Why Self-Tuning Databases?(2)

15. 15 SKKU VLDB Lab.Toward Self-Tuning Databases Why Self-Tuning Databases?(3)

16. 16 SKKU VLDB Lab.Toward Self-Tuning Databases Why Self-Tuning Databases?(4) Commercial DBMSs are expontially complicated -Even 10-year veteran Oracle DBAs do not know 50% of all the new features of Oracles latest release -Worsely, it would take more than 1 week for them to understand the performance behavior of a single new feature -The Gap between human DB learning curve and new features becomes larger -Crossing this chasm seems to be impossible!!

17. 17 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases: IBM DB2 SMART project: Self-Managing And Resource Tuning -database part of IBMs Autonomic Computing Initiatives: IBM Autonomic Computing IBM Autonomic Computing -Features 1.Up and Running: configuration advisor -98.4% of experts; outperform average DBAs 2.Design: index/design advisor 3.Maintenance: LEO(LEarning Optimizer) 4.Problem Determination and Correction: health center 5.Availability and Disaster Recovery: recovery expert

18. 18 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases: IBM DB2

19. 19 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases(2): Oracle9i Oralce 9i Manageability: more conservative approach 1.Self-Manageability(more focused) -space, memory, processor, operations 2.Self-Tuning -Dynamic memory management: buffer cache, run-time SQL memory(sort/hash area) Oracle Memory Model Sort Area, Hash Area, Bitmap Index Creation/Merge Pre-Oracle9i -Fixed PGA size -Sort(Hash)_Area_Size Oracle9i -PGA_Aggregate_Target -Dynamically allocated according to the needs

20. 20 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases(2): Oracle9i Avg. Response time in seconds Number of Users

21. 21 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases(2): Oracle9i Total Memory Usage in MB Number of Users

22. 22 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases(3): MS SQL Server AutoAdmin: Self-Tuning and -Administering Databases We achieve this by enabling databases to track the usage of their systems and to gracefully adapt to application requirements. Thus, instead of applications having to track and tune databases, databases actively auto-tunes itself to be responsive to application needs. -See AutoAdmin homepage for details -http://www.research.microsoft.com/research/dmx/autoa dmin/http://www.research.microsoft.com/research/dmx/autoa dmin/ -Rethinking Database System Architecture: Toward a Self- Tuning RISC-style Database System

23. 23 SKKU VLDB Lab.Toward Self-Tuning Databases Industry Cases(4): XM DBMax for Oracle(http://www.ex-em.com)http://www.ex-em.com -Not Self-Tuning Databases -Domestic Oracle Performance Monitoring Tool -We plan to incorporate intelligent recommendation wizards for DB tuning -Application logic -Batch program scheduling -Automatic problem tracking down -Configuration setting -Dynamic memory adjustment -SQL tuning: based on Query Graph methodology -remedy for optimizers internal size estimation cost formula & attribute value independence assumption

24. 24 SKKU VLDB Lab.Toward Self-Tuning Databases References Gerhard Weikum et al., Self-Tuning Database Technology and Information Services: from Wishful Thinking to Viable Engineering VLDB 2002, Ten-Year Award Gerhard Weikum et al, The COMFORT Automatic Tuning Project, Information Systems 19(5), 1994 Gerhard Weikum et al, Special Issues on Self-Tuning Database and Applications, Data Engineering Bulletine, Jun., 1999 IBMs Autonomic Computing Perspectives: http://www.resarch.ibm.com/autonomic/manifesto/autonomic_computing.pdf http://www.resarch.ibm.com/autonomic/manifesto/autonomic_computing.pdf Guy Lohman et al, Toward Automomic computing with DB2 Universal Database, SIGMOD Record, Sep., 2002 MS AutoAdmin Homepage: http://www.research.microsoft.com/http://www.research.microsoft.com/ Oracle Manageability: http://technet.oracle.com/products/manageability/content.html http://technet.oracle.com/products/manageability/content.html

25. 25 SKKU VLDB Lab.Toward Self-Tuning Databases Lets DBMax!