Presentation on theme: "EXTREME PERFORMANCE TUNING GAME PLAN FOR SUCCESS Chris Lawson."— Presentation transcript:
EXTREME PERFORMANCE TUNING GAME PLAN FOR SUCCESS Chris Lawson
Agenda What is Extreme Performance Tuning? Seven attributes of Extreme Perf Tuning Chris favorite tuning techniques Some pitfalls to avoid When it doesnt work the missing piece of the puzzle
A Key Question After you attend a seminar on performance tuning, do you discover that you have become a performance expert?
Performance Tuning is Not: Having really tricky scripts Running colorful programs Delivering long reports full of generalities Being smarter, faster, better- looking than other DBAs
The Paradox of Performance Tuning Being extremely good means being adept in multiple areas A single strength will not work. Example: New DBA uses tool (e.g., OEM) to analyze performance Q: What will be the likely outcome?
The 7 Facets of Extreme Performance Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning
Q: Which Ones are Taught in Seminars? Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning
Only These are Taught in Class Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning
Lets Examine the 7 1. Good Judgment 2. Logical Reasoning 3. Good Techniques 4. Scripts 5. Tools (maybe) 6. Focus 7. Hands-on Experience
Strategy for Success Scripts Tools Techniques Hands-on Experience Good Judgment (1) Focus Logical Reasoning
Good Judgment: Thinking Clearly Good Judgment Difficult to explainits a sound approach to gray areas Q: How do you spot this in candidates? I dont mean I.Q. The man who was too smart to talk to
Good Judgment: Thinking Clearly Good Judgment Easier to spot bad judgment than good Bad judgment stands out Tale of the No-good Nested Loops
Good Judgment May be Hidden; Bad Judgment Stands Out Good Judgment ? The Thrill of Thrill Hill The result...
Good Judgment Firms want people with good judgment, but how do you measure? Common sense reasoning maybe not so common? Some examples of smart people using not-so-smart judgment
Common Sense Reasoning not so Common...
Sybase DBA violated SOX
Common Sense Reasoning not so Common... Sqlserver DBA QA ?
A Class for Good Judgment? Realistically, I probably cant teach you good judgment. Is there some class to attend, to teach you to not do certain things, like these:
Common Sense Reasoning not so Common...
NOT PG&E employees Note parallelism Degree = 3
Common Sense Reasoning not so Common... DBA of the day On-Call DBA DBA in training
Strategy for Success Scripts Tools Techniques Hands-on Experience Good Judgment Focus 2 Logical Reasoning
Focus: Keep Your Eye on the Ball Focus
Doggedly pursue the root cause
Losing Focus: Some Distractions Pet ideasmaybe RAC or Streams? To redesign the applicationlets make it more elegant Trying to solve problem in general My favorite: Avoid solving root problem by blaming others The burnt-up power box
An Unhelpful Tactic When my application runs slowly, likely root cause is found in: 1) My app, 2) Database issue, or 3) Somebody else's app Those other guys are slowing the disk!
Staying on Focus is Hard Work Its tempting to switch to generalities. Benefit: 1. You dont need to know much. 2. Avoids hard analysis. 3. Everybody knows what Add CPUs means. Cary Milsap calls this using the aggregate
Avoid the Dark Side Aggregate Method Avoiding hard work of detailed analysis is immensely appealing. Its widespread because anyone can become an expert You fix lots of problems at once! Solution usually Add more hardware. But it only works for a while...
Focus: Stay on Target! Home-in on the exact problem & fix Micro not Macro. Solutions are directly applicable to the problem at hand, so… 1. You can estimate improvement 2. It will often be a huge pickup
Strategy for Success Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning (3)
This is a Huge Deal … Logical Reasoning Similar to staying focused My definition: An inductive process of working from facts to conclusion.
Barriers to Logical Reasoning Logical Reasoning Presupposing form of the answer Pet solutions Must be the db writer..
A Logical Process: Physician to Magician Logical Reasoning (1) State problem (2) Gather facts (3) Identify bottleneck (4) Create solution (5) Document & Declare Victory
Strategy for Success Scripts Tools Techniques Good Judgment Focus Logical Reasoning Hands-on Experience (4)
There Really Isnt a Substitute Hands-on Experience We dont want DBAs with OCP! (esp. if in bold red on the resume)
An Important Question Why is experience so important? What does it achieve?
Why is Experience So Critical? Save time by skirting crazy ideas. Reduces scope of possible solutions Know how Oracle works in practice Pattern recognition: AhaIve seen that one before! Gain credibility with your customers. Inspired by other ideas youve seen.
You Cant Transfer Experience Hands-on Experience Spend 5 minutes & tell me what you do and …
A Big Turn-off: the Hard Work Hands-on Experience Expert: 10,000 hours of deliberative practice Talent Is Overrated By Geoff Colvin
Strategy for Success Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning
Helpful in Some Cases Tools A matter of personal preference--could be useful. Caution: Tool may direct your effort: OEM says … Pitfall: Never gain competence ? A useful analogy >
Doctor versus DBA: What do YOU want in a Doctor? Pleasant? Nice Office? Cheap?
My Thoughts Takes the job seriously Listens to what you say Understands the problem Has expertise in the field Knows practical solutions How often is he sued?* * Courtesy Dan Grey, PG&E
Do You Ask the Doctor This? What brand of stethoscope do you use?
Strategy for Success Scripts (6) Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning
Scripts Are Important Scripts Ten DBAs > 10 different scripts. There usually isnt just 1 way. Your process should drive your scripts, not the other way around. Roll your own scripts or borrow ideas from others
Be Open to Different Methods Theres more than 1 way to solve a problem. Theres tons of smart people here. Learning from others is better than formal classesits supremely practical. Its hard for anyone to be an expert at more than 1 thing. Watch for people who have a better way at doing something and copy them!
Script ideas Most of my scripts are from others Other ideas came from user groups and forums, etc. One idea (following) came from a beginner with almost no experience!
What I Learned from a Newbie Object Waited-on SeleCT DISTINCT v$session.module, Sid, username, Substr(program,1,19) PROG, sql_text, object_name From V$Session, V$Sql, dba_objects o Where v$session.status = 'ACTIVE' And username is not null and o.object_id = row_wait_obj# And v$session.sql_hash_value = hash_value and v$session.sql_address = v$sql.address and username <> 'SYS'
The Reality of Scripts Good scripts, by themselves do not a performance expert make.
Strategy for Success Scripts Tools Techniques (7) Hands-on Experience Good Judgment Focus Logical Reasoning
Aha! Good Techniques Techniques Absolutely critical This is where we can learn from others White papers, seminars Pretty close to experience Ten DBAs will use 10 different techniques
Techniques Butnot all techniques are created equal. Here are some sub- optimum Techniques
Bad Techniques When Driving forklift, do not drive near edge of loading dock.
Common Sense Reasoning not so Common... Bomb safety violation ?
Bad Techniques Do not drive car into power lines
Common Sense Reasoning not so Common...
Bad Techniques Think carefully when putting name of company on van.
Common Sense Reasoning not so Common... Not the greatest choice of names
Develop Your Own Techniques Which method should I use?
Thats not the way Tom does it! Your Way May Be the Best!
Aha! Chris Ten Techniques Techniques Here are 10 techniques I use You will have others of course
Technique #1: Use Simple Rules of Thumb Q: What are some simple metrics that you like to use?
Develop Rules of Thumb My favorite: Single-block read rate select EVENT, TOTAL_WAITS, TIME_WAITED, round(100*total_waits/time_waited) rate from v$system_event where event like '%db file sequential read%' order by 1 Samples from unrelated databases: 200, 320, 125, 307, 472, 154
Some Rules of Thumb I Use On RAC, I expect about 15% overhead due to cluster. You can see this in AWR reports: SQL ordered by Cluster Wait Time SQL ordered by Cluster Wait Time Cluster Wait Time (s) CWT % of Elapsd Time Elapsed Time(s) CPU Time(s)Execs 4, , , ,054 4, , , ,178 1, , ,420
Other RAC Rules of Thumb On RAC, I expect to see a fast interconnect between nodes. Script node_times.sql calculates how fast nodes communicate. Uses very simple internode event called 'gc cr grant 2-way I found that 1 ms internode rate is typical.
Reference: Node_Times.sql WITH BASE AS (SELECT instance_number, SNAP_ID, TOTAL_WAITS, time_waited_micro/1000 timemsec, LAG(time_waited_micro/1000, 1, 0) OVER (ORDER BY snap_id) AS PREV_TIME_MSEC, LAG(total_waits, 1, 0) OVER (ORDER BY snap_id) AS PREV_waits FROM dba_hist_system_event WHERE event_name ='gc cr grant 2-way' and instance_number = 5 and snap_id > 30130) SELECT b.SNAP_ID, b.instance_number NODE, to_char(begin_interval_time, 'dd-mon-yy-hh24:mi') BEG, (TOTAL_WAITS-PREV_WAITS) "#WAITS", ROUND((TIMEMSEC-PREV_TIME_MSEC)/(.001+TOTAL_WAITS-PREV_WAITS), 1) "RATE" FROM BASE b, dba_hist_snapshot S where b.instance_number = s.instance_number and b.snap_id = s.snap_id and (total_waits-prev_waits) > ORDER BY 1
Technique # 2: Map Complex Joins A VERY powerful technique
Graph Complex Joins: How? List the tables in the join Show join condition Note size of table Note how selective the filters are Thanks to Daniel Tow, of Singing Sql who introduced me to this method 13 years ago.
Sample 3 I cover this in detail in The Art & Science of Oracle Performance Tuning
Technique # 3 Verify Parallel Processing Please Dont bolt-on parallelism But the cost is less if I use default parallelism.
Consider WHY You are Using Parallelism Is this what I intended?
Are slaves doing multi-block or single-block reads? Is This What I Intended?
column child_wait format a30 column parent_wait format a30 column server_name format a4 heading 'Name' column x_status format a10 heading 'Status' column schemaname format a10 heading 'Schema' column x_sid format 9990 heading 'Sid' column x_pid format 9990 heading 'Pid' column p_sid format 9990 heading 'Parent' column program format a12 break on p_sid skip 1 set linesize 200 For Reference: The Parallel Slave Script (part 1)
select x.server_name, x.pid as x_pid, x.sid as x_sid, w2.sid as p_sid, v.osuser, v.schemaname, program, w1.event as child_wait, w2.event as parent_wait from v$px_process x, v$lock l, v$session v, v$session_wait w1, v$session_wait w2 where x.sid <> l.sid(+) and to_number (substr(x.server_name,2)) = l.id2(+) and x.sid = w1.sid(+) and l.sid = w2.sid(+) and x.sid = v.sid(+) and nvl(l.type,'PS') = 'PS' and x.status not like 'AVAIL%' and w2.event not like 'SQL*Net% order by 1,2 For Reference: The Parallel Slave Script (part 2)
Technique # 4 Get Familiar with Wait Events Sequential versus Scattered reads Log sync SQL*Net message from client [RAC] gc current request Esp. the top summary in AWR
Use ASH to Find Wait Events for a Specific Time With P1 As (Select /*+Parallel(a 6) */ Distinct Sample_time, Session_id, Sql_text, Event, Instance_number, Blocking_session From Dba_hist_active_sess_history A, V$sqltext B Where A.Sql_id = B.Sql_id AND Sample_time Like '30-SEP %AM' And Piece = 0 ) Select Instance_number Ins,session_id, SAMPLE_TIME, Sql_text, Event, Blocking_session From P1 Order By 3 AWR is an aggregate and gives summary. ASH is great for nailing specific, brief issue. Drawback: 10 minute runtime on big db.
Technique #5 Find Bind Variables You dont need to guess.
Find Bind Variables Select INST_ID, c.name||'/'||c.value_string bind_var from GV$sqlarea a, dba_users b, v$sql_bind_capture c where b.user_id=a.parsing_user_id and b.username != 'SYS and c.address=a.address and a.sql_id = [enter] We often need to know typical values the user sets for a bind variable. Useful for testing a performance solution For historical, use DBA_HIST_SQLBIND
Find Bind Variables Actual case: Sql looks good, but always scans all partitions.
Technique # 6 Understand Stored Outlines Another extremely powerful technique Example: Critical Sql has changed exec plan in production. Its vendor code. What are you going to do?
Stored Outlines: How They Work A stored outline preserves an execution plan: 1. Oracle captures the hints that ensure a particular plan. 2. When that exact sql is run in the future, Oracle applies those sql hints to keep same exec plan. 3. Outlines are stored in 3 tables.
A Powerful Trick with Stored Outlines What if you dont want to preserve a plan you want a different plan? The scheme: 1. Create outline of exact sql; 2. Use sql hint to create another outline with plan you want; 3. Trick Oracle to use the new exec plan even when we dont supply the sql hint.
For reference: Stored Outline detail* Turn on stored outline gathering. Run Sql. Then run 2nd sql with hint added. We now have 2 stored outlines: No hint >> Oracle uses Outline 1 (bad plan) With hint >> Oracle uses Outline 2 (good plan) Reverse hints so that Oracle will apply Outline 2 when it sees the sql without the hint. Update Outln.Ol$hints << this table has the hints Set Ol_name = Decode(ol_name, One, Two, Two, One) *Thanks to Simon Leung for this tip!
Technique #7 Become Adept w/Sql Hints Q: Why do we ever need to use sql hints?
Some Hints I Use a Lot FULL LEADING ORDERED USE_NL, USE_HASH PARALLEL, PARALLEL_INDEX, INDEX_FFS INDEX CARDINALITY
Technique # 8 Dont Blindly Apply OLTP Ideas Indexing strategies for OLTP are not appropriate for large batch jobs. Always use index is not a reliable maxim for batch jobs. Row by row processing is often a disaster. Consequence of wrong exec plan with batch job is catastrophic. Large batch jobs often need a sql hint.
Technique #9 Dont Presuppose Form of Solution This is very common, & impedes entire process. Similar error: Assuming problem even exists. Adding to confusion: Problem descriptions are often inaccurate, or really a conclusion, not an observation: It is a capital mistake to theorize in advance of the facts. Sherlock Holmes, A Scandal in Bohemia You need more indexes We are having contention. Were out of memory.
Case: Presupposed Solution Wastes Several Days at a Publisher Scenario: Unknown performance problem on critical database. DBA advised: Dont spend any time checking out the network. We already know its fine. The real bottleneck was... THE NETWORK!
Case 2: Presupposed Solution: More Threads are Better! Scenario: Payment processing job uses 4 threads, but still slow. Without identifying actual bottleneck, threads set to 10. New run time Same as Before! Reason: This solution addressed transactions that only accounted for 10% of runtime. Real bottleneck was a single Insert statement that didnt use the multiple threads.
Technique # 10 Insist on Root Cause Before Solution When a solution is proposed, this suggests that the root cause has been identified. So ask (nicely), What problem does this fix?
WaitIve Forgotten Something! Im missing a critical component in performance tuning
What is Missing! Scripts Tools Techniques Hands-on Experience Good Judgment Focus Logical Reasoning ??
A Clue: What Saved 3 Lives? April 14, 1970, Apollo 13 explosion. Problem: No more air scrubbers. Result: Crew will soon die a slow and painful death--poisoned by carbon dioxide. Your Mission: Devise an air scrubber using simple materials. Timeline: A day or so.
The Pep Talk Gene Kranz: Failure is not an option. This crew is coming home. Now get going!
Can We Use our Extreme Strategy? Tools: No way. Logic: Helpful. Judgment: Helpful, but inadequate. Techniques: None Scripts: None Focus: Helpful, but inadequate. Hands-on Experience: Never done before.
A Bizarre Contraption Saves Lives Bill of Materials: A bunch of cardboard (1) plastic bag, (1) air hose, (1) sock, + some duct tape.
Creativity?? Without creativity, we are dull people in a dull profession. A thought experiment to foster creativity: Pretend people will die Ask, Isnt there anything you can do? The un-Innovative Innovation magazine
Dont Be a Dullard! DBA taking class at OpenWorld [before picture] This DBA didnt have any creativity
After Following Chris Performance Tuning Program After picture Look what happened!
Focus Case Study: Getting Some Breathing Room Daily inventory report: 48 hours. Root cause: Running 5 million Sql Director: Fix it today! Short-term answer: Change report- server parameter to run 20 threads Result: 2 hour runtime. Everyone kept their jobs.
Lets Test Your Creativity Scenario: Critical batch job to company needs to finish faster. Runs 1 million sql via one session. Bottleneck is mostly disk. No time to redesign program. Pretend people will die unless you fix it. What can you do?
JUST RUN SUPER BOOST Just warm up the cache Used 25 threads to touch the blocks that will be needed shortly. Got about 50% pickup in performance.
Summary: The Art & Science of Oracle Performance Tuning Extreme Performance tuning demands good analytical skillsleft brain thinking. It also requires excellent right-brain thinking Its almost all art! -- Brian Keating, independent consultant
Thank You for Attending! Extreme Performance Tuning: The Greatest Job on Planet Earth