Presentation on theme: "DATA CONFUSION How to confuse yourself and others with Data Analysis."— Presentation transcript:
DATA CONFUSION How to confuse yourself and others with Data Analysis
AGENDA FOR TODAY’S TALK Good Graphs – Bad Graphs The Law of Averages PTBD Analysis Enumerative & Analytical Problems PARC Analysis Wrong Methods of Analysis
“There are three kinds of lies: Lies, damned lies and statistics” Attributed to Benjamin Disraeli by Mark Twain
GOOD GRAPHS AND BAD GRAPHS
DATA RELEVANCE Graphs are only as good as the data they display No amount of creativity can produce good graphs from dubious data
DATA CONTENT Don’t produce graphs from very small amounts of data The human brain can grasp 1, 2 or 3 numbers without a graph
RULES FOR PRODUCING GOOD GRAPHS KEEP IT SIMPLE AND STUPID –Jesse Ventura Tell the truth – don’t distort the data
GOOD GRAPHS Portray information without distortion Contain no distracting elements –No false third dimensions, irrelevant decoration, or colour (chartjunk) Use an appropriate scale Label axes and tick marks properly, including measurement units Have a descriptive title and/ or caption and legend Have a low ink – to – information ratio
BAD GRAPH GOOD GRAPH BAD GRAPH EVEN BETTER GRAPH
BAD GRAPH GOOD GRAPH
GRAPHS THAT CONFUSE
GRAPHS THAT TELL A STORY
HISTOGRAMS No meaningless gaps Reasonable Choice of bins Easy to choose or adjust bins Good aspect ratio Meaningful labels on axes Appropriate labels on bin tick marks
THE LAW OF AVERAGES “If I sit in a freezer and plunge my head into a pan of boiling chip fat..... on average, I’m quite comfortable.”
SHEWHART’S RULES FOR PRESENTATION OF DATA Rule One –Data should always be presented in a way that preserves the evidence in the data Rule Two –When an average, standard deviation or histogram is used to summarize data, the user should not be misled into to taking action they would not take if the data were presented in a time series
USING THE WRONG METHODS Descriptive Statistics: A, B, C, D Variable N Mean StDev CoefVar Minimum Maximum A B C D Process:ABCD
NO SIGNIFICANT DIFFERENCE HERE!
ALWAYS CARRY OUT PTBD ANALYSIS PLOT THE B….. DOTS!
TYPES OF STATISTICAL STUDIES Descriptive Enumerative Analytic
DESCRIPTIVE STUDY Count all fish in barrel Count number of goldfish Proportion of goldfish applies to the fish population in this barrel and no other barrels of fish
ENUMERATIVE STUDY Take a sample of fish from the barrel, and count the number of goldfish in the sample Point estimate of the proportion of goldfish in the barrel population Many statistical procedures do this Cannot make any inference about any other barrels of fish
ANALYTICAL STUDY Will we get the same proportion of goldfish in the future as we got in the past? An analytical study allows prediction within limits Fish Packing Process over Time
ANALYTICAL STUDY Proportion of goldfish is stable over time Fish packing process is predictable within limits We can expect, on average, 4 goldfish per barrel, but as many as 10 and as few as 0 in any single barrel
ENUMERATIVE vs ANALYTICAL METHODS Enumerative methods –seek to provide numeric summaries, confidence intervals,etc –use significance tests, ANOVA, descriptive stats, etc., assume single, stable population Analytical methods –seek to understand the system under study –use primarily graphical tools such as run charts, control charts, histograms, box plots, etc –in the real world, most problems are analytical
“Analysis of variance, t-tests, confidence intervals, and other statistical techniques taught in books,….., are inappropriate because they provide no basis for prediction and because they bury the information contained in the order of production.” W.E. Deming, Out of the Crisis Traditional statistical methods have their place, but are widely abused in the real world. When this is the case, statistics do more to cloud the issue than to enlighten.
PARC ANALYSIS Practical Accumulated Records Compilation Passive Analysis (by) Regression Correlations Planning After Research Completed Profound Analysis Relying (on) Computers note inverse relationship with Continuous Recording (of) Administrative Procedures Constant Repetition (of) Anecdotal Perceptions
PLANNING A PROCESS IMPROVEMENT STUDY Why collect the data? What statistical methods for analysis? What data will be collected? How much data do we need? How will the data be measured? How good is the measurement system? When and where will data be collected? Who will collect the data? Remember:
GARBAGE IN – GARBAGE OUT
WHAT’S SIGNIFICANT? Two-sample T for C1 vs C2 N Mean StDev SE Mean A B Difference = mu (C1) - mu (C2) Estimate for difference: % CI for difference: ( , ) T-Test of difference = 0 (vs not =): T-Value = P-Value = DF = 8 Both use Pooled StDev = Two-sample T for C3 vs C4 N Mean StDev SE Mean A A Difference = mu (C3) - mu (C4) Estimate for difference: % CI for difference: ( , ) T-Test of difference = 0 (vs not =): T-Value = P-Value = DF = 398 Both use Pooled StDev = Mean A = 13.7, Mean B = 14.4 Not significant? Mean A = 13.5, Mean B = 13.7 Significant?
WHAT SHOULD I DO WITH OUTLIERS? Data point far away from the rest of the data Don’t remove outliers to make data “look good” Do you know why it is different? If you do, remove it. If you don’t, leave it in Could have a big impact on the analysis Re – run analysis without outlier, and compare results
“REGRESSION” WITH EXCEL Usually means drawing an X-Y plot, fitting a straight line and coming up with an R 2 value. As long as R 2 is high, everything’s hunky-dory. WRONG!
“REGRESSION” WITH EXCEL Relationship is clearly not linear, and should not be presented as such
“REGRESSION” WITH EXCEL Regression model checking – in Excel? Residual plots: –Normally distributed –Random pattern when plotted vs fitted values OKVariance not homogeneous Model incorrect
PITFALLS OF REGRESSION ANALYSIS Non-Linear Relationships Influential Points Extrapolating Lurking Variables Summary Data Assuming Causation
THAT’S (WITH REASONABLE PROBABILITY) THE END FOLKS! And remember, With statistics, you never have to say you’re certain!
THANK YOU FOR YOUR ATTENTION ARE THERE ANY QUESTIONS? GOOD LUCK!!