1 Statistics in Research & Things to Consider for Your Proposal May 2, 2007

2

3 “Statistical Conclusion” Validity The ability to trust what the statistics tell us Statistical Power –The ability to detect “true” relationships Assumptions of Statistics –The conditions under which the statistic will give us accurate results –Parametric versus Non-parametric tests Unreliability

4 Statistical Significance Testing

5 The Problems with SST doesWe misunderstand what it does tell us. notIt does not tell us what we want to know. We often overemphasize SST.

6 Four Important Questions 1.Is there a real relationship in the population? Statistical Significance 2.How large is the relationship? Effect Size or Magnitude 3.Is it a relationship that has important, powerful, useful, meaningful implications? Practical Significance 4.Why is the relationship there? ??????

7 SST is all about... Sampling ErrorSampling Error –The difference between what I see in my sample and what exists in the target population. –Simply because I sampled, I could be wrong. –This is a threat to Internal Validity

8 Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48,

9 How it works: 1.Assume sampling error occurred; there is no relationship in the population. null hypothesis 2.Build a statistical scenario based on this null hypothesis 3.How likely is it I got the sample value I got when the null hypothesis is true? (This is the fabled p-value.)

10 How it works (cont’d): How unlikely does my result have to be to rule out sampling error? alpha (  ). when there isn’t a relationship in the population.If p < , then our result is statistically rare, is unlikely to occur when there isn’t a relationship in the population.

11 does What it does tell us What is the probability that we would see a relationship in our sample when there is no relationship in the population? Can we rule out sampling error as a competing hypothesis for our finding?

12 does not What it does not tell us Whether the null hypothesis is true. Whether our results will replicate. Whether our research hypothesis is true. How big the effect or relationship is. How important the results are. Why there is a relationship.

13 Carver’s Scientific Method Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48,

14 Carver’s Corrupt Scientific Method Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48,

15 What To Do Always use the phrase statistically significant Always report effect size or measures of association in addition to SST. Replicate and/or Cross-validate What does it mean? -- Discuss Practical Significance

16 Statistical Methods in Psychology Journals (Wilkinson, 1999) Nice article –Check your proposal to make sure it addresses the points made in this article Design –Make design clear Do not cloak the design –Each form of research has its own strengths, weaknesses and standards of practice. Population –Define clearly

17 Wilkinson (cont.) Sample –How was the population sampled? –Does it represent the target population? –Describe recruitment procedures –Emphasize inclusion and exclusion criteria –If stratified explain why Note sample sizes for subgroups Assignment –Random vs. Nonrandom Control of confounds

18 Wilkinson (cont.) Measurement –Variables Explicitly state and explain how they are measured –How do they vary? –Instruments What evidence exists for the usage of the scores? –Reliability »Internal consistency, test-retest, etc. –Validity »Predictive, concurrent, etc.

19 Wilkinson (cont.) Procedures –Attrition How will you prevent it? –How will you control for threats to internal validity? Power and Sample size –Do you have a large enough sample? Based on previous studies Pilot work Power analysis

20 Wilkinson (cont.) Results –Before presenting results Report –Complications –Protocol violations –Missing data Look at your data –Make sure that there are no errors in the data –Look at histograms, means, etc.

21 Wilkinson (cont.) Analysis –KISS Keep it simple ****** Design studies that you know how to analyze Beware of computer programs –You can perform incredibly complex analyses –Assumptions Check them –Normality, independence, etc.

22 Wilkinson (cont.) Hypothesis tests –Report p values Effect sizes Interval estimates Causality –Be careful –Alert reader to plausible rival hypotheses

23 Wilkinson (cont.) Discussion –Try not to Overgeneralize Tear your study apart –Relate to other studies Conclusions –Speculation Use sparingly –Do not interpret study’s importance independent of other studies.

24 Random Thoughts on the (In)credibility of Educational-Psychological Intervention Research (Levin, 2004) Random Selection vs. Random Assignment –How are they different? The concept of evidence –“Show me the data” (C)omparison (A)gain and again (R)elationship (E)liminating other plausible explanations

25 Medical Comparison (Levin, cont.) Clinical Trials –Phase I: Maximum tolerated dose Side effects, etc. –Phase II: Biological activity Does the drug have the effect of interest? –Phase III: Is the drug, or treatment, effective when compared to an alternative? –Phase IV: Is the drug, or treatment, effective in the long run?

26 Levin (cont.) Educational Research –Phase I: Idea generation, theory building, pilot work, etc. –Phase II: Demonstrations Design Experiments –Phase III: Randomized classroom trials

27 Levin (cont.) Three critical considerations in prescribing research-based educational interventions –Scientific credibility Internal and statistical validity –Contextual Accretability Construct and external validity –Educational Creditability Social validity

28 In Pairs: –Each person should: Summarize his/her study in 3 minutes or less. Discuss explicitly the role of causality in each study (5 Minutes). –What in the design aids that causal argument? – What inhibits the causal argument? Then read your partners methods section –Does it make sense? –Are steps in logical order? –Does it use proper terminology? –What threats to internal and external validity exist? –Repeat with the other person.

29 Basic Outline –Problem to be investigated Purpose & Justification Literature review –Theory and definitions –Hypotheses –Methods Sampling –Including human subjects. Instrumentation –How you will measure each IV and DV. »Be sure to identify IVs and DVs –Reliability & Validity Design –Experiment, correlational, etc. Procedural Detail –What will happen and when it will happen. Data Analyses –Strengths and Limitations Internal and External.