1. Statistics Micro Mini Threats to Your Experiment!
January 7-11, 2008
Beth Ayers
Threats to Your Experiment

2. Threats to Your Experiment
Here I am using threats to mean things that will reduce the impact, credibility, or generalizability of the results your experiment or study, particularly the things that we have control over
Threats to Your Experiment

3. Threats to Your Experiment
Validity
Statistical conclusion validity
Internal validity
Construct validity
External validity
etc.
Type I Error
Power
Threats to Your Experiment

4. A note of caution Many of these ideas are inter-related
I expect some of this to be confusing at first, it will help to reread the notes once you’ve seen all the terminology
Threats to Your Experiment

5. What is Validity?
In general, validity refers to the degree to which a study supports the intended conclusion drawn from the results
Validity is broken down into many different types
A Google search will yield many pages explaining different types
At a quick glance Wikipedia articles seem to have many of the types and are correct
Threats to Your Experiment

6. (Statistical) Conclusion Validity
Conclusion validity is the degree to which conclusions we reach about our data are reasonable
In particular, is there a relationship between the two variables?
Threats to Your Experiment

7. Threats to Conclusion Validity
Concluding there is a relationship when in fact there is not one – Type I Error
Concluding there is not a relationship when in fact there is one – Type II Error
Low statistical power
Violation of assumptions
Fishing for results
Threats to Your Experiment

8. Controlling Conclusion Validity Threats
When testing multiple hypotheses, adjust the error rate, see slide 20
Take reliable measurements
Increase power (see later definitions)
Use the right statistical test and perform it correctly!!!
Threats to Your Experiment

9. Association does not imply causation!
Internal Validity
Internal validity is the degree to which we can conclude that the changes in the explanatory variable caused the changes in the response variable
Remember
Association does not imply causation!
Threats to Your Experiment

10. Threats to Internal Validity
Confounding – in the form of a lurking variable
Selection Bias – before treatment there are differences between the groups
History – some historical event affects the study outcome
Maturation – if making observations at different times points, the natural aging of the subjects may invalidate results
Instrument change – not using the same measurement tools or methods with each subject
Repeated testing – subjects may remember answers between testing sessions
Experimenter bias – the experimenter treats subjects differently based on treatment type
Threats to Your Experiment

11. Controlling Internal Validity Threats
Random assignment of treatment or randomization
Blinding – subjects don’t know which treatment they are receiving
Double blinding – neither subjects nor experimenters know which treatment subjects are receiving
Threats to Your Experiment

12. Construct Validity
Construct validity refers to the degree to which inferences can be made from the measurements in your study
In particular, are you manipulating what you claim you are manipulating (the causal construct) and are you measuring what you claim you are measuring (the effect construct)
Often defined with multiple subcategories of validity
Threats to Your Experiment

13. Threats to Construct Validity
Poor study design
Using new or unreliable methods of measurement
Measurements depend on who is collecting them
Interaction of testing and treatment
Interaction of treatments
Threats to Your Experiment

14. Controlling Construct Validity Threats
Carefully design your study
Have others critique your design
Measurements should be reliable and not depend on who is administering your test
If there is a “gold standard” of measurement for your response variable, your measure should have a high correlation with that test
Threats to Your Experiment

15. External Validity
External validity is the extent to which we can generalize the findings to particular target persons, places, or times and to which we can generalize across types of persons, places, or times
Often simply called generalizability
Threats to Your Experiment

16. Threats to External Validity
The particular group of people used in your study
The particular place where you performed your study
The particular time at which you performed your study
Threats to Your Experiment

17. Controlling External Validity Threats
Make sure to get a representative sample of the population
Do your study in a variety of places, with different people, and at different times
Threats to Your Experiment

18. Validity Questions Are Cumulative
Conclusion – is there a relationship between the explanatory and response variables?
Internal – is the relationship causal?
Construct – can we generalize to the construct?
External – can we generalize to other persons, places, times?
Threats to Your Experiment

19. Type I Error
Although this was mentioned in conclusion validity, it is worth discussing again
Due to the inherent uncertainties of nature, we can never make definite claims from our experiments
Retain H0
Reject H0
H0 is true
Correct
Type I error
False discovery
H1 is true
Type II error
Threats to Your Experiment

20. Type I Error
However, we can set a limit on now often we will make a false claim
Done by setting the ®-level of a test
Do not “data snoop”
Performing many tests on your data until you find a significant result
Correct for multiple testing
Threats to Your Experiment

21. Correcting for Multiple Testing
When performing hypothesis tests, 100*® will result in Type I errors
When doing many tests, need to lower the error rate
Several common methods are:
Bonferroni correction
Benjamini-Hochberg method
False discovery rate
Threats to Your Experiment

22. Statistical Power
The power of an experiment refers to the probability that we will correctly conclude that the treatment caused change in the outcome given that it actually does
Low power leads to Type II Errors
Have some control of the power of your experiments prior to running them
Performing experiments with low power is a waste of time and money!
Threats to Your Experiment

23. Statistical Power The power varies from ® to 1
Typically people agree that 80% power is a minimal value for good power
Poor power is a common problem. It can NOT be fixed with statistical analysis. It must be dealt with before running your experiment.
Threats to Your Experiment

24. Statistical Power Statistical power depends on
the ®-level of the test
the size of the difference or the strength of the similarity (that is, the effect size) in the population
the sensitivity or variability of the data
More power to detect larger effects than smaller ones
Threats to Your Experiment

25. Ways to Increase Power Increase sample size Reduce variability
Increase the spacing between population means
Threats to Your Experiment

26. Increasing Sample Size
Unfortunately, this is not always an option
Not always cost effective
Threats to Your Experiment

27. Reducing Variability Types of variability
Measurement
Environment
Treatment application
Subject-to-subject
There is a trade-off between reduced variability and external validity
If we to do much to control the variability, we lose generalizability
Threats to Your Experiment

28. What can power calculations do?
Suppose we’re creating an on-line tutor to help students study for the SATs. Since we’d like to sell the package for $200, we believe that we’ll need a 100 point increase in SAT scores in order for parents to consider the tutor worthwhile.
Given a sample size, an ®-level, and an estimate of the variability of SAT scores, we can calculate the power of our experiment
Given a power we’d like to have, an ®-level, and an estimate of the variability of SAT scores, we can calculate the sample size needed to obtain that power
Threats to Your Experiment

29. Notes on Power Calculating power can be hard…
However, there are many on-line applets to help you calculate power!
As we go through examples I will show you one such applet
Find one you understand and can use!
Threats to Your Experiment