1. Statistical Inference What can go wrong?

2. Tests of significance are an important part of much of scientific study today. The validity of the conclusions of scientific studies are supported by the same statistical tests used in this course, and on tests that frequently rely on the same underlying principles.

3. I. Choosing a significance level, α, can cause some level of confusion. 1) α can be too large 2) α can be too small This can lead us to reject null hypotheses that really are true. This can lead us to fail to reject null hypotheses that really are not true.

4. What happens when I am wrong? Choosing α requires knowledge of the subject being studied and depends on the answer to the following questions: We will also see that there is more than one way to be wrong. Don’t feel like it’s hopeless! Is there a special importance to this study? Does this go against established knowledge?

5. II. Statistical significance doesn’t always mean practical significance. Often it does, but consider: There is a new treatment for septic shock, an often fatal illness. This treatment has shown, with great statistical significance, that persons undergoing this treatment live longer.

6. Sounds impressive? Suppose it turns out that patients undergoing this treatment live, on average, 1.1 days longer than those not receiving the treatment. The lifespan was longer, but not enough longer to necessarily make the treatment useful.

7. III. Maybe you want to carry out many tests… You might have a lot of data and want to know if any of it is significant.

8. If you carry out 100 tests, each with an α of 0.05, how many tests will be significant, just by chance alone, even if each H 0 is true? This means that 5 tests, on average, will have a p-value small enough to cause us to reject H 0 even though it is true.

9. IV. Finally, if your experiment has a design flaw, or was not carried out appropriately, then any conclusion you draw may be suspect. What can you do?

10. Plan for your statistical analysis as you design your experiment. Make sure that your sample sizes are large enough to meet your needs. Make sure you have random assignment to treatment. Consider the consequences of being wrong, versus the benefits of being right. Consider the scientific strengths and weaknesses of your experiment.

11. The End