Hypothesis Testing Part IV – Practical Significance

This video is designed to accompany pages in Making Sense of Uncertainty Activities for Teaching Statistical Reasoning Van-Griner Publishing Company

Multivitamins and Cancer

Actual difference was 8% fewer cancers in treatment group. You can think of this as the actual “effect size” The fact that this difference was statistically significant means that it was big enough that the alternative could be accepted with a Type I error rate (false positive rate) that was low enough (probably 0.05) to be tolerable. This does not mean that the difference is big enough to care about clinically. When it comes to cancer reduction, though, 8% might catch our attention. Still, this question is not answered in the article, and the author does call the effect “small.”

Mental Illness and Obesity

Many fitness programs achieved “statistically significant” weight loss results. That means that while in the program the weights of participants dropped enough to make it safe to infer that there would be weight loss in the larger population. But the authors also note that in most of these programs the weight loss was not “clinically significant.” They say elsewhere in the article that in this context clinically significant means more than 5% of one’s body weight.

Apnea in Your Stockings

Statistically significant reduction. Effect size seems to be about a 36% reduction in apnea episodes. Average number of episodes decreased from 48 per hour to 31 per hour. But the blog author is concerned because this still leaves the “average” severe patient in the severe category.

Coffee and Pregnancy

Women who had less than 200 mg of coffee a day experienced a 40% increase in miscarriage risk compared to women who had no caffeine during pregnancy. Dr. Aaron Caughey, a perinatologist at UCSF, is quoted in the article as saying “I would probably not even recommend a cup a day, based on this.” Statistically “safe” is not ultimately always reassuring. There is solid evidence to suggest that a lot of practically meaningful findings may be getting suppressed because they are not statistically significant.

One-Sentence Reflection Statistical significance is a mathematical way of assessing whether treatment differences are big enough to be considered unlikely to have happened by chance; whereas practical significance addresses whether the observed difference is big enough that it is practically worth caring about.