Chapter 231 Use and Abuse of Statistical Inference

Chapter 232 Thought Question 1 When presenting the results of a study, would it be sufficient to only report the P-value? Why would it be a good idea to also give a confidence interval based on the results?

Chapter 233 Thought Question 2 Suppose a new study found that there was no difference in lung function, measured by average volume of air expired, for smokers and nonsmokers. What may have led to this finding? Do you think the lung function was exactly the same for both groups in the study?

Chapter 234 Thought Question 3 The results of a CNN/USA Today/Gallup public opinion poll in August of 2005 showed that a majority of Americans were pro-choice on the abortion issue. Would it be fair to claim that “significantly more than 50% of Americans were pro- choice”? Explain.

Chapter 235 Thought Question 3: Answer n= stated that they were pro- choice 95% C.I.: to 0.571

Chapter 236 Warnings about Reports on Hypothesis Tests: Data Origins For any statistical analysis to be valid, the data must come from proper samples. Complex formulas and techniques cannot fix bad (biased) data. In addition, be sure to use an analysis that is appropriate for the type of data collected.

Chapter 237 Warnings about Reports on Hypothesis Tests: P-value or C.I.? P-values provide information as to whether findings are more than just good luck, but P-values alone may be misleading or leave out valuable information (as seen later in this chapter). Confidence intervals provide both the estimated values of important parameters and how uncertain the estimates are.

Chapter 238 Warnings about Reports on Hypothesis Tests: Significance If the word significant is used to try to convince you that there is an important effect or relationship, determine if the word is being used in the usual sense or in the statistical sense only.

Chapter 239 Case Study: Patient Satisfaction Bertakis, Klea D., et. al., “The influence of gender on physician practice style”, Medical Care, Vol. 33, No. 4, 1995, pp “Women Doctors Fare Better in Patient Survey” reported in Sacramento Bee, April 26, 1995

Chapter 2310 Case Study: Patient Satisfaction u Alternative (Research) Hypothesis: The mean satisfaction rating by patients who first saw a female physician is different from the mean satisfaction rating by patients who first saw a male physician. u Null Hypothesis: There is no difference in the mean satisfaction rating by patients who first saw a female physician and the mean satisfaction rating by patients who first saw a male physician.

Chapter 2311 Case Study: Patient Satisfaction u The alternative hypothesis is two-sided. u Study was double blinded (neither patients nor physicians were told the purpose of the survey). u Survey was completed by 250 patients at the University of California at Davis Medical Center who rated medical residents on a scale 1 to 5 (very dissatisfied to very satisfied).

Chapter 2312 Case Study: Patient Satisfaction u Bee: “The female physicians received an average score of The men – a respectable, yet significantly lower score of 4.05.” u The average difference was u Medical Care: the difference was “small but statistically significant (P-value=0.02).” u Medical Care: “This difference is both statistically and clinically significant.”

Chapter 2313 Warnings about Reports on Hypothesis Tests: Large Sample If a study is based on a very large sample size, relationships found to be statistically significant may not have much practical importance.

Chapter 2314 Case Study: Drug Use in American High Schools Bogert, Carroll. “Good news on drugs from the inner city,” Newsweek, Feb , pp Alcohol Use

Chapter 2315 Case Study: Drug Use in American High Schools u Alternative Hypothesis: The percentage of high school students who used alcohol in 1993 is less than the percentage who used alcohol in u Null Hypothesis: There is no difference in the percentage of high school students who used in 1993 and in 1992.

Chapter 2316 Case Study: Drug Use in American High Schools 1993 survey was based on 17,000 seniors, 15,500 10th graders and 18,500 8th graders.

Chapter 2317 Case Study: Drug Use in American High Schools u The article suggests that the survey reveals “good news” since the differences are all negative. u The differences are significant. –statistically? –practically?

Chapter 2318 Warnings about Reports on Hypothesis Tests: Small Sample If you read “no difference” or “no relationship” has been found in a study, try to determine the sample size used. Unless the sample size was large, remember that it could be that there is indeed an important relationship in the population, but that not enough data were collected to detect it. In other words, the test could have had very low power.

Chapter 2319 Case Study: Memory Loss Levy, B. and E. Langer. “Aging free from negative stereotypes: Successful memory in China and among the American deaf,” Journal of Personality and Social Psychology, Vol. 66, pp Memory Loss in American Hearing, American Deaf and Chinese Adults

Chapter 2320 Case Study: Memory Loss u Average Memory Test Scores (higher is better) u 30 subjects were sampled from each population

Chapter 2321 Case Study: Memory Loss u Young Americans (hearing and deaf) have significantly higher mean scores. u Science News (July 2, 1994, p. 13): “Surprisingly,...memory scores for older and younger Chinese did not statistically differ.”

Chapter 2322 Case Study: Memory Loss u Since the sample sizes are very small, there is an increased chance that the test will result in a Type II error if indeed there is a difference between young and old subjects’ mean memory scores. u The “surprising” result may just be a Type II error. u The test could have very low power.

Chapter 2323 Warnings about Reports on Hypothesis Tests: 1 or 2 Sided Try to determine whether the test was one- sided or two-sided. If a test is one-sided, and details are not reported, you could be misled into thinking there was no difference, when in fact there was one in the direction opposite to that hypothesized.

Chapter 2324 Case Study: Seen a UFO? Seen a UFO? You May Be Healthier Than Your Friends Roper Organization. Unusual Personal Experiences: An Analysis of the Data from Three National Surveys, Las Vegas: Bigelow Holding Corp., 1992.

Chapter 2325 Case Study: Seen a UFO? u Research Hypothesis (Alternative): People who claim to have seen a UFO are on average more psychologically disturbed than those who make no such claim. u Null Hypothesis: People who claim to have seen a UFO are on average no more or less psychologically disturbed than those who make no such claim.

Chapter 2326 Case Study: Seen a UFO? u 49 subjects were recruited through a newspaper. –18 were UFO nonintense –31 were UFO intense (could explain details of encounter) u 127 control subjects were recruited –74 students of a psychology class (receiving credit for participation) –53 community members recruited through a newspaper

Chapter 2327 Case Study: Seen a UFO? u New York Times (1993): “Study Finds No Abnormality in Those Reporting UFOs.” u Results: UFO groups actually scored significantly better (statistically) on many of the psychological measures. u The stated one-sided alternative hypothesis was not supported. Does this mean the null hypothesis is true?

Chapter 2328 Warnings about Reports on Hypothesis Tests: Only Significant are Reported? Sometimes researchers will perform a multitude of tests, and the reports will focus on those that achieved statistical significance. Remember that if nothing interesting is happening and all of the null hypotheses tested are true, then [about] 1 in 20 (.05) tests should achieve statistical significance just by chance. Beware of reports where it is evident that many tests were conducted, but where results of only one or two are presented as “significant.”

Chapter 2329 Case Study: Spinach is Good? So You Thought Spinach Was Good for You? Norwak, R. “Beta-carotene: Helpful or harmful?” Science, Vol. 264, April 22, 1994, pp

Chapter 2330 Case Study: Spinach is Good? u Startling finding: Supplements of the antioxidant beta-carotene markedly increased the incidence of lung cancer among heavy smokers in Finland. u This is the result of a large, randomized clinical trial: 29,000 cases u But…there were multiple tests conducted.

Chapter 2331 Key Concepts u Difference between a statistically significant effect and a practically important one u Large Samples and Statistical Significance u Small Samples and Statistical Significance u Multiple Tests and Statistical Significance