1. Copyright © 2009 Pearson Education, Inc. 1.4 Should You Believe a Statistical Study? LEARNING GOAL Be able to evaluate statistical studies that you hear about in the media, so that you can decide whether the results are meaningful.

2. Slide 1.4- 2 Copyright © 2009 Pearson Education, Inc. Eight Guidelines for Critically Evaluating a Statistical Study 1.Identify the goal of the study, the population considered, and the type of study. 2.Consider the source, particularly with regard to whether the researcher may be biased. 3.Examine the sampling method to decide whether it is likely to produce a representative sample. 4.Look for problems in defining or measuring the variables of interest, which can make it difficult to interpret any reported results.

3. Slide 1.4- 3 Copyright © 2009 Pearson Education, Inc. 5. Watch out for confounding variables that can invalidate the conclusions of a study. 6.Consider the setting and wording in surveys or polls, looking for anything that might tend to produce inaccurate or dishonest responses. 7.Check that results are fairly represented in graphics and concluding statements, because both researchers and media often create misleading graphics or jump to conclusions that the results do not support. 8.Stand back and consider the conclusions. Did the study achieve its goals? Do the conclusions make sense? Do the results have any practical significance? Eight Guidelines for Critically Evaluating a Statistical Study

4. Slide 1.4- 4 Copyright © 2009 Pearson Education, Inc. Guideline 1: Identify the Goal, Population and Type of Study Based on what you hear or read about a study, try to answer these basic questions: What was the study designed to determine? What was the population under study? Was the population clearly and appropriately defined? Was the study an observational study, an experiment, or a meta-analysis?

5. Slide 1.4- 5 Copyright © 2009 Pearson Education, Inc. Imagine the following (hypothetical) newspaper report: “Researchers gave 100 participants their individual astrological horoscopes and asked whether the horoscopes were accurate. 85% of the participants said their horoscopes were accurate. The researchers concluded that horoscopes are valid most of the time.” Analyze this study according to Guideline 1. Solution: The goal of the study was to determine the validity of horoscopes. Based on the news report, it appears that the study was observational: The researchers simply asked the participants about the accuracy of the horoscopes. EXAMPLE 1 Appropriate Type of Study?

6. Slide 1.4- 6 Copyright © 2009 Pearson Education, Inc. Solution: (cont.) However, because the accuracy of a horoscope is somewhat subjective, this study should have been a controlled experiment in which some people were given their actual horoscope and others were given a fake horoscope. Then the researchers could have looked for differences between the two groups. Moreover, because researchers could easily influence the results by how they questioned the participants, the experiment should have been double-blind. In summary, the type of study was inappropriate to the goal and its results are meaningless. EXAMPLE 1 Appropriate Type of Study?

7. Slide 1.4- 7 Copyright © 2009 Pearson Education, Inc. TIME OUT TO THINK Try your own test of horoscopes. Find yesterday’s horoscope for each of the 12 signs and put each one on a separate piece of paper, without anything identifying the sign. Shuffle the pieces of paper randomly, and ask a few people to guess which one was supposed to be their personal horoscope. How many people choose the right one? Discuss your results.

8. Slide 1.4- 8 Copyright © 2009 Pearson Education, Inc. Guideline 2: Consider the Source Statistical studies are supposed to be objective, but the people who carry them out and fund them may be biased. It is therefore important to consider the source of a study and evaluate the potential for biases that might invalidate the study’s conclusions. Definition Peer review is a process in which several experts in a field evaluate a research report before the report is published.

9. Slide 1.4- 9 Copyright © 2009 Pearson Education, Inc. By 1963, research had so clearly shown the health dangers of smoking that the Surgeon General of the United States publicly announced that smoking is bad for health. Research done since that time built further support for this claim. However, while the vast majority of studies showed that smoking is unhealthy, a few studies found no dangers from smoking and perhaps even health benefits. These studies generally were carried out by the Tobacco Research Institute, funded by the tobacco companies. Analyze these studies according to Guideline 2. EXAMPLE 3 Is Smoking Healthy?

10. Slide 1.4- 10 Copyright © 2009 Pearson Education, Inc. Solution: EXAMPLE 3 Is Smoking Healthy? Even in a case like this, it can be difficult to decide whom to believe. However, the studies showing smoking to be unhealthy came primarily from peer-reviewed research. In contrast, the studies carried out at the Tobacco Research Institute had a clear potential for bias. The potential for bias does not mean the research was biased, but the fact that it contradicts virtually all other research on the subject should be cause for concern.

11. Slide 1.4- 11 Copyright © 2009 Pearson Education, Inc. Guideline 3: Examine the Sampling Method A statistical study cannot be valid unless the sample is representative of the population under study. Definitions Selection bias (or a selection effect) occurs whenever researchers select their sample in a biased way. Participation bias occurs any time participation in a study is voluntary. A self-selected survey (or voluntary response survey) is one in which people decide for themselves whether to be included in the survey.

12. Slide 1.4- 12 Copyright © 2009 Pearson Education, Inc. The television show Nightline conducted a poll in which viewers were asked whether the United Nations headquarters should be kept in the United States. Viewers could respond to the poll by paying 50 cents to call a “900” phone number with their opinions. The poll drew 186,000 responses, of which 67% favored moving the United Nations out of the United States. Around the same time, a poll using simple random sampling of 500 people found that 72% wanted the United Nations to stay in the United States. Which poll is more likely to be representative of the general opinions of Americans? EXAMPLE 5 Self-Selected Poll

13. Slide 1.4- 13 Copyright © 2009 Pearson Education, Inc. Solution: EXAMPLE 5 Self-Selected Poll The Nightline sample was severely biased. It had selection bias because its sample was drawn only from the show’s viewers, rather than from all Americans. The poll itself was a self-selected survey in which viewers not only chose whether to respond, but also had to pay 50 cents to participate. This cost made it even more likely that respondents would be those who felt a need for change. Despite its large number of respondents, the Nightline survey was therefore unlikely to give meaningful results. In contrast, the simple random sample of 500 people is quite likely to be representative, so the finding of this small survey has a better chance of representing the true opinions of all Americans.

14. Slide 1.4- 14 Copyright © 2009 Pearson Education, Inc. Until the mid-1990s, astronomers had never found conclusive evidence for planets outside our own solar system. But improving technology made it possible to begin finding such planets, and more than 200 had been discovered by early 2007. The existing technology makes it easier to find large planets than small ones and easier to find planets that orbit close to their stars than planets that orbit far from their stars. According to the leading theory of solar system formation, large planets that orbit close to their stars should be very rare. But they are fairly common among the first 200 discoveries. Does this mean there is something wrong with the leading theory of solar system formation? EXAMPLE 6 Planets Around Other Stars

15. Slide 1.4- 15 Copyright © 2009 Pearson Education, Inc. Solution: EXAMPLE 6 Planets Around Other Stars Although the theory suggests that large planets in close orbits should be rare, the technology makes these rare cases the easiest ones to find. The finding of many of these planets may therefore represent a selection effect that biases the sample (of discovered planets) toward a rare type. In fact, this seems almost certainly to be the case, because many “normal” planets are now being discovered, and astronomers have found that with slight modifications the existing theory can account for the rare types found in early discoveries.

16. Slide 1.4- 16 Copyright © 2009 Pearson Education, Inc. Guideline 4: Look for Problems in Defining or Measuring the Variable of Interest Results of a statistical study may be difficult to interpret if the variables under study are difficult to define or measure.

17. Slide 1.4- 17 Copyright © 2009 Pearson Education, Inc. TIME OUT TO THINK How would you measure your resting heart rate? Describe some difficulties in defining and measuring resting heart rate.

18. Slide 1.4- 18 Copyright © 2009 Pearson Education, Inc. A commonly quoted statistic is that law enforcement authorities succeed in stopping only about 10% to 20% of the illegal drugs entering the United States. Should you believe this statistic? EXAMPLE 8 Illegal Drug Supply Solution: There are essentially two variables in a study of illegal drug interception: quantity of illegal drugs intercepted and quantity of illegal drugs NOT intercepted. It should be relatively easy to measure the quantity of illegal drugs that law enforcement officials intercept. However, because the drugs are illegal, it’s unlikely that anyone is reporting the quantity of drugs that are not intercepted.

19. Slide 1.4- 19 Copyright © 2009 Pearson Education, Inc. Solution: (cont.) How, then, can anyone know that the intercepted drugs are 10% to 20% of the total? In a New York Times analysis, a police officer was quoted as saying that his colleagues refer to this type of statistic as “PFA” for “pulled from the air.” EXAMPLE 8 Illegal Drug Supply

20. Slide 1.4- 20 Copyright © 2009 Pearson Education, Inc. Guideline 5: Watch Out for Confounding Variables Often, variables that are not intended to be part of the study can make it difficult to interpret results properly.

21. Slide 1.4- 21 Copyright © 2009 Pearson Education, Inc. Radon is a radioactive gas produced by natural processes (the decay of uranium) in the ground. The gas can leach into buildings through the foundation and can accumulate to relatively high concentrations if doors and windows are closed. Imagine a (hypothetical) study that seeks to determine whether radon gas causes lung cancer by comparing the lung cancer rate in Colorado, where radon gas is fairly common, with the lung cancer rate in Hong Kong, where radon gas is less common. Suppose the study finds that the lung cancer rates are nearly the same. Would it be reasonable to conclude that radon is not a significant cause of lung cancer? EXAMPLE 9 Radon and Lung Cancer

22. Slide 1.4- 22 Copyright © 2009 Pearson Education, Inc. Solution: The variables of interest are amount of radon (an explanatory variable in this case) and lung cancer rate (a response variable). However, radon gas is not the only possible cause of lung cancer. For example, smoking can cause lung cancer, so smoking rate may be a confounding variable in this study—especially because the smoking rate in Hong Kong is much higher than the smoking rate in Colorado. As a result, we cannot draw any conclusions about radon and lung cancer without taking the smoking rate into account (and perhaps other variables as well). EXAMPLE 9 Radon and Lung Cancer

23. Slide 1.4- 23 Copyright © 2009 Pearson Education, Inc. Solution: (cont.) In fact, careful studies have shown that radon gas can cause lung cancer, and the U.S. Environmental Protection Agency (EPA) recommends taking steps to prevent radon from building up indoors. EXAMPLE 9 Radon and Lung Cancer

24. Slide 1.4- 24 Copyright © 2009 Pearson Education, Inc. Guideline 6: Consider the Setting and Wording in Surveys Even when a survey is conducted with proper sampling and with clearly defined terms and questions, you should watch for problems in the setting or wording that might produce inaccurate or dishonest responses. Dishonest responses are particularly likely when the survey concerns sensitive subjects.

25. Slide 1.4- 25 Copyright © 2009 Pearson Education, Inc. At a time when the U.S. government was running annual budget surpluses, Republicans in Congress proposed a tax cut and the Republican National Committee commissioned a poll to find out whether Americans supported the proposal. Asked “Do you favor a tax cut?,” 67% of respondents answered yes. Should we conclude that Americans supported the proposal? EXAMPLE 10 Do You Want a Tax Cut?

26. Slide 1.4- 26 Copyright © 2009 Pearson Education, Inc. Solution: EXAMPLE 10 Do You Want a Tax Cut? A question like “Do you favor a tax cut?” is biased because it does not give other options. In fact, an independent poll conducted at the same time gave respondents a list of options for using surplus revenues. This poll found that 31% wanted the money devoted to Social Security, 26% wanted it used to reduce the national debt, and only 18% favored using it for a tax cut. (The remaining 25% of respondents chose a variety of other options.)

27. Slide 1.4- 27 Copyright © 2009 Pearson Education, Inc. Guideline 7: Check That Results Are Fairly Represented in Graphics or Concluding Statements Even when a statistical survey is done well, it may be misrepresented in graphics or concluding statements.

28. Slide 1.4- 28 Copyright © 2009 Pearson Education, Inc. The school board in Boulder, Colorado, created a hubbub when it announced that 28% of Boulder school children were reading “below grade level” and hence concluded that methods of teaching reading needed to be changed. The announcement was based on reading tests on which 28% of Boulder school children scored below the national average for their grade. Do these data support the board’s conclusion? EXAMPLE 12 Does the School Board Need a Statistics Lesson? Solution: The fact that 28% of Boulder children scored below the national average for their grade implies that 72% scored at or above the national average.

29. Slide 1.4- 29 Copyright © 2009 Pearson Education, Inc. Solution: (cont.) Thus, the school board’s ominous statement about students reading “below grade level” makes sense only if “grade level” means the national average score for a particular grade. This interpretation of “grade level” is curious because it would imply that half the students in the nation are always below grade level—no matter how high the scores. It may still be the case that teaching methods needed to be improved, but these data did not justify that conclusion. EXAMPLE 12 Does the School Board Need a Statistics Lesson?

30. Slide 1.4- 30 Copyright © 2009 Pearson Education, Inc. Guideline 8: Stand Back and Consider the Conclusions Ask yourself questions such as these: Did the study achieve its goals? Do the conclusions make sense? Can you rule out alternative explanations for the results? If the conclusions make sense, do they have any practical significance?

31. Slide 1.4- 31 Copyright © 2009 Pearson Education, Inc. Suppose a (hypothetical) study concludes that wearing a gold chain increases your chances of surviving a car accident by 10%. The claim is based on a statistical analysis of data about survival rates and what people were wearing. Careful analysis of the research shows that it was conducted properly and carefully. Should you start wearing a gold chain whenever you drive a car? EXAMPLE 13 Extraordinary Claims Solution: Despite the care that went into the study, the claim that a gold chain can save your life in a collision is difficult to believe. After all, how could a thin chain help in a highspeed collision?

32. Slide 1.4- 32 Copyright © 2009 Pearson Education, Inc. Solution: (cont.) It’s certainly possible that some unknown effect of gold chains makes the conclusion correct, but it seems far more likely that the results were either a fluke or due to an unidentified confounding variable (for example, perhaps those who wear gold chains are wealthier and drive newer cars with more advanced safety features, lowering their fatality rate). EXAMPLE 13 Extraordinary Claims

33. Slide 1.4- 33 Copyright © 2009 Pearson Education, Inc. The End