1. Copyright © 2014 Pearson Education. All rights reserved. 1.4-1 1.4 Should You Believe a Statistical Study? LEARNING GOAL Be able to evaluate statistical studies that you hear about in the media, so that you can decide whether the results are meaningful.

2. Copyright © 2014 Pearson Education. All rights reserved. 1.4-2 Slide 1.4- 2 Eight Guidelines for Critically Evaluating a Statistical Study 1.Identify the goal of the study, the population considered, and the type of study. 2.Consider the source, particularly with regard to whether the researcher may be biased. 3.Examine the sampling method to decide whether it is likely to produce a representative sample. 4.Look for problems in defining or measuring the variables of interest, which can make it difficult to interpret any reported results.

3. Copyright © 2014 Pearson Education. All rights reserved. 1.4-3 Slide 1.4- 3 5. Watch out for confounding variables that can invalidate the conclusions of a study. 6.Consider the setting and wording in surveys or polls, looking for anything that might tend to produce inaccurate or dishonest responses. 7.Check that results are fairly represented in graphics and concluding statements, because both researchers and media often create misleading graphics or jump to conclusions that the results do not support. 8.Stand back and consider the conclusions. Did the study achieve its goals? Do the conclusions make sense? Do the results have any practical significance? Eight Guidelines for Critically Evaluating a Statistical Study

4. Copyright © 2014 Pearson Education. All rights reserved. 1.4-4 Slide 1.4- 4 Guideline 1: Identify the Goal, Population and Type of Study Based on what you hear or read about a study, try to answer these basic questions: What was the study designed to determine? What was the population under study? Was the population clearly and appropriately defined? Was the study an observational study, an experiment, or a meta-analysis?

5. Copyright © 2014 Pearson Education. All rights reserved. 1.4-5 Slide 1.4- 5 Imagine the following (hypothetical) newspaper report: “Researchers gave 100 participants their individual astrological horoscopes and asked whether the horoscopes were accurate. 85% of the participants said their horoscopes were accurate. The researchers concluded that horoscopes are valid most of the time.” Analyze this study according to Guideline 1. Solution: EXAMPLE 1 Appropriate Type of Study?

6. Copyright © 2014 Pearson Education. All rights reserved. 1.4-6 Slide 1.4- 6 Guideline 2: Consider the Source Statistical studies are supposed to be objective, but the people who carry them out and fund them may be biased. It is therefore important to consider the source of a study and evaluate the potential for biases that might invalidate the study’s conclusions. Definition __________________is a process in which several experts in a field evaluate a research report before the report is published.

7. Copyright © 2014 Pearson Education. All rights reserved. 1.4-7 Slide 1.4- 7 By 1963, research had so clearly shown the health dangers of smoking that the Surgeon General of the United States publicly announced that smoking is bad for health. Research done since that time built further support for this claim. However, while the vast majority of studies showed that smoking is unhealthy, a few studies found no dangers from smoking and perhaps even health benefits. These studies generally were carried out by the Tobacco Research Institute, funded by the tobacco companies. Analyze these studies according to Guideline 2. EXAMPLE 2 Is Smoking Healthy?

8. Copyright © 2014 Pearson Education. All rights reserved. 1.4-8 Slide 1.4- 8 Guideline 3: Examine the Sampling Method A statistical study cannot be valid unless the sample is representative of the population under study. Definitions _____________(or a selection effect) occurs whenever researchers select their sample in a biased way. ____________________________occurs any time participation in a study is voluntary. A _________________(or voluntary response survey) is one in which people decide for themselves whether to be included in the survey.

9. Copyright © 2014 Pearson Education. All rights reserved. 1.4-9 Slide 1.4- 9 The television show Nightline conducted a poll in which viewers were asked whether the United Nations headquarters should be kept in the United States. Viewers could respond to the poll by paying 50 cents to call a “900” phone number with their opinions. The poll drew 186,000 responses, of which 67% favored moving the United Nations out of the United States. Around the same time, a poll using simple random sampling of 500 people found that 72% wanted the United Nations to stay in the United States. Which poll is more likely to be representative of the general opinions of Americans? (Analyze this study according to Guideline 3). EXAMPLE 3 Self-Selected Poll

10. Copyright © 2014 Pearson Education. All rights reserved. 1.4-10 Slide 1.4- 10 Until the mid-1990s, astronomers had never found conclusive evidence for planets outside our own solar system. But improving technology made it possible to begin finding such planets, and more than 200 had been discovered by early 2007. The existing technology makes it easier to find large planets than small ones and easier to find planets that orbit close to their stars than planets that orbit far from their stars. According to the leading theory of solar system formation, large planets that orbit close to their stars should be very rare. But they are fairly common among the first 200 discoveries. Does this mean there is something wrong with the leading theory of solar system formation? (Analyze this study according to Guideline 3.) EXAMPLE 4 Planets Around Other Stars

11. Copyright © 2014 Pearson Education. All rights reserved. 1.4-11 Slide 1.4- 11 Guideline 4: Look for Problems in Defining or Measuring the Variable of Interest Results of a statistical study may be difficult to interpret if the variables under study are difficult to define or measure.

12. Copyright © 2014 Pearson Education. All rights reserved. 1.4-12 Slide 1.4- 12 A commonly quoted statistic is that law enforcement authorities succeed in stopping only about 10% to 20% of the illegal drugs entering the United States. Should you believe this statistic? (Analyze this study according to Guideline 4). EXAMPLE 5 Illegal Drug Supply

13. Copyright © 2014 Pearson Education. All rights reserved. 1.4-13 Slide 1.4- 13 Guideline 5: Watch Out for Confounding Variables Often, variables that are not intended to be part of the study can make it difficult to interpret results properly.

14. Copyright © 2014 Pearson Education. All rights reserved. 1.4-14 Slide 1.4- 14 Radon is a radioactive gas produced by natural processes (the decay of uranium) in the ground. The gas can leach into buildings through the foundation and can accumulate to relatively high concentrations if doors and windows are closed. Imagine a (hypothetical) study that seeks to determine whether radon gas causes lung cancer by comparing the lung cancer rate in Colorado, where radon gas is fairly common, with the lung cancer rate in Hong Kong, where radon gas is less common. Suppose the study finds that the lung cancer rates are nearly the same. Would it be reasonable to conclude that radon is not a significant cause of lung cancer? (Analyze this study according to Guideline 5). EXAMPLE 6 Radon and Lung Cancer

15. Copyright © 2014 Pearson Education. All rights reserved. 1.4-15 Slide 1.4- 15 Guideline 6: Consider the Setting and Wording in Surveys Even when a survey is conducted with proper sampling and with clearly defined terms and questions, you should watch for problems in the setting or wording that might produce inaccurate or dishonest responses. Dishonest responses are particularly likely when the survey concerns sensitive subjects.

16. Copyright © 2014 Pearson Education. All rights reserved. 1.4-16 Slide 1.4- 16 At a time when the U.S. government was running annual budget surpluses, Republicans in Congress proposed a tax cut and the Republican National Committee commissioned a poll to find out whether Americans supported the proposal. Asked “Do you favor a tax cut?,” 67% of respondents answered yes. Should we conclude that Americans supported the proposal? Analyze this study according to Guideline 6. EXAMPLE 7 Do You Want a Tax Cut?

17. Copyright © 2014 Pearson Education. All rights reserved. 1.4-17 Slide 1.4- 17 Guideline 7: Check That Results Are Fairly Represented in Graphics or Concluding Statements Even when a statistical survey is done well, it may be misrepresented in graphics or concluding statements.

18. Copyright © 2014 Pearson Education. All rights reserved. 1.4-18 Slide 1.4- 18 The school board in Boulder, Colorado, created a hubbub when it announced that 28% of Boulder school children were reading “below grade level” and hence concluded that methods of teaching reading needed to be changed. The announcement was based on reading tests on which 28% of Boulder school children scored below the national average for their grade. Do these data support the board’s conclusion? Analyze this study according to Guideline 7. EXAMPLE 8 Does the School Board Need a Statistics Lesson?

19. Copyright © 2014 Pearson Education. All rights reserved. 1.4-19 Slide 1.4- 19 Guideline 8: Stand Back and Consider the Conclusions Ask yourself questions such as these: Did the study achieve its goals? Do the conclusions make sense? Can you rule out alternative explanations for the results? If the conclusions make sense, do they have any practical significance?

20. Copyright © 2014 Pearson Education. All rights reserved. 1.4-20 Slide 1.4- 20 Suppose a (hypothetical) study concludes that wearing a gold chain increases your chances of surviving a car accident by 10%. The claim is based on a statistical analysis of data about survival rates and what people were wearing. Careful analysis of the research shows that it was conducted properly and carefully. Should you start wearing a gold chain whenever you drive a car? Analyze this study according to Guideline 8. EXAMPLE 9 Extraordinary Claims