1. Chapter 8 Data Evaluation and Interpretation
This chapter continues the emphasis on critical thinking: in evaluating sources and evidence
and in interpreting the material.
Students need reminding that (1) we do research not just to find answers but to find
answers that stand the best chance of being correct; and (2) as consumers, as well as producers,
of research, we need to be able to evaluate and interpret the findings accurately.
To work with their information from a critical perspective, students should be able to
evaluate the material for validity, reliability, and certainty; to recognize the influence of
bias (for example, in database sources, direct observation, or interpretation); to avoid
causal and statistical fallacies; and to reassess their research methods and reasoning
before settling on a conclusion.

2. Learning Objectives
Appreciate the role of critical thinking in evaluating research findings
Assess the dependability of information sources (print and digital)
Assess the quality of your evidence
Interpret your findings accurately and without bias
Understand that “certainty” in research is an elusive goal

3. Learning Objectives (continued)
Recognize common errors in reasoning and statistical analysis
Understand that research carries the potential for error

4. Evaluating and Interpreting
Not all information is equal. Not all interpretations are equal either:

5. Evaluate the Sources
Start the evaluating and interpreting process by evaluating your sources:
Determine the currency of the source.
Assess the reputation of a printed source.
Assess the perspective of a digital source.
Consider the possible motives of those who have funded a study.
Cross-check the source against other, similar sources.

6. Evaluate the Evidence
Then evaluate the evidence. Evidence is any finding used to support or refute a particular claim:
Determine the sufficiency of the evidence.
Differentiate hard from soft evidence. Hard evidence is facts, expert opinion, and statistics; soft evidence is uninformed opinion and speculation.
Decide whether the presentation of evidence is balanced and reasonable.
Consider how the facts are being framed.

7. Interpret Your Findings
Then try to uncover the truth of your findings:
Identify your level of certainty. Is it a conclusive answer (irrefutable), the probable answer (likely), or an inconclusive answer (uncertain)?
Examine the underlying assumptions. Assumptions are ideas we often accept without proof. Does the information you located make assumptions?
Be alert to personal bias. Does the information you’ve found seem biased?
Consider other possible interpretations. Is there another way you can look at what you’ve found?

8. Avoid Distorted or Unethical Reasoning
Next apply reason to the information you’ve found by seeing if it contains errors in reasoning, such as the following logical errors:
Faulty generalization. Generalizing on the basis of limited evidence. Ask yourself: “How much can we generalize from these findings?”
Faulty causal reasoning. Confusing or distorting why one thing caused or will cause another thing to happen. Ask yourself: “Did X possibly, probably, or definitely cause Y?”

9. Avoid Distorted or Unethical Reasoning (continued)
Faulty statistical analysis. Interpreting the numbers in a misleading way. Ask yourself: “Are these numbers really accurate?” Avoid:
* sanitized statistics (numbers that have manipulated to obscure the facts)
* meaningless statistics (quantifying things that really can’t be quantified)
* undefined averages (averages that are mathematically skewed)
* distorted percentage figures (percentages that aren’t mathematically valid or ignore the margin or error)

10. Avoid Distorted or Unethical Reasoning (continued)
* bogus rankings (items compared on the basis of ill-defined criteria)
* confusion of correlation and causation (misinterpreting coincidence as truth)
* biased meta-analysis (using only findings from other studies that support your own bias)
* fallible computer model (trusting a computer with biased assumptions programmed in)
* misleading terminology (using statistical terms that hide meaning)

11. Acknowledge the Limits of Research
Lastly, understand that there are limits to even the most careful research you use to support your own conclusions.
Not all research is valid (correct) and reliable (repeatable). Even a valid survey can lead to invalid results when respondents misunderstand survey questions, answer questions dishonestly, or respond in a way that the think they ought to. Even a reliable survey can become unreliable when the same respondents answer the same survey questions differently the second time.

12. Acknowledge the Limits of
Research (continued)
Research studies can be flawed. Epidemiological studies (the study of populations), laboratory studies (studies conducted under laboratory conditions), and human exposure studies (clinical trials) can go wrong in a variety of ways.
Research can sometimes be deceptively reported. Sometimes researchers avoid reporting results that are awkward, embarrassing, or unpopular.

13. Review Questions
1. What is the difference between evaluation and interpretation of information? 2. What are the five ways to evaluate a source? 3. What are four ways to evaluate evidence? 4. What are hard evidence and soft evidence? 5. What are the four ways of interpreting your findings? 6. What are the three levels of certainty? 7. What is a faulty generalization?
Answers
1. Evaluation is checking your sources for currency, motivation, reputation, and other factors that may influence its validity. Interpreting is uncovering the truth of your findings.
2. Determine the currency of the source, assess the reputation of a printed source, assess the perspective of a digital source, consider the possible motives of those who have funded a study, and cross-check the source against other, similar sources.
3. Determine the sufficiency of the evidence, differentiate hard from soft evidence, decide whether the presentation of evidence is balanced and reasonable, consider how the facts are being framed.
4. Hard evidence is facts, expert opinion, and statistics; soft evidence is uninformed opinion and speculation.
5. Identify your level of certainty, examine the underlying assumptions, be alert to personal bias, consider other possible interpretations.
6. The conclusive answer (irrefutable), the probable answer (likely), and the inconclusive answer (uncertain).
7. Generalizing on the basis of limited evidence.

14. Review Questions (continued)
8. What is faulty causal reasoning? 9. What are five types of faulty statistical reasoning? 10. In what three ways can research findings be limited?
Answers (continued)
8. Confusing or distorting why one thing caused or will cause another thing to happen.
9. Any of the following: sanitized statistics, meaningless statistics, undefined averages, distorted percentage figures, bogus rankings, confusion of correlation and causation, biased meta-analysis, fallible computer model, misleading terminology.
10. Not all research is valid (correct) and reliable (repeatable), research studies can be flawed, research can sometimes be deceptively reported.