Presentation on theme: "CHAPTER 3 Three Claims, Four Validities:"— Presentation transcript:
1 CHAPTER 3 Three Claims, Four Validities: Interrogation Tools for Consumers of ResearchChapter 3 is the guiding frame for the textbook.In my class, we spend three 50-minute class periods on this chapter. The first day we focus on identifying claims. The next two days, students practice asking questions about the four big validities and applying the three causal criteria to different examples.
2 Chapter 3 Detailed Learning Objectives 1. Identify variables and distinguish a variable from its levels (or values).2. Discriminate between measured and manipulated variables.Describe a variable both as a conceptual variable and as an operational definition.Indicate how many variables frequency, association, and causal claims typically involve.5. Describe positive, negative, and zero associations.6. Identify verbs that signal causal claims versus association claims.7. Apply the three criteria that are used to evaluate a causal claim: covariance, temporal precedence, and internal validity.8. Identify examples in which writers’ and researchers’ claims are not justified by the studies they are describing.9. Appreciate that few studies can achieve all four kinds of validity at once, so researchers must prioritize some validities over others.
3 Reading Quiz Reading Quiz 1. Three necessary criteria for causal claims areA. covariance, temporal precedence, and internal validity.B. association, construct validity, and generalizability.C. operationalization, temporal precedence, and construct validity.2. An association claim hasA. one variable. B. two manipulated variables. C. two or more measured variables.3. Frequency claimsA. are also known as anecdotal claims.B. can describe a particular rate or level in something.C. need to have internal validity.D. All of the above.4. According to the textbook, the conclusion that family meals prevent eating disorders cannot be supported becauseA. the study does not establish temporal precedence or internal validity. B. there is zero association between the variables. C. they covary positively. D. they manipulate too many variables.Reading QuizI expect students to read and study the textbook before coming to class so we can use our class time working through examples and doing active learning exercises rather than lecturing. To encourage students to do the reading in advance, I present them with these reading quizzes on the first day we cover a chapter. Students can write their answers on 3" x 5" note cards, or you could use clickers (personal response systems). The questions are meant to be simple enough for students to answer if they have read and understood the book. After students turn in their answers, we discuss the correct answers right away for immediate feedback. In my course, these quizzes and homework combined count for 10% of the total grade—they are low stakes.If you post slides on the course management system, please delete the quiz slide so that the questions do not circulate.Although these are intended to be “check your reading” quizzes, they might also be used to support “just in time teaching.” If most students miss a question, I’ll know I need to explain it well during class time. If most students get a difficult question right, then I may decide to go more quickly through that material.Key for this quiz:ACB
5 We’ll practice this shortly! VariablesMeasured versus manipulated variablesFrom conceptual variable to operational definitionWe’ll practice this shortly!Headings for this sectionI teach the material in this section through an application exercise that comes later, in which students identify claims and name the variables in each claim. They then can indicate whether each variable is manipulated or measured, and state it at a conceptual level and an operational level.
10 Practice Identifying Claims Worry may make women’s brains work overtime.High “normal” blood sugar may still harm brain.Want a higher GPA? Go to a private college.Those with ADHD do one month’s less work a year.When moms criticize, dads back off baby care.Report: 16% of teens have considered suicide.MMR shot does not cause autism, large study says.Breastfeeding may boost children’s IQ.Breastfeeding rates hit new high in United States.Smiling may lower your heart rate.OMG! Texting and IM-ing doesn’t affect spelling!Facebook users get worse grades in college.Mother’s heartburn means a hairy newborn.This is Learning Actively A, from the IIG.Students have read the book, so we practice identifying these claims in class in small groups. Working together, students answer the questions on the next slide. There is a table in the IIG they can also use.
11 Practice Identifying Claims: Continued Indicate if the claim is frequency, association, or cause.For each claim, identify the variable(s).For each variable, is it manipulated or measured?State each variable at the conceptual level.State each variable in terms of its operational definition: How might it have been operationalized?This is Learning Actively A, from the IIGStudents have read the book, so we practice identifying these claims in class in small groups. Working together, students answer these questions for the previous slide. There is a table in the IIG they can also use.
12 Association Claims: Types of Associations Draw a scatterplot for each of the following claims:Facebook users get worse grades in college.OMG! Texting and IM-ing doesn’t affect spelling!Mother’s heartburn means a hairy newborn.What kind of association is the one you drew?(positive, negative, zero?)After students have practiced identifying claims and variables, we can get more specific about association claims.Association claims come in three types; here students practice visually representing different kinds of association claims.The heartburn—hairy newborn headline comes from:
13 Interrogating the Three Claims Using the Four Big Validities
14 ReviewReview:VariablesThree claimsTypes of associations
15 Interrogating the Three Claims Using the Four Big Validities Interrogating frequency claimsInterrogating association claimsInterrogating causal claimsHeadings for this section
16 This reviews the four validities and their definitions.
17 Simple Explanation of the Four Big Validities Construct validity:Quality of the measures and manipulationsStatistical validity:Statistical conclusions are appropriate and reasonable.Internal validity:No alternative causal explanations for the outcomeExternal validity:To whom, what, or wherecan we generalize?Which of these is the most important?That depends on the claim.
19 Interrogating Frequency Claims Construct validity of the variableHow well was the variable measured?External validity is essential!Can we generalize from the sample to the population?Statistical validityHow large is the margin of error?
20 Interrogating Association Claims Construct validity of each variableHow well was each variable measured?Statistical validityHow strong is the association? Is it statistically significant?External validityTo whom or what can we generalize the association? May be less important to the researcher.
21 Interrogating Causal Claims Construct validity of the two variablesHow well was the independent variable manipulated?How well was the dependent variable measured?Statistical validityHow big is the difference? Is it statistically significant?External validityTo whom or what can we generalize this effect?External validity is rarely prioritized in an experimentInternal validityThis is the priority!
22 Three Criteria for Causation CovarianceTemporal precedenceInternal validityIt takes lots of practice for students to learn to use the three causal rules.In addition to reviewing examples in the text, I also worked through an example from the New York Times.It was a simple study testing the effectiveness of using chamomile tea to soothe colic in infants. As an experiment, it meets all three causal rules: Covariance (there was a difference between the chamomile tea group and the placebo tea group), temporal precedence (the tea came before the rating of colic), and internal validity (there was random assignment, and it was a double-blind placebo control).This is a short article that you could paste onto a slide.
23 Two Activities for Students on Interrogating Causal Claims Does recalling the Ten Commandments make you less likely to cheat in the matrix task? “We took a group of 450 participants, split them into two groups, and set them loose on our usual matrix task. We asked half of them to recall the Ten Commandments and the other half to recall 10 books that they had read in high school. Among the group who recalled the 10 books, we saw the typical widespread but moderate cheating. But in the group that was asked to recall the Ten Commandments, we observed no cheating whatsoever.”2. A recent study showed that“Corporal punishment was associated with increased odds of anxiety and mood disorders, including major depression, panic disorder, post-traumatic stress disorder, agoraphobia, and social phobia. Several personality disorders and alcohol and drug abuse were also linked to physical punishment, the researchers found.”Two Activities for Students on Interrogating Causal ClaimsHere is some more practice for students in small groups, on interrogating causal claims applying the three criteria. The first one is a follow up on the homework assigned in Chapter 1.1. Source:2. Source:Another example: Dessert for breakfastApply the three causal rules. Can the researchers support the claim that “eating dessert at breakfast causes you to lose more weight over 32 weeks?”
24 Three Claims, Four Validities Find examples of each kind of claim in your headlines. Write down a question to ask of the study, or of the journalist, for each of the appropriate validities. If you think a validity is not relevant, explain why not.For a full-day class exercise, I bring in copies of magazines. Students locate claims of each type and write questions about each claim. Students in groups fill out a blank 3c4v matrix.This activity is described in the IIG. This is an extremely helpful application activity for students.
25 Three Claims, Four Validities Matrix FrequencyAssociationCausalVariables in the claim:Construct validity questionStatistical validity questionInternal validity questionExternal validity questionThis is the blank matrix associated with the activity that is described in the IIG (and on the previous slide).
27 Prioritizing Validities Which validity is appropriate to interrogate for every study?Which validities are not always relevant for a study?Why can’t researchers achieve all four validities in a single study?Which two validities are most often in trade-off?Which validity is most under the researcher’s control?That study’s just not valid!To summarize the chapter, we discuss the nuances of the three claims, four validities framework.Rather than describing a study as globally “valid” or “invalid,” students learn to prioritize the right validity for the kind of claim. They also learn that a study might have good construct validity but poor external validity, or vice versa. Few studies are perfect in every way, so this model helps them identify any study’s strengths and weaknesses in a more systematic fashion.
28 That is not a valid study. Say this:Not that:How’s the construct validity?The question is, is the study valid?Is external validity relevant here?That is not a valid study.Can the study supporta causal claim?Here I explicitly remind students of the kinds of things they should say when interrogating.Rather than describing a study as globally “valid” or “invalid,” students learn to prioritize the right validity for the kind of claim. They also learn that a study might have good construct validity but poor external validity, or vice versa. Few studies are perfect in every way, so this model helps them identify any study’s strengths and weaknesses in a more systematic fashion.