1. Reliability and Validity
Chapter 5
Reliability and Validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

2. Overview Measuring Variables Choosing a Behavior to Measure
Overview of Types of Measurement Errors
Bias
Random error
Reliability
Validity
Manipulating Variables
Validity
Threats to
Establishing
Types of manipulations
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

3. Two Types of Measurement Error
Bias
Random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

4. Three “Places” Measurement Error Can Occur
Observer/Scorer
Participant
Person administering the measure
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

5. Two Types of Observer Error
Observer bias (Scorer bias)
Random observer error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

6. Minimizing Observer Errors
Why it is more important to reduce observer bias than random error
Techniques for reducing observer bias*
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

7. Techniques for Reducing Observer Bias
Eliminating human observer errors by eliminating the human observer
Limiting human observer errors by limiting the human observer’s role
Reducing observer bias by making observers “blind”
Conclusions about reducing observer bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

8. Reducing Random Observer Error
Most of the techniques that reduce observer bias reduce random observer error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

9. Errors in Administering the Measure
Types
Experimenter (researcher) bias
Random error
Solutions
Blind technique to reduce bias
Standardization to reduce both bias and random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

10. Errors Due to the Participant
Bias due to the participant (Subject bias)
Random error due to the participant
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

11. Subject (Participant) Bias
Obeying demand characteristics
Social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

12. Conclusions about Reducing Subject Biases
Blind techniques can reduce demand characteristics
Making participants anonymous can reduce social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

13. Summary of Types of Measurement Error
Try to reduce all forms of measurement error
Really focus on reducing bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

14. Reliability: The (Relative) Absence of Random Error
The importance of being reliable: Reliability as a prerequisite to validity
Using test-retest reliability to assess overall reliability: To what degree is a measure “random error free”?
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

15. Identifying (and Then Dealing with) the Main Source of a Measure’s Reliability Problems
Are observers to blame for low test-retest reliability?: Assessing observer reliability
Non-observer sources of random error
Using internal consistency measures to estimate random error due to participants
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

16. Internal Consistency: Test Questions Should Agree with Each Other
Random error due to participants may cause low internal consistency
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

17. Two Solutions to Problems Caused by Random Participant Error
Add questions to let random participant error balance out
Ask better questions to reduce random participant error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

18. Measuring Internal Consistency
Average inter-item correlations as indexes of internal consistency
Split-half coefficients as indexes of internal consistency
Additional indexes of internal consistency
Conclusions about internal consistency’s relationship to reliability
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

19. Conclusions About Reliability
Reliability is a prerequisite for validity
If test-retest reliability is low, try to find out where reliability problem is and fix it.
Reliability does not guarantee validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

20. Beyond Reliability: Establishing Construct Validity
Content Validity
Internal Consistency
Convergent Validity: Getting evidence that you are measuring the right construct
Discriminant Validity: Showing that you are not measuring the wrong construct
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

21. Manipulating Variables
Common threats to a manipulation’s validity
Evidence used to argue for a manipulation’s construct validity
Tradeoffs among three common types of manipulations
Conclusions
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

22. Common Threats to a Manipulation’s Validity
Random error
Experimenter bias
Subject biases
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

23. Evidence Used to Argue for a Manipulation’s Construct Validity
Consistency with theory
Manipulation checks
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

24. Tradeoffs Among Three Common Types of Manipulations
Instructional manipulations
Environmental manipulations
Manipulations involving stooges
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

25. Concluding Remarks Operational definitions should
Be consistent with dictionary/theory definitions
Be standardized to reduce bias and random error
Have evidence to support their validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley