Reliability and Validity Chapter 5 Reliability and Validity PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Overview Measuring Variables Choosing a Behavior to Measure Overview of Types of Measurement Errors Bias Random error Reliability Validity Manipulating Variables Validity Threats to Establishing Types of manipulations PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Two Types of Measurement Error Bias Random error PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Three “Places” Measurement Error Can Occur Observer/Scorer Participant Person administering the measure PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Two Types of Observer Error Observer bias (Scorer bias) Random observer error PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Minimizing Observer Errors Why it is more important to reduce observer bias than random error Techniques for reducing observer bias* PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Techniques for Reducing Observer Bias Eliminating human observer errors by eliminating the human observer Limiting human observer errors by limiting the human observer’s role Reducing observer bias by making observers “blind” Conclusions about reducing observer bias PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Reducing Random Observer Error Most of the techniques that reduce observer bias reduce random observer error PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Errors in Administering the Measure Types Experimenter (researcher) bias Random error Solutions Blind technique to reduce bias Standardization to reduce both bias and random error PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Errors Due to the Participant Bias due to the participant (Subject bias) Random error due to the participant PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Subject (Participant) Bias Obeying demand characteristics Social desirability bias PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Conclusions about Reducing Subject Biases Blind techniques can reduce demand characteristics Making participants anonymous can reduce social desirability bias PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Summary of Types of Measurement Error Try to reduce all forms of measurement error Really focus on reducing bias PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Reliability: The (Relative) Absence of Random Error The importance of being reliable: Reliability as a prerequisite to validity Using test-retest reliability to assess overall reliability: To what degree is a measure “random error free”? PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Identifying (and Then Dealing with) the Main Source of a Measure’s Reliability Problems Are observers to blame for low test-retest reliability?: Assessing observer reliability Non-observer sources of random error Using internal consistency measures to estimate random error due to participants PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Internal Consistency: Test Questions Should Agree with Each Other Random error due to participants may cause low internal consistency PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Two Solutions to Problems Caused by Random Participant Error Add questions to let random participant error balance out Ask better questions to reduce random participant error PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Measuring Internal Consistency Average inter-item correlations as indexes of internal consistency Split-half coefficients as indexes of internal consistency Additional indexes of internal consistency Conclusions about internal consistency’s relationship to reliability PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Conclusions About Reliability Reliability is a prerequisite for validity If test-retest reliability is low, try to find out where reliability problem is and fix it. Reliability does not guarantee validity PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Beyond Reliability: Establishing Construct Validity Content Validity Internal Consistency Convergent Validity: Getting evidence that you are measuring the right construct Discriminant Validity: Showing that you are not measuring the wrong construct PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Manipulating Variables Common threats to a manipulation’s validity Evidence used to argue for a manipulation’s construct validity Tradeoffs among three common types of manipulations Conclusions PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Common Threats to a Manipulation’s Validity Random error Experimenter bias Subject biases PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Evidence Used to Argue for a Manipulation’s Construct Validity Consistency with theory Manipulation checks PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Tradeoffs Among Three Common Types of Manipulations Instructional manipulations Environmental manipulations Manipulations involving stooges PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

Concluding Remarks Operational definitions should Be consistent with dictionary/theory definitions Be standardized to reduce bias and random error Have evidence to support their validity PowerPoint presentation to accompany Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley