1. Reliability and Validity in Experimental Research ♣
Chapter 7
Back to Brief Contents
Introduction 
Relationship between Reliability and Validity 
Experimental Reliability 
Reliability and the Independent Variable 
Reliability and the Dependent Variable 
Measurement Error 
Methods of Assessing Reliability 
Experimental Validity 
Statistic Conclusion Validity 
Internal Validity 

2. 7.0 Introduction -1 Reliability Experimental reliability
Back to Chapter Contents
Reliability
Consistency, stability, or repeatability
Experimental reliability
Consistency, stability, or repeatability of the results of an experimental study

3. 7.0 Introduction -2(end) Validity Experimental validity
Back to Chapter Contents
Validity
The correctness or truthfulness of an inference
Experimental validity
The correctness or truthfulness of an inference that is made from the results of an experiment

4. 7.1 Relationship between Reliability and Validity
Back to Chapter Contents
Experimental reliability can exist without experimental validity
Experimental validity cannot exist without experimental reliability
(e.g.) speedometer, tires

5. 7.2 Experimental Reliability -1
Back to Chapter Contents
Number of research participants
More Participants  more reliable
Large N: less likely to be influenced by random or chance factors
Reliability of identifying the type of participant needed for the study
Indicators used select P with the same characteristics
(e.g.) Beck Depression Inventory

6. 7.2 Experimental Reliability -2
Back to Chapter Contents
Characteristics of the researchers
(e.g.) sleep deprivation on heart rate: male P, male vs. attractive blond
Maximize reliability: the same researcher collects data from all P
Replication: researcher with the same characteristics should be employed

7. 7.2 Experimental Reliability -3(end)
Back to Chapter Contents
Ability of the researcher
Use the same procedure
Well trained  good recording
(e.g.) EEG

8. 7.3 Reliability and the Independent Variable
Back to Chapter Contents
Consistency: a psychological construct is created in the P
Concrete operations must produce the same effect in P for reliability to exist
(e.g.) Velten (1968): mood state
read a set of statements and to feel and place themselves in that situation

9. 7.4 Reliability and the Dependent Variable
Back to Chapter Contents
The effect measured should be about the same regardless of when it is measured

10. 7.5 Measurement Error -1
Back to Chapter Contents
Definition — inaccuracy that exists when measuring a psychological characteristic or behavior
Factors causing measurement error — P respond differently on different occasions
(e.g.) fatigue, emotionally upset
Relationship between measurement error and reliability
Measurement error   Reliability 

11. 7.5 Measurement Error -2(end)
Back to Chapter Contents
Measurement = true score + error score
Most experiments: single-occasion events
Epstein (1981): low reliability
Averaged over several days
Epstein (1979) Fig 7.1
Repeat and Average (but most exp. cannot)
Replication of the results of single-occasion studies

12. 7.6 Methods of Assessing Reliability
Back to Chapter Contents
Successive measurement
Test-retest, Equivalent forms
Simultaneous measurement
Interrater reliability: Consistency of ratings made by the different observers (continuous v.)
Interobserver agreement: Percentage of times different raters agree (discrete v.)
Internal consistency
Split-half, Cronbach’s alpha

13. 7.7 Experimental Validity
Back to Chapter Contents
Definition—accuracy or correctness of an inference
Types of validity that maximize obtaining experimental validity (Shadish et. Al., 2002) Tab 7.1
Statistical conclusion validity
Internal validity
Construct validity
External validity

14. 7.8 Statistical Conclusion Validity
Back to Chapter Contents
Refers to the accuracy of an inference that the IV and DV covary
Threats
insufficient research participants -- power
Many others but beyond this course to discuss

15. 7.9 Internal Validity -1
Back to Chapter Contents
Refers to the accuracy of the inference that the IV caused the effect observed in the DV
Primary threat — confounding extraneous variables (EV)
Confounding: systematically vary with IV
EV: variable other than IV that influences DV

16. 7.9 Internal Validity -2 Threats to Internal Validity
Back to Chapter Contents
Threats to Internal Validity
Shadish, Cook & Campbell (2002)
History
Maturation
Instrumentation
Testing
Regression Artifact
Attrition
Selection
Additive and Interactive Situation

17. 7.9 Internal Validity -3 Threats to Internal Validity History Fig 7.2
Back to Chapter Contents
Threats to Internal Validity
History Fig 7.2
Any event occurring after the experimental treatment is introduced that could produce the observed outcome
Maturation
Changes in biological and psychological conditions that occur with the passage of time
Instrumentation
Changes in the assessment of the DV
— human observers: more than one (check)
training program

18. 7.9 Internal Validity -4 Threats to Internal Validity Testing
Back to Chapter Contents
Threats to Internal Validity
Testing
Changes in a person’s score on the second administration of a test as a result of previously having taken the test
Regression Artifact Tab 7.2 Fig 7.3
Attrition 耗損
Some people do not show up for the study or do not complete it

19. 7.9 Internal Validity -5(end)
Back to Chapter Contents
Threats to Internal Validity
Selection
The choice of P for the various treatment groups is based on different criteria
(e.g.) type of psychosis
Additive and Interactive Situation
The combined effect of several threats to internal validity
— selection-maturation effect Fig 7.4
— selection-history effect
— selection-instrumentation effect