1. PSY 614 Instructor: Emily Bullock, Ph.D.
Reliability
PSY 614
Instructor: Emily Bullock, Ph.D.

2. Reliability Outline Sources of Error Types of Reliability
Factors effecting reliability
Nature of the Test

3. Sources of Error Observed Score = True Score + Error
Test Construction-item or content sampling
Test Administration-test environment, test taker variables, examiner related variables
Test Scoring and Interpretation

4. Types of Reliability Test Retest Parallel-Forms and Alternate-Forms
Internal Consistency
Inter-Rater

5. Types of Reliability- Test Retest
Test-Retest Reliability: estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test
Only appropriately used to measure reliability of stable constructs (ex. personality)
Consider intervening variables due to time
Interval time must be reported with test-retest

6. Types of Reliability- Parallel Forms and Alternate Forms
True Parallel form: indicates that each form of the test produces equivalent means and variances of test scores
Alternate forms: does not meet the above standard but attempts at two versions of a test that have been constructed to be equivalent with respect to content and difficulty
Often termed the coefficient of equivalence

7. Types of Reliability- Internal Consistency
Split-Half Reliability: obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once
Steps of Split-Half
Administer the entire test to participants
Divide the test item scores into equivalent halves
Compute a correlation (Pearson’s r) between the two halves
Adjust the half-test reliability using Spearman-Brown formula
How do you split a test?

8. Types of Reliability- Internal Consistency
Inter-item consistency: degree of correlation among all the items on a scale
Only appropriate for homogeneous scales or tests

9. Types of Reliability- Inter-Rater
Inter-Rater or Inter-scorer reliability: degree of agreement or consistency that exists between two or more scorers

10. Purpose of the Reliability Coefficient
Types of Reliability
# of Test Sessions
# of Test Forms
Sources of Error Variance
Stat. Procedure
Test-retest 2 1
Administration
Pearson’s r
Alternate forms
1 or 2
Test construction & Administration
Internal consistency
Test Construction
Pearson’s r with various corrections
Inter-rater
Scoring and Interpretation
r, or kappa coefficient -

11. Factors Effecting Reliability
Some legitimate factors contribute to variability of test scores (see Murphy & Davidshofer p. 118)
Characteristics of the people taking the test
What is your group of test takers like?
Characteristics of the test itself
Is the test worth making longer?
The intended uses of the test scores
Are your differences in test takers big or small?
The method used to estimate reliability
How long do you plan to make decisions based on this test?

12. What is good reliability?
Commonly used standard:
standard for decision-making tests
.70 standard for research instruments
Charter (2003) Article:
Less than .70 unacceptable
fair
.80 to .90 good
Greater than .90 excellent
Text:
.80 high reliability
.70 moderate reliability
.60 unacceptable reliability
*This information is most applicable to internal consistency reliability

13. The Nature of the Test Homogeneous or Heterogeneous test items
Characteristics, ability, or trait being measured is presume to be dynamic or static
Range of test scores is or is not restricted
Test is a speed or a power test
Test is or is not criterion-referenced

14. Using Reliability and Error in Interpretation
Standard Error of Measurement
an index of the extent to which one individual’s scores vary over tests presumed to be parallel
Indicates how much variability can be expected in test scores as a result of measurement error
Can be used to form confidence intervals or bandwidths around scores