PSY 614 Instructor: Emily Bullock, Ph.D. Reliability PSY 614 Instructor: Emily Bullock, Ph.D.

Reliability Outline Sources of Error Types of Reliability Factors effecting reliability Nature of the Test

Sources of Error Observed Score = True Score + Error Test Construction-item or content sampling Test Administration-test environment, test taker variables, examiner related variables Test Scoring and Interpretation

Types of Reliability Test Retest Parallel-Forms and Alternate-Forms Internal Consistency Inter-Rater

Types of Reliability- Test Retest Test-Retest Reliability: estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test Only appropriately used to measure reliability of stable constructs (ex. personality) Consider intervening variables due to time Interval time must be reported with test-retest

Types of Reliability- Parallel Forms and Alternate Forms True Parallel form: indicates that each form of the test produces equivalent means and variances of test scores Alternate forms: does not meet the above standard but attempts at two versions of a test that have been constructed to be equivalent with respect to content and difficulty Often termed the coefficient of equivalence

Types of Reliability- Internal Consistency Split-Half Reliability: obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once Steps of Split-Half Administer the entire test to participants Divide the test item scores into equivalent halves Compute a correlation (Pearson’s r) between the two halves Adjust the half-test reliability using Spearman-Brown formula How do you split a test?

Types of Reliability- Internal Consistency Inter-item consistency: degree of correlation among all the items on a scale Only appropriate for homogeneous scales or tests

Types of Reliability- Inter-Rater Inter-Rater or Inter-scorer reliability: degree of agreement or consistency that exists between two or more scorers

Purpose of the Reliability Coefficient Types of Reliability # of Test Sessions # of Test Forms Sources of Error Variance Stat. Procedure Test-retest 2 1 Administration Pearson’s r Alternate forms 1 or 2 Test construction & Administration Internal consistency Test Construction Pearson’s r with various corrections Inter-rater Scoring and Interpretation r, or kappa coefficient -

Factors Effecting Reliability Some legitimate factors contribute to variability of test scores (see Murphy & Davidshofer p. 118) Characteristics of the people taking the test What is your group of test takers like? Characteristics of the test itself Is the test worth making longer? The intended uses of the test scores Are your differences in test takers big or small? The method used to estimate reliability How long do you plan to make decisions based on this test?

What is good reliability? Commonly used standard: .75-.80 standard for decision-making tests .70 standard for research instruments Charter (2003) Article: Less than .70 unacceptable .70-.80 fair .80 to .90 good Greater than .90 excellent Text: .80 high reliability .70 moderate reliability .60 unacceptable reliability *This information is most applicable to internal consistency reliability

The Nature of the Test Homogeneous or Heterogeneous test items Characteristics, ability, or trait being measured is presume to be dynamic or static Range of test scores is or is not restricted Test is a speed or a power test Test is or is not criterion-referenced

Using Reliability and Error in Interpretation Standard Error of Measurement an index of the extent to which one individual’s scores vary over tests presumed to be parallel Indicates how much variability can be expected in test scores as a result of measurement error Can be used to form confidence intervals or bandwidths around scores