Assessment Procedures for Counselors and Helping Professionals, 7e © 2010 Pearson Education, Inc. All rights reserved. Chapter 5 Reliability

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-2 Reliability refers to the degree to which test scores are dependable, consistent, and stable upon additional testings. Multiple factors can impact reliability. What is Reliability?

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-3 Measurement error is any fluctuation in scores that results from factors related to the measurement process that are irrelevant to what is being measured. Error may also be understood as the difference between an individual’s observed score and his or her true score. Measurement Error

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-4 Observed score – the actual score on a test. True score – the true reflection of one’s ability, skills or knowledge. Observed score = True Score + Error

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-5 Time Sampling Error Content Sampling Error Interrater Differences Quality of Test Items Other Sources of Error Test Length Task Taker Variables Test Administration Sources of Measurement Error

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-6 The most common method of estimating reliability involves correlating two sets of test scores to obtain a reliability coefficient. The reliability coefficient is the ratio of the true score variance to the observed score variance. Methods of Estimating Reliability

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-7 Reliability coefficients relate strictly to groups of test scores and cannot be applied to individual test scores. Reliability coefficients measure reliability on a scale of 0 to 100. Error = 1 - r

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-8 Test-retest Alternate Forms Internal Consistency Split-half KR formula and Coefficient Alpha Interrater Methods of Measuring Reliability

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved. 5-9 Test-Retest

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved Alternate Forms

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved Internal Consistency: Split-Half

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved KR Formulas and Coefficient Alpha

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved Interrater

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved What are the potential sources of error? Does the test measure one, or more than one, construct? Choosing and Estimate of Reliability

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved The level of acceptable reliability depends on: the construct being measured the way the test scores will be used the method used for estimating reliability Level of Acceptable Reliability

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved The Standard Error of Measurement measures the spread of scores for a single individual across multiple tests. The SEM is not a measure of reliability, but can be used to create confidence intervals around specific observed scores. Confidence intervals establish the upper and lower limit in which a test takers true score falls. Standard Error of Measurement

Assessment Procedures for Counselors and Helping Professionals, 7e Drummond/Jones © 2010 Pearson Education, Inc. All rights reserved Increase the number of test items Provide clearly stated administration and scoring procedures Require training before individuals can administer, grade, and interpret a test Increasing Reliability