Reliability: Introduction

Reliability Session Definitions & Basic Concepts of Reliability Theoretical Approaches Empirical Assessments of Reliability Interpreting Coefficients Test Refinement and Reliability

Conceptions of Reliability “He’s often late!” “My car won’t start!”   S.E.M.

Components of Measurement Measured Value = True Value + Systematic Error (Bias) + Random Error The usefulness of a measure depends on the ratio of the true value to any error variance that it produces

Classical Test Theory Random error vs. systematic, or bias Classical theory assumptions: –Error independent of score –Mean of errors = 0 –Observed score = true score + error –Random errors tend to cancel out if sufficient observations made

Internal Consistency Logic: problem of change over time Alternate forms Split-half Kuder-Richardson & alpha Internal consistency Item-total correlations Number of items & reduction in error term Spearman-Brown formula # items reliability

Sources of Variance: Which to Include in Estimating Reliability? Error Patients Observers Time Measurement instrument

Reliability Subject Variability Subject variability + Measurement Error Reliability = Subject Variability Subject Var. + Observer Variability + Meas’t Error or,

Generalizability Theory Separates sources of variability: –Observer inconsistency –Between observers –Subject change over time Quantifies these Helps to show how to optimize design (and administration) of test given these performance characteristics.

Reliability versus Sensitivity of a Measurement Metaphor of the combs

Statistics to use: ICC vs. Pearson r ICC = 1.0; r = 1.0 r = 1.0; ICC < 1.0

What is the Reliability when: Every student is rated “above average” Physician A rates every BP as 5 mm Hg higher than physician B The measure is applied to a different population The observers change The patients do, in reality, improve over time