# Methods for Estimating Reliability

## Presentation on theme: "Methods for Estimating Reliability"— Presentation transcript:

Methods for Estimating Reliability
Dr. Shahram Yazdani

Types of Reliability Inter-Rater or Inter-Observer Reliability:
Used to assess the degree to which different raters or observers give consistent estimates of the same phenomenon Test-Retest Reliability: Used to assess the consistency of a measure from one time to another Parallel-Forms Reliability: Used to assess the consistency of the results of two tests constructed same way from the same content domain Internal Consistency Reliability: Used to assess the consistency of results across items within a test Dr. Shahram Yazdani

Interrater or Interobserver Reliability
object or phenomenon observer 1 observer 2 = ? Dr. Shahram Yazdani

Inter-rater Reliability
Statistics used Nominal/categorical data Kappa statistic Ordinal data Kendall’s tau to see if pairs of ranks for each of several individuals are related Two judges rate 20 elementary school children on an index of hyperactivity and rank order them Interval or ratio data Pearson r using data obtained from the hyperactivity index Dr. Shahram Yazdani

Test-Retest Reliability
= Stability over Time test time 1 time 2 Dr. Shahram Yazdani

Test-Retest Reliability
Statistics used Pearson r or Spearman rho Important caveat Correlation decreases over time because error variance INCREASES (and may change in nature) Closer in time the two scores were obtained, the more the factors which contribute to error variance are the same Dr. Shahram Yazdani

Parallel-Forms Reliability
form B form A Stability Across Forms = time 1 time 2 Dr. Shahram Yazdani

Parallel Forms Reliability
Statistic used Pearson r or Spearman rho Important caveat Even when randomly chosen, the two forms may not be truly parallel Dr. Shahram Yazdani

Internal consistency Internal consistency
Average inter-item correlation Average item total correlation Split-half reliability Dr. Shahram Yazdani

Average Inter-item Correlation
Definition: calculate correlation of each item (Pearson r) with all other items. Dr. Shahram Yazdani

Internal Consistency Reliability
item 1 Average Inter-Item Correlation item 2 I1 I2 I3 I4 I5 I6 I1 I2 I3 I4 I5 I6 1.00 item 3 test item 4 item 5 .90 item 6 Dr. Shahram Yazdani

Average Item Total Correlation
Definition: calculate correlation of each item scores with total score. Dr. Shahram Yazdani

Internal Consistency Reliability
Average Item-Total Correlation item 1 I1 I2 I3 I4 I5 I6 item 2 I1 I2 I3 I4 I5 I6 Total 1.00 item 3 test item 4 item 5 item 6 .85 Dr. Shahram Yazdani

Split-half Reliability
Definition: Randomly divide the test into two forms; calculate scores for Form A, B; calculate Pearson r as index of reliability Dr. Shahram Yazdani

Internal Consistency Reliability
Split-Half Correlations item 1 item 1 item 3 item 4 item 2 .87 item 3 test item 4 item 5 item 2 item 5 item 6 item 6 Dr. Shahram Yazdani

Cronbach’s alpha & Kuder-Richardson-20
Measures the extent to which items on a test are homogeneous; mean of all possible split-half combinations Kuder-Richardson-20 (KR-20): for dichotomous data Cronbach’s alpha: for non-dichotomous data Dr. Shahram Yazdani

Internal Consistency Reliability
Cronbach’s alpha () item 1 .87 item 1 item 3 item 4 item 2 item 5 item 6 .85 item 1 item 3 item 4 item 2 item 5 item 6 .91 item 1 item 3 item 4 item 2 item 5 item 6 item 2 item 3 test SH1 .87 SH2 .85 SH3 .91 SH4 .83 SH5 .86 ... SHn .85 item 4 item 5 item 6  = .85 Dr. Shahram Yazdani

Reducing Measurement Error
pilot test your instruments -- get feedback from respondents train your interviewers or observers make observation/measurement as unobtrusive as possible double-check your data triangulate across several measures that might have different biases Dr. Shahram Yazdani

Validity vs Reliability
Dr. Shahram Yazdani

Thank you ! Any Question ? Dr. Shahram Yazdani