Measuring Research Variables chapter 11 Measuring Research Variables

Chapter Outline Validity Reliability Methods of establishing reliability Intertester reliability (objectivity) Standard error of measurement Using standard scores to compare performance (continued)

Chapter Outline (continued) Measuring movement Measuring written responses Measuring affective behavior Scales for measuring affective behavior Measuring knowledge Item response theory

Four Basic Types of Measurement Validity The American Educational Research Association and American Psychological Association agree on the definition of four types of validity: Logical validity Content validity Criterion validity Concurrent Predictive Construct validity

Desired Qualities in a Criterion Measure Relevance (e.g., the extent to which the criterion exemplifies success) Freedom from bias (Everyone must have same chance to achieve a good score.) Reliability of criterion (You can’t predict a criterion if you can’t measure it.) Availability (how hard is it to obtain criterion score)

Measurement Reliability Overview Definition Observed score = True score + Error score Sources of measurement error Expressing reliability through correlation Interclass Intraclass

Estimating Reliability Interclass Simple correlation Weaknesses Intraclass: ANOVA with repeated measures Treating trial-to-trial variation as measurement error Discarding trials Ignoring trial-to-trial variation

Trial-to-Trial Variation Can be treated as measurement error R = (MSS – MSE)/MSS Can be solved by discarding trials, then using above formula Can be ignored R = (MSS – MSres)/MSS

Methods of Establishing Reliability Determining stability: test, retest, then use intraclass method Constructing alternate forms Obtaining internal consistency Same-day test-retest Split-half technique Kuder-Richardson (KR-20 and KR-21) Coefficient alpha

Intertester Reliability (Objectivity) Agreement among testers or raters

Other Measurement Issues Standard error of measurement Standard scores z scores: z = (X – M)/s T scale: T = 50 + 10z S y · x = s 1 . – r

Measuring Various Types of Characteristics Movement Affective behavior Attitudes Personality Scales for affective behavior Likert-type Semantic differential

Rating Scales Types of rating scales Rating errors Numerical Checklist Forced choice Rankings Rating errors Leniency Central tendency Halo Proximity errors Observer bias Observer expectation

Measuring Knowledge Analyzing test items Types of knowledge test items Item difficulty = (# correct)/total Item discrimination = (nH – nL)/n Types of knowledge test items Multiple choice True/false Completion Matching Essay Item response theory (IRT)