Validity and Item Analysis Chapter 4

Validity Concerns what the instrument measures and how well it does that task Not something an instrument has or does not have Informs counselor when it is appropriate to use the instrument and what can be inferred from the results Reliability is a prerequisite for validity

Movement Away from Traditional Categories of Validity Traditional types of validity Content-Related Criterion-Related Construct-Related

Evidence Based on Instrument Content The degree to which the evidence indicates the items, questions, or tasks adequately represent the intended behavior domain. Central focus is typically on how the instrument’s content was determined Content-related validation evidence should not be confused with face validity

Evidence Based on the Response Processes Information processing Examining the response processes used by different groups Computer simulated performances

Evidence Based on Internal Structure Examining internal structure using factor analysis Examining the internal structure of an assessment with groups (e.g., gender, race or ethnicity)

Evidence Based on Relations to Other Variables Correlational method Convergent evidence/Discriminant evidence Multitrait-multimethod matrix Prediction or instrument-criterion relationship Concurrent and predictive validity Regression Decision Theory Validity Generalization

Correlational Method Correlational method- examine relationship between instrument and other pertinent variables or a criterion Validity coefficient(s)

Prediction or instrument- criterion relationship Concurrent validity Predictive validity

Regression … … ….. ….. …......

Regression Line … … ….. ….. … a = intercept b = slope X = score

Standard Error of Estimate The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity.

Decision Theory: Expectancy Table ABC Scale Ratings

Expectancy Table (cont.)

Validity Generalization A method of combining validation studies to determine if the validity evidence can be generalized Must be a substantial number of studies Meta-analysis

Evidence-based consequences of testing Social consequences of racial/ethnic group differences Consider social implications when examining the validation evidence

Conclusion on validation evidence Gradual accumulation of evidence Counselor must evaluate the information to determine if it appropriate for which client under what circumstance Validation evidence should also be considered in informal assessments

Item Analysis Focus is on examining and evaluating each item on an instrument. Item difficulty: p = # who answered correctly total number Item discrimination: d = upper % - lower % (or correlational method)

Item Response Theory or Latent Trait Theory Focus is on each item and on establishing items that measure the individual’s ability or level of a latent trait. Involves examining the item characteristic function (or characteristic curve) and the calibration of each individual item. Parameters: Difficulty Slope Basal level

Item Characteristic Curve Grade %