Can you hear me now? Keeping threats to validity from muffling assessment messages Maureen Donohue-Smith, Ph.D., RN Elmira College

Plan for Discussion:  Review common “threats to internal validity”  Identify how these threats might affect assessment at participants’ own institutions  Identify assessment strategies that minimize the impact of these threats on assessment

Internal Validity Your ability to say that your program produced the changes that you observed Your ability to say that your program produced the changes that you observed Infers a causal relationship between the program or intervention and the outcome that was NOT caused by other alternative factors Infers a causal relationship between the program or intervention and the outcome that was NOT caused by other alternative factors

Educational Assessment Program Outcomes

Threats to Internal Validity What are some of the common alternative explanations that obscure the real relationship between your educational program and students’ learning outcomes?

History: Specific events occur between the first Specific events occur between the first and second measurement of program and second measurement of program outcomes that could have independent outcomes that could have independent effects on your program outcomes effects on your program outcomes

Maturation Changes in program participants that result from individual development and experience over time and not from the program intervention Changes in program participants that result from individual development and experience over time and not from the program intervention

Testing Participation in the first assessment Participation in the first assessment affects performance during later affects performance during later assessments (also known as assessments (also known as sensitization or “practice effects”) sensitization or “practice effects”)

Instrumentation Measurement tool is inaccurate Measurement tool is inaccurate Wrong instrument selected for the construct you want to measure Wrong instrument selected for the construct you want to measure Pretest and posttest assess different domains Pretest and posttest assess different domains Questionnaire not standardized for reliability Questionnaire not standardized for reliability Inconsistency in data collection procedures or coding Inconsistency in data collection procedures or coding

Statistical regression: (regression to the mean) Extreme scores on the first test tend to move toward the center or average on the second test Extreme scores on the first test tend to move toward the center or average on the second test

Selection Bias Students participating in assessment are not representative of the overall student population Students participating in assessment are not representative of the overall student population

Program Mortality: Program Mortality: Program participants drop out differentially (e.g., all the men, all the women, all the most severely depressed, etc.) Program participants drop out differentially (e.g., all the men, all the women, all the most severely depressed, etc.)

Reactive Effects: Participants change their behavior as a result of being assessed (Hawthorne Effect), not as result of your program Participants change their behavior as a result of being assessed (Hawthorne Effect), not as result of your program

Statistical Conclusion Validity Problems with the data produce invalid conclusions. Problems with the data produce invalid conclusions. - Too few subjects - Too few subjects - Wrong statistical tests used to analyze data - Wrong statistical tests used to analyze data - Violates rules of statistical analysis - Violates rules of statistical analysis

Social Interaction Effects Diffusion or Imitation of Treatment Diffusion or Imitation of Treatment Compensatory Rivalry: “John Henry Effect” Compensatory Rivalry: “John Henry Effect” Resentful Demoralization: “Screw You Effect” Resentful Demoralization: “Screw You Effect” Compensatory Equalization of Treatment Compensatory Equalization of Treatment

Solutions?