1. The Power of Comparison Groups Rita O’Sullivan Evaluation, Assessment, & Policy Connections School of Education, University of North Carolina at Chapel Hill (ritao@unc.edu)ritao@unc.edu NCCEP Capacity Building Workshop, Las Vegas, NV- February 2013

2. Important GEAR UP Evaluation Questions: A.Which GEAR UP activities contribute to improved achievement? B.Do students who participate in GEAR UP do better academically? A.Does GEAR UP participation influence college enrollment and retention?

3. Possible Evaluation Design Strategies: A. Which GEAR UP activities contribute to improved achievement?  Increase in Percentage/Number of students participating in GEAR UP Activities  Improved achievement from last year  Growth in Pre-test/Post-results

4. Possible Evaluation Design Strategies: B. Do students who participate in GEAR UP do better academically?  Improved achievement from last year  Growth in Pre-test/Post-results  Comparison of GEAR UP students with non- GEAR UP students in a school

5. Possible Evaluation Design Strategies: C. Does GEAR UP participation influence college enrollment and retention?  Comparison of GEAR UP students with previous cohort in a school  Comparison of GEAR UP students with school district average  Comparison of GEAR UP schools with non-GEAR UP schools within a district

6. Possible Alternative Explanations for Evaluation Findings (i.e., Threats to Internal Validity) Evaluation findings vary in how convincing they are due to design strategies used with accompanying possible alternative explanations for results: History Pre-Post Testing Maturation (physical/emotional factors) Sampling Instrumentation Attrition Statistical Regression Interactions

7. Key Threats for the Designs: Which GEAR UP activities contribute to improved achievement? Do students who participate in GEAR UP do better academically? Does GEAR UP participation influence college enrollment and retention?

8. EQUIVALENT Comparison Designs: EQUIVALENT Groups directly address Sampling threats and indirectly: History (i.e. services comparison group students received) can still be a problem  Pre-Post Testing  Maturation (physical/emotional factors)  Instrumentation  Attrition  Statistical Regression  Interactions

9. Three Credible Comparison Group Design Options: Cohort Analysis Multiple Variable Matching Propensity Score Matching

10. Cohort Analysis Definition: Comparing one GEAR UP cohort with a previous cohort within the same school that did not experience GEAR UP Strengths:  Controls for school differences Cautions:  One cohort is not necessarily the same as another  One year is not necessarily the same as the next

11. Multiple Variable Matching Definition: Creating a “comparable” comparison group based on a variety of relevant variables (e.g., previous achievement, ethnicity, SES, etc.) by setting selection criteria Strengths:  Matching on multiple variables will provide relatively more equivalent groups for comparison Challenges:  Need matching data on comparison group  Don’t necessarily know what services selected students have received.

12. Propensity Score Matching Definition: Statistically creating a “comparable” comparison group by generating a propensity using a variety of relevant variables (e.g., previous achievement, ethnicity, SES, etc.) Strengths:  Large number of matching variables make the identification of an “equivalent” group more possible Cautions:  Need matching data on comparison group  Don’t necessarily know what services selected students have received.

13. Thank you! Look for a Propensity Score Workshop at NCCEP at July annual meeting in San Francisco