1 The Impact of the Promise of Scholarships and Altering School Structure on College Plans, Preparation, and Enrollment The Impact of the Promise of Scholarships and Altering School Structure on College Plans, Preparation, and Enrollment By Jerald R. Herting, Charles Hirschman and Nikolas Pharris-Ciurej Department of Sociology and Center for Studies in Demography and Ecology, University of Washington University of Washington Thanks to the Andrew W. Mellon and the Bill and Melinda Gates Foundations and to the Tacoma Public Schools, their staff and students

2 Washington State Achievers Program: Increase college attendance rates through various means (e.g. scholarship programs, school redesign, and mentoring programs) at selected high schools Increase college attendance rates through various means (e.g. scholarship programs, school redesign, and mentoring programs) at selected high schools Reduce financial barriers for low income students that are motivated to attend college Reduce financial barriers for low income students that are motivated to attend college Schools targeted were historically lower achievers, lower SES population, and higher population of minority youth Schools targeted were historically lower achievers, lower SES population, and higher population of minority youth

3 Program altered Motivation Motivation Preparation Preparation Ability Ability Mechanisms Mechanisms ScholarshipsScholarships MentorMentor Restructuring schoolsRestructuring schools

4 Research Question: We are interested in assessing the effect of the WSA program on college plans, actual preparation, and college attendance.

5 Evaluating the Achievers Program: A Serendipitous “Natural Experiment” Two data sources available: 1.UW Beyond High School Project 1. Prior to Implementation: 2000 Senior Survey 2. Post Implementation: 2002 to 2005 Senior Surveys 3. 3 Program High Schools and 2 Non-program High Schools. 2.School records from large metropolitan school district from mid 1990’s to present 1. Prior: School enrollment data for the class of Post: Enrollment data for the classes of 2001 to Program Schools and 2 Non-program schools

6 The Natural Experiment: Given lack of true experimental design the natural experiment allows a pre- vs post-program evaluation Given lack of true experimental design the natural experiment allows a pre- vs post-program evaluation Our expectations are that relative to pre-program, post- program educational attitudes/behaviors should increase Our expectations are that relative to pre-program, post- program educational attitudes/behaviors should increase And that we should observe this increase in program schools but not in non-program schools And that we should observe this increase in program schools but not in non-program schools This observed change vs non-change provides evidence of a WSA effectThis observed change vs non-change provides evidence of a WSA effect This observed change should narrow the gap between the WSA schools and the non-WSA schoolsThis observed change should narrow the gap between the WSA schools and the non-WSA schools

7 The comparisons we would like to make have problems Comparisons of WSA schools to non-WSA schools Comparisons of WSA schools to non-WSA schools Schools were selected on specific criteriaSchools were selected on specific criteria Schools have different “input” populationSchools have different “input” population Direct comparisons of WSA scholarship recipients to non-recipients poses problems Direct comparisons of WSA scholarship recipients to non-recipients poses problems Without a strict experimental design comparisons are problematicWithout a strict experimental design comparisons are problematic Youth are not necessarily comparable (i.e. not all differences among them have been nullified by randomization) Youth are not necessarily comparable (i.e. not all differences among them have been nullified by randomization) Selection bias and endogeneity Selection bias and endogeneity

8 Given these limitations our best comparison is youth in the same school pre-program to youth in the same school post-program Given these limitations our best comparison is youth in the same school pre-program to youth in the same school post-program We expect change in post-program WSA schools that narrows the gap in educational oriented behaviors between the WSA and non-WSA schools We expect change in post-program WSA schools that narrows the gap in educational oriented behaviors between the WSA and non-WSA schools We might expect differences in WSA effects across settings We might expect differences in WSA effects across settings

9 Pattern of Change Expected

10 Data Sources: Focus on race/ethnic disparities in transition to college Focus on race/ethnic disparities in transition to college 2000, 2002, 2003, 2004 and 2005 Senior Surveys 2000, 2002, 2003, 2004 and 2005 Senior Surveys 5 high schools in metropolitan school district5 high schools in metropolitan school district Paper and pencil survey: modeled on NELSPaper and pencil survey: modeled on NELS About 70-80% coverage of all potential high school seniorsAbout 70-80% coverage of all potential high school seniors High school seniors – an elusive concept and population High school seniors – an elusive concept and population In school survey and multiple follow ups In school survey and multiple follow ups Only 2 percent refusal rate Only 2 percent refusal rate One year follow up survey: 90% response rateOne year follow up survey: 90% response rate 4,300 (approx) seniors in merged file: 2000, 2002 to 20054,300 (approx) seniors in merged file: 2000, 2002 to UW Beyond High School Project

11 Description of Data Sources (cont.): School district in which UWBHS survey was conducted School district in which UWBHS survey was conducted School records for each semester from 1994 to 2005 School records for each semester from 1994 to 2005 Can track individual students, courses, grades, credits earned, and school attended Can track individual students, courses, grades, credits earned, and school attended Trace the progress of 4 cohorts of 9 th graders: graduating classes of 2000 to 2005 Trace the progress of 4 cohorts of 9 th graders: graduating classes of 2000 to 2005 Roughly 9,000 students in the analysis Roughly 9,000 students in the analysis 2. School records from large metropolitan school district in Puget Sound area

12 Outcomes of Interest College Plans and Preparation: College Plans and Preparation: Plan to attend 4-year college?Plan to attend 4-year college? Have taken the SAT/ACT?Have taken the SAT/ACT? College Enrollment -- one year later College Enrollment -- one year later Enrolled in any college: 2 year or 4 yearEnrolled in any college: 2 year or 4 year Enrolled in a four year college or universityEnrolled in a four year college or university

13 Pattern of Change Expected

14 Control variables: Race/EthnicityRace/Ethnicity GenderGender Generational Status (Senior Survey only)Generational Status (Senior Survey only) Social Class (Parental Education, Home ownership Family Income above/below poverty level)Social Class (Parental Education, Home ownership Family Income above/below poverty level) Family Structure (Senior Survey only)Family Structure (Senior Survey only)

15 Findings The social experiment to increase the transition from high school to college is effecting students’ behaviors consistently in 2 of 3 WSA high schools. The social experiment to increase the transition from high school to college is effecting students’ behaviors consistently in 2 of 3 WSA high schools. An observed effect on 4 year college plansAn observed effect on 4 year college plans Strong effect by 2004 and 2005Strong effect by 2004 and 2005 An observed effect on SAT/ACTAn observed effect on SAT/ACT Strong effect by 2004 and 2005Strong effect by 2004 and 2005 An observed effect on attending 4 year collegeAn observed effect on attending 4 year college Consistent impact for HS #1 for 2003, 2004, and 2005Consistent impact for HS #1 for 2003, 2004, and 2005 Consistent impact of HS #3 by 2004, 2005Consistent impact of HS #3 by 2004, 2005

16

17

18 Implications WSA initiatives appear to have success on key outcomes of plans, preparation, and college attendance WSA initiatives appear to have success on key outcomes of plans, preparation, and college attendance Since initiatives are implemented in different settings with different forms of implementation--not all strategies and settings were equal in outcomes Since initiatives are implemented in different settings with different forms of implementation--not all strategies and settings were equal in outcomes One setting failed to show significant effects One setting failed to show significant effects Expectation of delayed effects Expectation of delayed effects

19 Pattern of Change Expected

20 Thanks and Comments

21

22

23 Effects of WSA on Educational Outcomes

24 Source: Stoops 2004: 2

25 Underlying Analytical Design Y= B0 + B1 (WSA) + B2 (Year 2) + B3 (Year 3) + B4 (Year 4) + B5 (Year 5) + B6(Year 2 * WSA) + B7 (Year 3 * WSA) + B8 (Year 4 * WSA) + B9 (Year 5 * WSA) + Bn (Xn) This model is applied separately for each WSA program school

26 Interpreting Results: We want to see a significant value greater than 0 for B4 (Year 1 * WSA), B5 (Year 2 * WSA) and B5 (Year 3 * WSA) which implies that being in a WSA school post-program implementation has increased the probability of these behaviors compared to pre-program We want to see a significant value greater than 0 for B4 (Year 1 * WSA), B5 (Year 2 * WSA) and B5 (Year 3 * WSA) which implies that being in a WSA school post-program implementation has increased the probability of these behaviors compared to pre-program