1. Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade Students' Learning and Study Skills Jenny Nagaoka, Melissa Roderick, and Melanie LaForce June 29,2010 2010 IES Conference

2. What is AVID? History: –Began in 1980 in San Diego, CA –National AVID office was created to provide standards and oversight in implementation and expansion –2009-2010 school year: nearly 4,500 schools in 45 states AVID in the Chicago Public Schools (CPS): –Part of a larger initiative to improve student preparation for and access to postsecondary education –First implemented in CPS in 2003-2004 in 8 high schools –Expanded to 42 high schools by 2007-2008 and 59 high schools currently, including 2 all-AVID high schools.

3. The AVID Program Program & Goals: School-based program, provides academic and social support Designed to improve the preparation and college access of middle-achieving students The AVID Elective Class- “How to be a student” Curriculum around Supports & Skill Building: Academic – organizational, note-taking, writing, critical thinking skills, and questioning skills Social – peer support, teacher guidance, college advising The AVID Site Team Site Coordinator/Teacher; AVID Administrator; English, math, science, social studies teachers

4. AVID in the Chicago Public Schools (CPS) Part of a larger initiative to improve student preparation for and access to postsecondary education First implemented in CPS in 2003-2004 in 8 high schools Expanded to 42 high schools by 2007-2008 and 59 high schools currently, including 2 all-AVID high schools.

5. Research on AVID Some positive evidence –More likely to be enrolled in college (Mehan, et al., 1996 ) –Schools with AVID have higher graduation rates, advanced course-taking (Watt, et al., 2006) –But do not address selection effects SRDC British Columbia Study –Randomized Experiment –Non-urban, non-minority district

6. Data and Sample Data –Consortium on Chicago School Research data archive CPS school records, including high school transcripts Biennial survey of CPS students Sample –First-time 9 th graders in 14 CPS AVID high schools, not in special education, enrolled in both fall and spring semesters 7,357 students in pre-AVID cohorts (2002 and 2003) 14,031 students in post-AVID cohorts (2005 to 2008), 2,521 AVID students and 11,510 non-AVID students 51% female and 49% male, 68 % African-American, 23% Latino, 10% White

7. 1.Are AVID students doing better than other students on critical 9 th grade outcomes that predict high school and college graduation? 2. Are these differences due to selection effects? Research Questions

8. Why 9 th Grade Outcomes? 9 th grade critical for HS graduation and college enrollment. –Freshman year success strongly linked to HS graduation (Allensworth and Easton, 2005) –Early GPA benchmark critical (Roderick, Nagaoka and Allensworth, 2006)

9. Does AVID have an impact on 9 th grade outcomes? Supports & Skill Building –Academic: study habits –Social supports: peer support, sense of belonging, teacher- student trust, teacher support, engagement Academic outcomes –Cumulative weighted GPA and probability of 3.0 or higher GPA –On-track –Absences –English GPA and probability of B or better (English I) –Math GPA and probability of B or better (Algebra or geometry)

10. Dealing with Selection Bias 1. Propensity score matching –Estimate probability for treatment based on measured variables Drawbacks: Only matching on measured characteristics (test scores, demographic characteristics, fixed effects for elementary school, 7 th and 8 th survey responses) –Sample of students in 14 AVID schools is divided into deciles based on propensity scores

11. Dealing with Selection Bias 2. Simulated Propensity Scores –Pre-post comparison of treatment constructed by simulating a comparison group of students who have a high propensity for AVID in the pre-AVID period –Apply propensity model coefficient weights and simulate probability for AVID in students before AVID was implemented –Three groups: pre, post-period AVID, post-period not in AVID Pre- and post non-AVID should be similar once we control for background characteristics and cohort effects. –Sample is divided into deciles based on propensity scores with same cut points for pre and post cohorts.

12. Average GPA for 9 th grade AVID and Non-AVID students Post-AVID cohorts 2003-2008

13. Probability of having a B or better GPA for 9 th grade AVID and Non-AVID students Post-AVID cohorts 2003-2008

14. On-track and absences for 9 th grade AVID and Non-AVID students Post-AVID cohorts 2003-2008 9 th grade on-track rates Days absent per semester in 9 th grade

15. Allensworth & Easton (2007) What matters for staying on-track and graduation from high school in Chicago Public Schools

16. Propensity Score Matching 9 th grade academic outcomes for Post-AVID cohorts 2003-2008 Decile 7Decile 8Decile 9Decile 10 Core GPA 0.11 0.15*0.17** 0.08 Probability of 3.0+ GPA 0.15 0.36*0.40* 0.05 English GPA 0.10 0.27**0.22*0.26** Probability of B in English 0.18 0.23 0.38**0.31* Math GPA -0.050.21 0.27†-0.09 Probability of B in math 0.150.09 0.16†0.01 Log Absences -0.12 -0.23**-0.19*-0.19** On-track 0.32* 0.14-0.01-0.03

17. Propensity Score Matching 9 th grade academic outcomes for Post-AVID cohorts 2003-2008 Core GPA Probability of B or Better Core GPA English GPAProb B or Better (Eng) Math GPAProb B or Better (Math) Log Absences On-Track Decile 10 0.080.050.26**0.31*0.01-0.09-0.19**-0.03 Decile 9 0.17**0.40*0.22*0.38**0.16†0.27†-0.19*-0.01 Decile 8 0.15*0.36*0.27**0.230.090.21-0.23**0.14 Decile 7 0.110.150.100.180.15-0.05-0.12.32*

18. Simulated Propensity Scores 9 th grade academic outcomes for Pre-AVID cohorts (2001 and 2002) and Post-AVID cohorts (2003-2008) Core GPA Probability of B or Better Core GPA English GPAProb B or Better (Eng) Math GPAProb B or Better (Math) Log Absences On-Track Decile 10 -0.00-0.210.110.23-0.020.07-0.13*-0.141 Decile 9 0.100.36*0.140.220.070.13-0.23***-0.22 Decile 8 0.16*0.35 † 0.22*0.140.130.29-0.22***0.00 Decile 7 0.15*0.290.070.180.26**0.20-0.120.23

19. Simulated Propensity Scores 9 th grade academic outcomes for Pre-AVID cohorts (2001 and 2002) and Post-AVID cohorts (2003-2008) Decile 7Decile 8Decile 9Decile 10 Core GPA 0.15*0.16* 0.10 -0.00 Probability of 3.0+ GPA 0.290.35 † 0.36* -0.21 English GPA 0.07 0.22* 0.14 0.11 Probability of B in English 0.180.14 0.220.23 Math GPA 0.26** 0.130.07-0.02 Probability of B in math 0.20 0.290.130.07 Log Absences -0.12 -0.22***-0.23**-0.13* On-track 0.23 0.00 -0.22 -0.141

20. 9 th grade study habits for Post-AVID cohorts 2003-2008 How much do you agree with the following: I set time aside to do my homework and study. I try to do well on my schoolwork even when it isn’t interesting to me. If I need to study I don’t go out with my friends. I always study for tests.

21. Propensity Score Matching 9 th grade academic skill and social outcomes for Post-AVID cohorts 2003-2008 Decile 10Decile 9Decile 8Decile7 Sense of belonging in school.59*.41 †.10.53* Study habits.28.18.06-.15 Importance of high school for the future.67*-.27.43 † -.05 Peer support for academic achievement.21.28-.08.03 Teacher-student trust.16.10.02-.03 Teacher support.13.52 †.06.81*

22. Propensity Score Matching 9 th grade academic skill and social outcomes for Post- AVID cohorts 2003-2008

23. Conclusions AVID is clearly doing a good job of selecting students AVID students are performing poorly in school –Average GPA is <2.0 Lack of Strong Effects may be linked to differences between CPS and other AVID districts –Implementation issues –“Middle Achieving” may mean different things –Teaching study skills may not translate to large urban districts AVID focuses on improving student skills without changing what is happening in the classroom.