1 Children Left Behind in AYP and Non-AYP Schools: Using Student Progress and the Distribution of Student Gains to Validate AYP Kilchan Choi Michael Seltzer Joan Herman Kyo Yamashiro UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST)

2 Research Questions  Are there schools that meet AYP yet still have children who are not making substantial progress? i.e., leaving some children behind?  Are there schools that do not meet AYP yet still enable students to make substantial progress?  Do AYP schools achieve a more equitable distribution of student growth? Are students at all ability levels making progress in AYP schools?  Are there non-AYP schools that are reducing the achievement gap?

3 Sample  Large, Urban District in WA  2,524 students  2 time-point ITBS reading scores (Grade 3 in 2001 & Grade 5 in 2003)  Standard Errors of Measurement (SE) on ITBS reading scores (Bryk, et.al., 1998)  72 schools Average # students/school: 35 Average % qualifying for FRPL: 36.4% Average % Minority (African American, Native American, or Latino): 68.6%

4 AYP vs. Non-AYP schools In WA  School AYP decision made based on 4 th grade performance on WA Assessment of Student Learning (WASL)  51 schools made AYP; 21 did not make AYP in baseline year (2002), according to WA State Dept of Ed  Our study re-evaluates AYP and non-AYP schools with a new value-added model (an advanced hierarchical Modeling technique)

5 A New Methodology for School Effect / Accountability: Latent Variable Regression in Hierarchical Model  Additional Questions and Interest using LVR-HM  Move beyond school mean growth rates and examine hidden/underlying process  How equitably is student achievement distributed? (The distribution of student growth: Children Left Behind or No Child Left Behind)  Why is it that student achievement is distributed in a more equitable fashion in some schools than in other schools?

6 Distribution of Student Growth (Relationship between initial status and rate of change)

7 Why a New Value-Added Model (LVR-HM)?  Gains or Growth might be highly dependent upon a status at certain point of time (i.e., initial status)  Initial status can be a strong and important factor to “valued-added gain or growth”  New value-added gain or growth:  Adjusting student intake characteristics PLUS student initial difference  Adjusting school intake characteristics, policies and practice PLUS school initial difference  Thus, providing value-added gain or growth PLUS revealing the distribution of student achievement

8 Latent Variable Regression Hierarchical Model (LVR-HM)  Level 1: Time series within student Y ti =  0i +  1i Time ti +  ti  ti ~ N (0, 1)  Estimating initial status and gain for each student i with standard errors  Level 2: Student level  0i =  00 + r 0i r 0i ~ N (0,  00 )  1i =  10 + b( 0i -  00 ) + r 1i r 1i ~ N (0,  11 ) Cov(r 0i, r 1i ) = 0  Gain for student i is modeled as function of his or her initial status

9 Different Levels of Initial Status  Many ways to define performance subgroups based on initial status  Examined gains for 3 performance subgroups within each school  Defined by initial status  Hi Performers: 15 pts above the school mean initial status  Mean: School mean initial status  Low Performers: 15 pts below the school mean initial status

10 Estimating Expected Gains for Different Levels of Initial Status  We estimate expected (predicted) gain for each of the performance subgroups using LVR-HM  Model-based estimation, not separate group analysis  Point estimate of gain & its 95% confidence interval (statistical inferences)  Possible to estimate expected gains after controlling for factors that lie beyond school’s control (e.g., student SES, school compositional factors)

11  Only 12 of 52 AYP schools have 95% interval above the district avg.  1 AYP school’s 95% interval includes 0 Expected mean gain in ITBS reading scores for AYP schools

12 Expected mean gain in ITBS reading scores for non-AYP schools  2 Non-AYP schools have 95% interval above district avg.

13  7 AYP schools’ 95% interval  30  3 AYP schools’ 95% interval includes 0 (low performers make no gains) Expected gain for low-performing students (AYP schools)

14 Expected gain for low-performing students (non-AYP schools)  5 Non-AYP schools have gains for low performers >20

15 Expected gain for high-performing students (AYP schools)  9 AYP schools’ 95% interval  30  3 AYP schools’ 95% interval < 10 (high performers make little or no gains)

16 Expected gain for high-performing students (non-AYP schools)  5 Non-AYP schools’ 95% interval  30  3 Non-AYP schools’ 95% interval < 10 (high performers make little or no gains)

17 Distribution of Gains Within A School  Type I: Substantial gain across all performance subgroups (e.g., no child left behind – ex: AYP school #8, non-AYP school #26)  Type II: No adequate gain for high performers; substantial gain for low performers (ex: AYP schools #19, non-AYP school #27)  Type III: No adequate gain for low performers; substantial gain for high performers (ex: AYP schools, non-AYP school #6 )

18 15pts above the school meanEqual to the school mean15pts below the school mean Estimate95% intervalEstimate95% intervalEstimate95% interval AYP School Type I Sch. #8 Sch. #22 Sch. # ( 30.7, 43.8 ) ( 30.0, 43.2 ) ( 30.8, 42.8 ) ( 35.2, 42.2 ) ( 31.7, 42.0 ) ( 32.2, 40.5 ) ( 35.0, 45.7 ) ( 31.1, 43.9 ) ( 31.1, 41.1 ) Type II Sch. #19 Sch. # ( 6.9, 35.9 ) ( 18.2, 39.0 ) ( 22.8, 44.1 ) ( 27.8, 40.3 ) ( 30.7, 60.3 ) ( 30.5, 49.6 ) Type III Sch. #28 Sch. # ( 33.0, 51.2 ) ( 31.4, 39.5 ) ( 27.4, 35.1 ) ( 28.5, 34.0 ) ( 12.1, 28.7 ) ( 23.7, 40.4 ) Non-AYP school Type I Sch. # ( 24.1, 41.7 )32.2( 27.6, 36.9 )31.6( 22.5, 40.4 ) Type II Sch. # ( 9.0, 27.2 )24.6( 17.6, 31.5 )30.5( 21.5, 40.1 ) Type III Sch. #6 Sch. #38 Sch. # ( 31.5, 50.8 ) ( 30.0, 49.9 ) ( 31.2, 44.2 ) ( 26.2, 38.5 ) ( 25.5, 34.2 ) ( 26.9, 34.4 ) ( 13.6, 33.7 ) ( 9.3, 30.2 ) ( 17.0, 30.2 )

19 Distribution of student gain for 3 AYP schools

20 Distribution of student gain for 3 non-AYP schools

21 Comparing Features: AYP & the CRESST Approach AYPCRESST Approach Data Structure Cross-sectional (follow grade levels, e.g., 4 th graders in a school, over time) Longitudinal (follow individual students over time) Performance Measure (Outcome) Proficiency levels (using cut scores) Individual gains or growth Subgroup Demographic characteristicsPerformance-level groups Plus Demographic characteristics Adjustments / Controls for Student or School Characteristics No controls or adjustments, just disaggregations – loss of advantages when comparing against other schools Can adjust for differences between schools and students in the model Type of Growth Examined Percent Proficient may mask different underlying growth patterns: Even flexibility given to schools through Safe Harbor option is only for movement around the proficiency cut score More complete picture of growth PLUS growth distribution

22 Different Growth By Performance Subgroups & Demographic Subgroups

23 Conclusions  Analyses using our alternative approach: More informative picture of growth using individual, longitudinal student gains More complete picture of how student growth is distributed within a school  Stimulate discussion among teachers and administrator to identify students in need earlier (Seltzer, Choi & Thum, 2003)  Encourage educators to think about achievement levels rather than (or in addition to) current subgroup categories - may be more productive and actionable