1. Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy October 3, 2012 Can Formal–Informal Collaborations Improve Science Literacy in Urban Middle Schools? The Impact of Urban Advantage Meryle Weinstein Emilyn Ruble Amy Ellen Schwartz

2. What is Urban Advantage? Collaboration between New York City Department of Education and 8 New York City informal science education institutions Led by American Museum of Natural History Provides professional development to middle school science teachers and opportunities to students to engage in authentic science practice Workshops for science teachers and school administrators Science materials/equipment for schools, teachers, & students Vouchers for class field trips, family field trips and visits Launched in 2004-05 with 31 schools and in 2011-12 had 137 Funded by NY City Council and DOE

5. Why Urban Advantage? Increased calls for collaboration between formal and informal institutions, particularly around science Growing aversion to science among students, particularly by the time they reach middle school Middle school is time to grab students attention – “gateway” for high school science courses Strong science instruction has been found to impact science persistence in high school

6. Our goal is to determine if the Urban Advantage program is effective. Does Urban Advantage lead to increased student achievement? UA students outperform non-UA students on 8 th grade Intermediate Level Science Test Modest impact Magnitude increases over time Students who attend UA schools are more likely to take and pass a Science Regents in 8 th or 9 th grade than students at non-UA schools

7. We make use of a rich longitudinal student level database. Student-Level Data NYCDOE Administrative Data, 2004-05 – 2009-10 Socio-demographic characteristics, educational needs, and test scores

8. We use multiple approaches to estimate the impact of UA. Compare the observable differences between students at UA schools and students at non-UA schools Use quasi-experimental techniques to estimate UA’s total effect on student achievement. Use a “difference-in-difference” approach.

9. Raw performance data suggests UA is effective Student Weighted Mean Achievement, 8 th Grade Intermediate Level Science (ILS) Test – Percent Proficient

10. Differences between UA and non-UA schools prior to joining UA are largely insignificant. 200420062008 UANon-UAUANon-UAUANon-UA N of Schools31289433667227 Total Enrollment1079851611586738667 (434)(468)(426)(425)(647)(373) % Black41.8436.4737.6538.9533.8938.52 (28.1) (29.3)(28.9)(34.0)(29.3) % Hispanic35.1039.9142.2140.4535.5641.18 (22.9)(25.4)(27.5)(26.0)(23.2)(26.4) % Asian/Other13.269.677.168.7712.569.14 (19.6)(12.1)(12.0)(13.3)(14.5)(14.1) % White9.8213.9612.5311.1817.7810.71 (18.2)(19.6)(22.1)(18.0)(19.1)(18.0) % ELL10.2610.6010.7910.809.6711.23 (7.8)(10.6)(9.9)(11.1)(4.2)(11.7) % Free Lunch75.3771.1063.3069.9055.8966.30 (21.8)(23.5)(23.1)(23.3)(31.5)(25.8) % Prof. ELA33.1739.4236.1140.1458.4148.75 (16.6)(20.5)(20.4)(21.3)(18.9)(21.6) % Prof. Math38.1043.6336.3443.0873.2762.42 (17.4)(20.6)(23.2)(22.2)(18.7)(23.5) % Prof. Science38.2345.0336.6139.5247.5750.16 (20.9)(24.8)(23.0)(24.0)(27.7)(22.7) Standard deviations are in parentheses Bold indicates differences are statistically significant at.05 level or less % Proficient is the percent scoring in levels 3 or 4

11. Basic Model Y ijt = β 1i jt + β 2 PreUA ijt β 3 Post UA ijt + β 4 ST ijt + θ j + ε ijt  Y = individual student outcome  PreUA = indicator variable for whether school joined UA in next year  PostUA = indicator variable for whether school has joined UA  ST = vector of student characteristics  θ = school fixed effect   = random error term

12. Controlling for student characteristics, students at UA schools outperform those at non-UA schools. Overall, students at UA schools perform.041 sd higher than those at non-UA schools in science Little change is seen on ELA or math or 8 th grade students After their first year of joining UA, students at UA schools perform, on average,.056 sd higher than those at non-UA schools in science

13. Demographic Characteristics 8 th Grade Sample TotalUANon-UA % Black31.326.533.7 % White14.216.613.0 % Asian15.117.713.7 % Hispanic39.439.139.5 % Female50.249.450.6 % Poor83.081.184.0 % Spec. Ed9.29.19.0 % LEP10.611.410.2 % UA33.71000.0 N337987113974223923 N Schools458157301

14. OLS Regression with School Fixed Effects, 2004 to 2010 ScienceMathELA Model 1Model 2 β /se Yr Prior UA0.0020.011 -0.001 (0.018)(0.021)(0.024)(0.017) UA in Any Year0.041* (0.016) Yr Ent. UA0.0440.0360.026 (0.024)(0.027)(0.021) Yr Post UA0.056*0.0140.022 (0.028)(0.031)(0.023) Black-0.397*** -0.408***-0.375*** (0.017) (0.022)(0.023) Hispanic-0.226*** -0.270***-0.275*** (0.015) (0.020)(0.021) Asian0.162*** 0.407***0.064** (0.019) (0.027)(0.022) Female-0.072*** 0.027***0.194*** (0.005) (0.004) School FEYES R-Square0.35 0.330.32 N401270 425820409572 * p<0.05, ** p<0.01, *** p<0.001 Year, LEP, and Special Education dummies not shown Robust clustered standard errors in parentheses

15. Robustness Checks Controlled for prior achievement Magnitudes are smaller but still significant when controlling for prior math or reading scores Lagged Math Scores, Post UA Yrs β =.037, p <.10 Lagged Reading Scores, Post UA Yrs β =.045, p <.10 Caveat: Sample size decreases by 50,000 No statistically significant findings for percent proficient

16. Descriptive Statistics, High School Sample Full Sample UA Not-UA N=252,129N=79,090N= 173,039 Mean UA 31.37 (0.46) 1000.0 % Black 32.38 (0.46) 27.97 (0.45) 34.40 (0.48) % Hispanic 38.96 (0.48) 38.56 (0.49) 39.14 (0.49) % Asian 16.74 (0.37) 20.05 (0.40) 15.22 (0.36) % White 13.31 (0.34) 14.7 (0.35) 12.66 (0.33) % Poor 87.88 (0.33) 82.62 (0.38) 84.62 (0.36) Attended a STEM High School 9.83 (0.30) 9.72 (0.30) 9.87 (0.30) Took Living Environment in 8 th or 9 th Grade 60.33 (0.49) 67.39 (0.47) 57.10 (0.50) Took Earth Science in 8 th or 9 th Grade 10.60 (0.31) 10.94 (0.31) 10.44 (0.31)

17. Linear Probability Coefficients, High School Outcomes Model 3Model 4 β /s.e Attending a STEM School 0.014*** (0.003) 0.008* (0.004) Attending a Partial STEM SchoolNS Taking Living Environment Regents in 8 th or 9 th Grade 0.255*** (0.012) 0.246*** (0.012) Passing Living Environment RegentsNS Passing Living Environment Regents with 65 or higher 0.040*** (0.006) 0.032*** (0.006) Passing Living Environment Regents with 85 or higher 0.062*** (0.005) 0.054*** (0.005) Taking Earth Science Regents in 8 th or 9 th Grade 0.039*** (0.007) 0.033*** (0.007) Passing Earth Science Regents 0.029*** (0.0006) 0.012* (0.0006) Passing Earth Science Regents with 65 or higher 0.059*** (0.007) 0.037*** (0.008) Passing Earth Science Regents with 85 or higher 0.062*** (0.005) 0.054*** (0.005) School Fixed EffectsYES * p<0.05, ** p<0.01, *** p<0.001 Robust clustered standard errors in parentheses Control variables not shown are: Black, Hispanic, Asian, Female, Poor, Special Education, LEP, and for Model 4 lagged_zmath.

18. Conclusions Student performance increases with the implementation of UA and the magnitude of the difference increases over time. Little change on ELA or math for 8th grade students, suggesting the effect is not merely reflecting coincident overall school improvement UA also contributes to post-8 th grade outcomes. Biggest impact is on the likelihood of taking the Living Environment Regents in 8 th or 9 th grade.

19. Policy Implications First estimates of the impact of a science program on academic achievement Inquiry as a method to approach science instruction is not emphasized in schools but more common in informal science institutions Benefits of collaboration between formal and informal science institutions Importance of strong partnerships between these institutions and between the institutions and the school district in which they work Need for improved data linking teachers and students Future research: Inside the black box

20. http://steinhardt.nyu.edu/iesp