School Reform and Beyond Brian Rowan Burke A. Hinsdale Collegiate Professor in Education Research Professor, Institute for Social Research Professor of Sociology

Part I: The Problem

How Big Are the Gaps? At first testing, high risk students cannot identify letters, low risk can. By the end of 3rd grade (month 40), high risk students can identify common sight words, low risk students can make basic inferences from connected text and are moving toward extrapolation. High-Risk vs. Average-Risk High-Risk vs. Low-Risk SDMonthsSDMonths Beginning of Kindergarten End of Kindergarten End of 1st Grade End of 3rd Grade

Approaches to Narrowing Gaps High quality pre-schooling: can increase achievement at entry into schooling High quality elementary schooling: accelerates academic growth during schooling Out-of-school reforms address factors “beyond the control of schools” that limit academic growth before and during schooling

Projected Outcomes: High Quality Pre-School + Effective Early Grades Schooling Model-Based Estimates of Reading Achievement Growth from ECLS Data Gaps in Achievement for High Risk vs. Low Risk Students Under Different Conditions Hi Risk Child Hi Start in K Hi Risk Child Hi Start in K Effective School Beginning of Kindergarten 0.72 months0.25 months End of Kindergarten 0.56 months-0.65 months End of 1st Grade 0.72 months-0.54 months End of 3rd Grade 2.76 months-0.68 months

Part II: Participation in R&D Development Full Scale Effectiveness TrialExploratory Research Efficacy Trial The Cycle of R&D

Part II: Participation in R&D Development Full Scale Effectiveness TrialExploratory Research Efficacy Trial The Cycle of R&D

Development Projects Schools and researchers have interest in addressing/resolving a common “problem of practice.” Researchers and school professionals work together in a few schools to: conceptualize problem(s) design and test “tools/strategies” to address problem(s) conduct “informal” research to improve/evaluate Products: new curriculum units new assessment instruments new teaching practices new administrative practices/arrangements

Development Projects: An Exemplar at Michigan The Henry Ford Museum/Michigan SOE Partnership Team: Museum curator, UM faculty and graduates students, classroom teachers. Initial Problem: Could you help us develop technology to improve access to museum resources for students and enhance the quality of their learning? Initial Findings: Students have a “one shot” engagement at museum. The tools: “Virtual Curator” and “Virtual Explorer” keep students engaged before and after visit. The cycle: Six revisions over two years with Allen Park schools.

Development Projects: Features of District Participation Small scale project: a few schools a limited focus Intensive collaboration: multiple partners assumption of shared expertise mutual learning mutual benefit Multi-year cycle of development and testing

Part II: Participation in R&D Development Full Scale Effectiveness TrialExploratory Research Efficacy Trial The Cycle of R&D

Exploratory Research Often emerges from a development project in a handful of schools. Additional schools now recruited into project. Developers conduct more systematic research to examine: theory of action (are things working as they should?) effects on outcomes (are we getting the results we want?) Research findings: can serve as basis for further development work can serve as basis for efficacy trial

Exploratory Research: An Exemplar at Michigan Project-based science (PBS) in the Detroit Public Schools middle schools. Project begins with development in pilot sites. In 1998, project moves to 13 teachers in 10 schools. Six years later: 63 teachers in 26 schools. Researchers found: among “matched” students, the more PBS units a classroom used, the higher the MEAP science scores. among “matched students”, those using PBS show higher levels of science motivation. successful scaling required more “specified” implementation procedures.

Exploratory Research: Features of District Participation District agrees to allow spread of initial project Students, teachers, administrators now become: “implementors” of project (not co-developers) “subjects” of research (not co-researchers) Research becomes more intensive and rigorous: matched samples formal instruments (e.g., surveys, observations)

Part II: Participation in R&D Development Full Scale Effectiveness TrialExploratory Research Efficacy Trial The Cycle of R&D

Efficacy/Effectiveness Trials Aim is to evaluate the effectiveness of well-developed interventions as implemented under “normal conditions of practice.” Efficacy trials examine effectiveness of intervention in favorable conditions of implementation (e.g., in medicine, a clinic) Effectiveness trials examine effectiveness of intervention under broad conditions of use (e.g., in medicine, regular practice setting) Researchers often use “random assignment” to treatment vs. control conditions (RCT). Researchers will want to gather detailed data on: degree of implementation effects of treatment on outcomes, where control group is counterfactual To assure adequate statistical power, number of schools/classrooms involved in research can be quite large classrooms (40 treatment/40 control) 70 – 80 schools (40 treatment/40 control)

Efficacy/Effectiveness Trials: An Exemplar at Michigan  A Study of Instructional Improvement examined the effectiveness of three of America’s largest CSR programs:  Accelerated Schools Project (ASP)  America’s Choice (AC)  Success for All (SFA) This was a 4-year, quasi-experiment conducted in ~120 high poverty elementary schools: Matched samples of ~30 schools working with each of the three interventions, plus a matched sample of ~30 schools not implementing one of the interventions Inside all 120 schools, data were gathered annually on school improvement processes, instruction, and student achievement  Looking at these programs, researchers examined:  whether and how the CSR programs altered instruction in schools  whether and how the instruction inside CSR schools affected student achievement

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) The CSR Programs used different strategies to implement their proposed instructional changes ASP emphasized adaptation/discovery AC and SFA emphasized fidelity

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) ASP did not produce changes in instructional practice

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) SFA succeeded in producing “skill based” reading instruction

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) AC produced “literature-based” reading instruction

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) SFA’s “skill-based” program improved reading achievement at the early grades

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII) AC’s literature based program improved reading achievement in the upper grades

Efficacy/Effectiveness Trials: Features of District Participation  Districts as “subjects” of research.  Districts “recruited” through:  Direct contact from researchers  Support from intervention programs  Districts allowed to serve as gatekeepers to schools  District research office requires proposals  District provides access to schools  School choice to participate  Schools are given “incentives” for participation  Schools receive $2000/year  Teachers receive incentives ($20/q’naire, $ for logging)  Parental consent carefully managed (passive vs. active)  Benefit defined as “cosmopolitan” not “local” These strategies had varying degrees of success in promoting instructional change. Two of the three programs improved student achievement in reading, but not at all grades.

Summary of District Participation in Research Development Efficacy/Effectivness Symmetric CollaborationAssymetric Collaboration Intensive but localized impact Less intensive but broader impact Informal/flexible reckoning of benefits Formal reckoning of Benefits (i.e., incentives)

“School Reform and Beyond” in the Research Cycle Pre-school research: Ready to Learn: Developing new pre-school literacy program (development phase, going to exploratory research, then to efficacy trial) Early elementary: CARSS: Combining effective reading interventions with effective behavioral interventions (moving from design to development stage) K-12: UM SOE: Education Leadership Center (development partners needed) Working at “system” level to meld school, family, community supports for student success. Based on principals of Kellogg Foundation Spark Initiative