Conducting Randomized Evaluations in Schools Lisa Markman Associate Director Education Research Section Princeton University

Education Research Section Promotes the use of research in education decision-making. Conducts and promotes high quality education research through the use of experimental and quasi-experimental research designs

No Child Left Behind, Schools and Research No Child Left Behind Act (2001) requires programs and practices to be based on “scientifically based research”. Term “scientifically based research” appears throughout the law over 100 times. Applies to curricular interventions, professional development programs, psychosocial interventions and supplemental education services.

What is Scientifically Based Research ? The term scientifically based research means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs (According to NJ Dept. of Ed)

Necessary Components of SBR Employs systematic, empirical methods that draw on observation or experiment Involves rigorous data analysis that are adequate to test the stated hypotheses and justify the general conclusions drawn Relies on measurement or observable methods that provide reliable and valid data Is evaluated using experimental or quasi-experimental designs with appropriate control or comparison groups (with a preference for random assignment) Reports results with sufficient detail and clarity Has been subject to external (peer) review (According to NJ Dept. of Ed)

Evaluations in Education Undesirable Features of Many Evaluations No comparison group or comparison group is not comparable to treatment group Only “successfully” treated schools/students are included in the analysis The evaluators are not independent of the developers of the program Desirable features of Evaluations Participants are randomly assigned to “treatment” status All initial participants are included in the analysis The evaluators are independent of the developers of the program

Recipe for an Ideal Measurement Step 1: Randomly assign students. Step 2: Obtain PERFECT test. Step 3: Test students in fall. Step 4: Test students in spring. Step 5: Set non-school parameters back to original conditions. Step 6: Repeat Steps 3 & 4. Step 7: Impact = (Step 4 – Step 3) Year 2 - (Step 4 – Step 3) Year 1

The “2 nd Best” Option Randomization Randomly assign some students, classes, school districts, or states to the reform in question; the others are randomly assigned to the status quo (or another reform).

Why Randomize ? To obtain an appropriate control group Allows one to answer what would happen in the absence of the intervention (creates a credible counterfactual)

Common Concerns About Randomized Evaluations It is too disruptive Unethical to deny service to some children Legal obligations require providing services to all children It does not answer all questions Fear of “random”

Is It Too Disruptive ? It can be. Good researchers work with teachers, schools and school districts to understand school culture, and school needs.

Example of a Randomized Research Evaluation To determine the effectiveness of a computerized math curriculum 3 Urban School districts Randomize on the class level by school

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A Randomized Evaluation of A Computerized Math Curriculum Current School Schedule

TeacherPeriod 1Period 2Period 3Period 4 AAlg 1Alg 1AAlg 1 B Alg 1A C Alg I D Alg 1Alg 1A TeacherPeriod 1Period 2Period 3Period 4 ARegular LAB B Regular C D LAB Randomized Evaluation of A Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

It’s Unethical/Illegal to Deny Service We don’t know if program/reform actually works Ways around completely excluding all students All children receive “something” or can contrast two programs

It Doesn’t Answer All Questions True Benefit of Mix-Methods Approach

The Fear of “Random” Usually can not treat all students Way to organize groups to get meaningful information

To Sum…. Need more high-quality impact evaluations to help guide everyday decisions Conducting evaluations does not have to be super disruptive Schools should make sure that researchers work with them It’ll take cooperation from all to improve the quality of education research on impacts

Information for Librarians What Works Clearinghouse Education Databases

What Works Clearinghouse Created as a resource for decision makers in education. Provides databases and reports regarding the effectiveness scientifically based programs and practices. Current topics – –Adult Literacy, –Character Education, –Delinquency, Disorderly and Violent behavior, –Dropout Prevention, –English Language Learners, –Math (Elementary and Middle School) –Peer Assisted Learning –Reading, –Early Childhood

Examples of Education Databases National Center for Education Statistics (NCES) –National Assessment of Educational Progress (NAEP) –High School and Beyond (HS&B) –The Common Core of Data (CCD) – National Longitudinal Study of Youth (NLSY) –NLSY97 –NLSY79 – –Check websites for a list of all databases