1. Bringing Evidence-Driven Progress To Education MSP Regional Conference - Boston Jon Baron The Coalition for Evidence-Based Policy March 30, 2006

2. Coalition for Evidence-Based Policy Board of Advisors Robert Boruch - Co-Chair, Campbell Collaboration Jonathan Crane - Sr Fellow, Progressive Policy Institute David Ellwood - Dean, Harvard’s JFK School Judith Gueron – fmr President, MDRC Ron Haskins – Sr Advisor to President for Welfare Policy Robert Hoyt – Founder, Jennison Associates Blair Hull – Founder, Hull Trading Co David Kessler – fmr FDA Commissioner Jerry Lee – President, Jerry Lee Foundation Dan Levy – Researcher, Mathematica Diane Ravitch – fmr Asst Secretary of Education Laurie Robinson – fmr Asst Attorney General, DOJ Howard Rolston – fmr Director of Research, HHS/ACF Isabel Sawhill – fmr Associate Director of OMB Martin Seligman – fmr President, American Psychological Assn Robert Slavin – Co-Director of CRESPAR at Johns Hopkins Univ Robert Solow – Nobel Laureate in Economics, MIT Nicholas Zill – Vice-President, Westat

3. The Problem: Little progress in many areas of social policy U.S. has made no significant progress versus substance abuse since 1990. U.S. has made very limited progress in raising K-12 achievement over past 30 years. U.S. poverty rate today is higher than in 1973.

4. Randomized trials have identified a few highly- effective interventions : SMART tutoring program for at-risk readers in grades 1-2  At 2-yr followup, increases students’ national percentile ranking in reading ability from ~20 th percentile to ~30 th percentile. LifeSkills Training Reduces smoking by 20% and serious levels of substance abuse by ~30% by end of high school. Good Behavior Game At age 19-21, reduces substance abuse by 30-60%; at age 11-13, reduces school suspensions, conduct disorder, smoking/hard drug use by 25-60%.

5. Randomized trials have identified a few interventions that are ineffective/harmful: DARE Ineffective in preventing substance use, according to randomized trials (is now being redesigned). Scared Straight Causes a small increase in subsequent criminal activity by participating youth.

6. We seek to advance a major federal/state strategy to: 1. Fund rigorous studies – particularly randomized trials – to build the knowledge base of research-proven interventions. 2. Spur widespread use of such interventions by recipients of federal and state funds.

7. Challenges: How one measures an intervention’s effectiveness is extremely important Randomized trials are considered the “gold standard” in other fields for establishing what works. Nonrandomized designs – even the best designs – can produce erroneous conclusions. Well-matched comparison-group studies can produce “possible” evidence, but results need to be confirmed in randomized trials wherever feasible.

8. Job Training Partnership Act: Impact on Earnings of Male Youth (Non-arrestees) Program group Control group

9. Impact of Career Academies on High School Graduation Rates Graduated late Graduated on time Randomized trial results Comparison-group study results* 0 10 20 30 40 50 60 70 80 90 100 Career Academy Students Source: Data provided by James Kemple, MDRC Inc. Randomized Control Group 48.6 12.6 11.8 72.9 8.4 11.1 72.2 10.8 8.6 *The comparison group is similar students in similar schools nationwide, who are enrolled in the general (as opposed to career) curriculum. Their estimated graduation rates are statistically adjusted to control for observed differences in their background characteristics versus the program group. 72.9 8.4 11.1 Career Academy Students Non-randomized Comparison Group* Percent Graduating

10. Evidence-Based Education “Help Desk” Launched January 2006 by the What Works Clearinghouse of the U.S. Education Department’s Institute of Education Sciences www.whatworkshelpdesk.ed.gov

11. Recent federal efforts to advance evidence-based education: Education Sciences Reform Act of 2002 “Scientifically-based research” provisions in No Child Left Behind. ED’s competitive priority for grant applications that include a rigorous evaluation. Increased Congressional funding for rigorous education research.

12. We held focus groups of potential Help Desk users: Focus groups included (i) fed/state/local officials, (ii) researchers, and (iii) program providers. They asked: What are the concrete, practical steps we need to take to -- 1. Advance rigorous evaluations; and 2. Identify and implement evidence-based programs and practices.

13. Illustrative areas of researcher need for assistance: Can we randomly assign individual students, or do we need to randomly assign whole classrooms or schools? How do we gain the cooperation of school officials in (i) random assignment, and (ii) access to student educational records? What are the key items to get right when conducting a randomized controlled trial?

14. Illustrative Areas of Publisher/Program Provider Need for Assistance We’d like to sponsor a rigorous evaluation of our program that will meet WWC standards. How can we find a researcher(s) capable of conducting a rigorous evaluation? Are there conditions in which a randomized controlled trial can be conducted at low cost (e.g., $50,000 - $75,000)? What are the key items I should ask the research team to include in its study report?

15. Illustrative Areas of Federal/State/Local Officials Need for Assistance: Federal or State official: How do we incentivize our grantees to do a randomized controlled evaluation, or effectively implement evidence-based practices? (e.g., what language to include in our RFP?) School district superintendent: What are key items to get right when replicating an evidence-based program in our school district.

16. Help Desk seeks to provide answers to many of these questions from the user’s desktop:  Thru practical, easy-to-use resources (e.g., WWC, and user-friendly “how-to” guides).  Users access these tools through a web site managed by knowledgeable “moderators” immediately available by phone or email.

17. Illustrative Examples of Help Desk “How-To” Resources:  Key Items to Get Right When Conducting a Randomized Controlled Trial in Education  Reporting the Results of Your Study: A User-Friendly Guide  Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User-Friendly Guide  How To Conduct Rigorous Evaluations of Math and Science Partnership (MSP) Projects: A User-Friendly Guide  Future Guides: Finding a capable evaluator, interpreting and applying What Works Clearinghouse reports, conducting low- cost randomized controlled trials

18. Jon Baron The Coalition for Evidence-Based Policy www.excelgov.org/evidence www.excelgov.org/evidence jbaron@excelgov.org 202-530-3279