What Works Clearinghouse Susan Sanchez Institute of Education Sciences

Purpose: To promote education decision making through a web-based dissemination system featuring comprehensive, systematic, high- quality reviews of studies on the effectiveness of educational interventions (programs, products, practices and policies).

The WWC Does Not: l Endorse educational interventions l Conduct field studies of the effects of interventions Rather, the WWC reports on the effectiveness of educational interventions as measured by available evidence.

Challenges l Research literature too abundant for individual selection. Where to begin? l Difficult to sift and know what to trust and what to use. Lots of claims of effectiveness out there, but what can you trust? l It has to be transparent if it’s to be believed. l Complex, technical issues can take more than forever and ever to resolve.

Challenges l Opps, we got the answer but now there are new advances in methodology. l Consumers need a fast response for evidence on what works. l Pretty wonky stuff. How do you create user-friendly web-based reports and products that meets your customers’ needs?

WWC Systematic Review Process A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review.

WWC Systematic Review Process l Select the Topic l Develop the review protocol l Research questions l Search parameters l Inclusion/Exclusion criteria l Conduct a literature search & identify the research l Screen and code studies for relevance and methodological quality l Analyze the results of eligible studies l Summarize results

Study Review Process Does Not Pass Screen Meets WWC Evidence Standards with Reservations Submissions from the Public & Intervention Developers Literature Searches Screening Standards Pass Screen Study Reviewed Against the WWC Evidence Standards Does Not Meet WWC Evidence Screens Develop Protocol

Three Stages of Review l Screen studies for relevance l Select only relevant evidence l Assessing strength of the evidence and sort by rigor l Select only credible evidence l Sort credible evidence by whether any reservations l Identify other important study characteristics

Stage 1: Screening for Relevance l All potentially relevant studies identified through the extensive search Does the study provide evidence that, if credible, would be a valuable addition to the knowledge base regarding the effectiveness of the focal intervention?

Six Essential Screeners 1) Relevant time frame 2) Relevant intervention 3) Relevant sample 4) Relevant outcome 5) Adequate outcome measures 6) Adequate reporting

Stage 2: Assessing Strength of Evidence l The goal is to apply consistent criteria to sort evidence by its credibility into three buckets: Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Screens

The WWC Evidence Standards (applied to individual studies) Meets Evidence Standards RCTs without severe design or implementation flaws Meets Evidence Standards with Reservations RCTs with severe design or implementation flaws QEDs with equating and without severe design or implementation flaws Does Not Meet Evidence Screens Studies not relevant to review Studies with fatal design or implementation flaws

Five Factors Govern Judgments l Method for forming intervention and comparison groups l Evidence on baseline equivalence l Sample attrition l Possible contamination of study conditions l Threat of teacher confound

Stage 3: Other Study Characteristics l Variations in people, settings and outcomes represented l How generalizable are the findings? l Results reported by subgroups, settings and outcomes l What is the breadth of the outcomes reported? l Statistical results available l How complete is the reporting? l Do the estimates reflect statistical controls for baseline characteristics?

Evidence Base of Character Education Programs l Over 70 programs were submitted to or identified by the WWC. l 41 school-based programs met WWC definition and were eligible for review. l 93 studies on the 41 programs were collected. l 13 programs had at least one study meeting evidence standards, either with or without reservations l 27 programs had no studies passing evidence screens l 1 program is under review

13 Character Education Programs with at Least one Study Meeting Evidence Standards, either with or without Reservations 1. An Ethics Curriculum for Children8. Skills for Action 2. Building Decision Skills9. Skills for Adolescence 3. Caring School Communities10. Too Good for Drugs 4. Connect with Kids11. Too Good for Violence 5. Facing History and Ourselves12. Too Good for Violence and Drugs 6. Lessons in Character 7. Positive Action 13. Voices Literature and Character Education Program

Distribution of 92 Studies on 40 Programs across Evidence Categories Meet evidence standards with reservations Meet evidence standards Do not meet evidence screens

Common Reasons for Studies Failing to Meet Evidence Screens l Not meet relevance screens l Lack of a comparison group l Severe overall or differential attrition for QEDs l Lack of baseline equivalence for QEDs l Confound in one-teacher/school-per-condition studies l Inadequate statistical reporting for ES computation

For More Information on the WWC Visit our website at Subscribe for WWCUpdate, the WWC’s electronic news alert through our website Phone: WWC