1. RESEARCH SYNTHESIS: The Core Methodology of Evidence-Based Reviews Susan N. Labin, Ph.D. susan@susanlabin.com

2. Susan N. Labin-AEA 20102 Research Synthesis: What & Why ? Secondary data analysis-aggregates findings from primary research Secondary data analysis-aggregates findings from primary research Establishes base of existing knowledge Establishes base of existing knowledge Informs future policies, programs, practices, & research Informs future policies, programs, practices, & research

3. Susan N. Labin-AEA 20103 Types of Syntheses Broad-Based: from variety of designs & types of data Broad-Based: from variety of designs & types of data –GAO –CDC: Guide to Community Preventive Services Meta-Analysis: only from randomized control trials (RCT) Meta-Analysis: only from randomized control trials (RCT)

4. Susan N. Labin-AEA 20104 Broad-Based Research Synthesis: GAO & CDC Moderately systematic Moderately systematic Qualitative & quantitative data Qualitative & quantitative data Draw from a variety of designs Draw from a variety of designs Consensus building-between academics and practitioners Consensus building-between academics and practitioners

5. Susan N. Labin-AEA 20105 Research Synthesis: Systematic Decision Rules Research Synthesis Research Synthesis –Broad-based and meta-analysis –Systematic decision rules –Six basic steps, e.g. selection & inclusion criteria Traditional Literature Reviews Traditional Literature Reviews –Broad-based –Less systematic decision rules

6. Susan N. Labin-AEA 20106 Brief History: Designs @ Primary & Secondary Evaluation 1960’s – RCT “gold standard” from physical and biological sciences Evaluation 1960’s – RCT “gold standard” from physical and biological sciences 1970’s and 1980’s interest in synthesis and meta-analysis for social sciences 1970’s and 1980’s interest in synthesis and meta-analysis for social sciences Resolve contradictory findings Resolve contradictory findings Improve external validity or generalizability of small sample RCTs Improve external validity or generalizability of small sample RCTs Meta-Analysis physical & biological roots Meta-Analysis physical & biological roots

7. Susan N. Labin-AEA 20107 Evidence-Based Practice Reviews (EBPR) As synthesis-assess evidence to inform policy & practice As synthesis-assess evidence to inform policy & practice Who: predominately federal Who: predominately federal How same & different from synthesis? How same & different from synthesis? How can synthesis inform EBPR? How can synthesis inform EBPR?

8. Susan N. Labin-AEA 20108 EBPR Who Is Doing Them? 1. Agency Healthcare Research & Quality 2. Centers for Disease Control & Prevention-2 3. Office Juvenile Justice & Delinquency Prevention 4. Substance Abuse & Mental Health Services Administration 5. US Dept of Ed 6. Coalition for Evidence-Based Policy (non- federal)

9. Susan N. Labin-AEA 20109 Syntheses & EBPR: Common Steps & Decision-rules Many of same steps Many of same steps Quality ratings Quality ratings Inclusion-Design Inclusion-Design –Some restrict to only RCT –Others include quasi-experimental, observational –Some only RCT can get highest rating, most RCT highest ratings

10. Susan N. Labin-AEA 201010 EBPR Differs from Synthesis Inclusion-some EBPR include only those with positive outcomes Inclusion-some EBPR include only those with positive outcomes Only a few EBPR aggregate analyses across studies Only a few EBPR aggregate analyses across studies Less emphasis w EBPR on reliability between reviewers Less emphasis w EBPR on reliability between reviewers

11. Susan N. Labin-AEA 201011 Addressing Practice Challenges to RCT Social interventions linked with reinforcing social interactions or culture- address w group/place/cluster/saturation trials Social interventions linked with reinforcing social interactions or culture- address w group/place/cluster/saturation trials Assumptions re replicability of protocol or intervention Assumptions re replicability of protocol or intervention Feasibility Feasibility

12. Susan N. Labin-AEA 201012 RCT Challenges for Social & Behavioral Sciences Inherent differences between physical and biological sciences and social and behavioral sciences Inherent differences between physical and biological sciences and social and behavioral sciences Lack of blinding and double blinding Lack of blinding and double blinding Relevance of individual choice Relevance of individual choice

13. Susan N. Labin-AEA 201013 Comparing Results: RCT versus Quasi Designs Complex: quasi may underestimate (Shadish) Complex: quasi may underestimate (Shadish) Same labor market, same questionnaire quasi & exp Same labor market, same questionnaire quasi & exp give “fairly close” results give “fairly close” results

14. Susan N. Labin-AEA 201014 Comparing Results: RCT vs Quasi Designs cont. Lipsey & Wilson (2001) Lipsey & Wilson (2001) –> 300 M-A; > 16,000 ind studies –Exp vs Quasi not much difference –One-group pre-post over estimates compared to quasi –Measurement =.07 proportion of variance –Exp vs Quasi=.04

15. Susan N. Labin-AEA 201015 How To Strengthen EBPR? Include negative results to inform counter- factual Include negative results to inform counter- factual –How consistent results –What else could be contributing to the effects Participatory and transparent reviews Participatory and transparent reviews –Quality criteria and decision rules –Reliability & resolving discrepancies Common measurement-include criteria of respondent meanings Common measurement-include criteria of respondent meanings

16. Susan N. Labin-AEA 201016 How To Strengthen EBPR cont. Aggregate across projects to improve Aggregate across projects to improve –Generalizability –> Confidence in “best practice” designation –Choice of projects for replication –Knowledge Base Develop & use technologies for categorizing studies and depicting patterns of results. Develop & use technologies for categorizing studies and depicting patterns of results.

17. Susan N. Labin-AEA 201017 References Labin, S. 2008. Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, (eds.) Nick L. Smith and Paul R. Brandon, Guilford Press, NY, NY. Labin, S. 2008. Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, (eds.) Nick L. Smith and Paul R. Brandon, Guilford Press, NY, NY. GAO-30-10, 2009. Program Evaluation: A Variety of Methods Can Help Identify Effective Interventions GAO-30-10, 2009. Program Evaluation: A Variety of Methods Can Help Identify Effective Interventions Websites for the Effective Practice Reviews Websites for the Effective Practice Reviews

18. Susan N. Labin-AEA 201018 THANK YOU Susan N. Labin, Ph.D. 301.564.0764 susan@susanlabin.com www.susanlabin.com