1. November 15, 20151 Regional Educational Laboratory - Southwest The Effects of Teacher Professional Development on Student Achievement: Finding from a Systematic Review of Evidence Presented by: Kwang Suk Yoon (AIR) Teresa Duncan (AIR) Sylvia Wen-Yu Lee (National Taiwan U) Kathy Shapley (Edvance Research, Inc.) American Education Research Association New York March 27, 2008

2. November 15, 20152 Teacher Professional Development  One of key policy strategies for standards-based reform efforts  No Child Left Behind Act Teacher Quality provisions “High-quality” professional development Expectations: PD activities are to be regularly evaluated for their impact on teacher effectiveness and improved student achievement  Until recently, little systematic effort on vetting the PD effectiveness Background

3. November 15, 20153 Objective AIR completed a fast-turnaround study, sponsored by the Regional Education Laboratory Southwest (RELSW), which was funded by IES To conduct a systematic review of research-based evidence on the effects of teacher professional development (PD) on student achievement

4. November 15, 20154 Research-based Evidence on the effects of PD on student achievement Challenges in demonstrating research-based evidence Quality of professional development  Workable theory of actions Logic model  Sufficient implementation Quality of empirical study  Valid causal inferences  Rigorous study design

5. November 15, 20155 Overview of methodology Systematic review  Using explicit & transparent methods  Following a set of standards  Being accountable, replicable, and updatable Review protocol  Aligned with What Works Clearinghouse (WWC) evidence standards  Study selection criteria  Multi-stage, multi-coder review process Screening, coding & reconciliation Evidence Review Tool (ERT)

6. November 15, 20156 Study selection criteria Topic: Inservice teacher professional development Population: K-12 students and their teachers Subject: reading/English/language arts, mathematics, or science Study design  Randomized controlled trial (RCT)  Quasi-experimental design (QED) with matched comparison group Student achievement outcome Measures and their psychometric properties Time: 1986-2006 Country: Australia, Canada, the United Kingdom, or the United States

7. November 15, 20157 Overview of the review process

8. November 15, 20158 Literature search & prescreening Literature searches  Keyword searches on 7 major electronic databases  Contacted key researchers  Identified 1,343 citations potentially addressing the effects of PD on student achievement Prescreening  Quickly scanning the abstracts for a few selection criteria (e.g., empirical study?)  Narrowed down to 132 relevant studies

9. November 15, 20159 Reasons for failing Stage-1 screening criteria  Of 132 relevant studies, about two-thirds failed in the “study design” criterion Stage-1 coding

10. November 15, 201510 Stage-2 Coding Determining the study quality ratings  N = 27 relevant studies eligible for quality ratings 9 met WWC evidence standards  5 met evidence standards with reservations  4 met evidence standards without reservations 18 failed to meet the evidence standards  RCT – randomization, attrition, disruption, etc.  QED – baseline equivalence, attrition, disruption, etc.

11. November 15, 201511 Nine studies Carpenter et al., 1989 (RCT) Cole, 1992 (RCT) Duffy et al., 1986 (RCT) Marek & Methven, 1991 (QED) McCutchen et al., 2002 (QED) McGill-Franzen et al., 1999 (RCT) Saxe et al., 2001 (QED) Sloan, 1993 (RCT) Tienken, 2003 (RCT with group equivalence problems)

12. November 15, 201512 Documentation of studies  Effect size  Characteristics of PD Form Duration, contact hours Content – Kennedy’s (1998) classification Provider Participants (volunteers?) Information about the implementation of PD Stage-3 coding

13. November 15, 201513 Results (1) Paucity of rigorous studies  Only 9 studies met evidence standards  Mostly small-scale, underpowered efficacy trials Distribution of 9 studies  By study design 5 RCT 4 QED  By content area Concentrated in reading/English/language arts  By grade level All focused on elementary school level

14. November 15, 201514 Results (2) Overall effect size  Average of 20 effect sizes (drawn from 9 studies) =.54 Of 20 effect sizes, 12 were not statistically significant. Effects by subject area  Fairly consistent across three subject areas Effects by form, duration, and intensity of PD  Lack of variability in form: all workshop or summer institute  Some evidence of the effect of intensive PD Effects by content of PD  No consistent pattern  Failed to replicate Kennedy’s (1998) finding

15. November 15, 201515 Conclusion There is some evidence of positive effects of PD on student achievement.  Caveats  The limited number of studies and the variability in their professional development approaches preclude detailed conclusions about the effectiveness of particular professional development programs or about the effectiveness of professional development by such features as form and content. The PD impact research is still at its developmental stage.

16. November 15, 201516 Suggestions (1) Doing more well-conducted efficacy trials, replications, and effectiveness trials Improving the design of PD impact studies  Redressing common reasons for failing WWC evidence standards For example, lack of baseline equivalence in QED  Increasing statistical power to detect effects  Aligning outcome measures with PD  Examining mediation effects  Addressing potential confounding effects of PD with other important instructional factors (e.g., curriculum)

17. November 15, 201517 Suggestions (2) Adequate documentation of PD & study  PD: theory of action, implementation  Sample and cluster, if any  Group assignment  Baseline equivalence  Effect size (ES) Use of structured abstract to facilitate research synthesis  Structured abstract (Mosteller et al, 2004)  Claim-based structured abstract (Kelly & Yin, 2007)

18. November 15, 201518 Contact info & links Kwang Suk Yoon  kyoon@air.org (202-403-5358) Link to our RELSW report on the IES website  http://ies.ed.gov/ncee/edlabs/projects/project.asp? id=70 Link to our presentation material  http://www.pdal.net/reports.asp Thank you!

19. November 15, 201519 Message A “stainless steel” law of systematic reviews may be operating—namely, “the more rigorous the review, the less evidence there will be that the intervention is effective.” (Peter Rossi)

20. November 15, 201520 Logic Model