June 25, Regional Educational Laboratory - Southwest Review of Evidence on the Effects of Teacher Professional Development on Student Achievement: Finding & Suggestions for Future Evaluation Designs Presented by: Kwang Suk Yoon (AIR) Teresa Duncan (AIR) Sylvia Lee (Nat’l Taiwan U) Kathy Shapley (Edvance Research, Inc.) American Evaluation Association Washington, DC November 8, 2007

June 25, Objective To conduct a systematic review of research-based evidence on the effects of teacher professional development (PD) on student achievement

June 25, Overview of Methodology Systematic review  Using explicit & transparent methods  Following a set of standards  Being accountable, replicable, and updatable Review protocol  Aligned with What Works Clearinghouse (WWC) standards  Study selection criteria  Rigorous evidence standards  Multi-coder, multi-stage review process Screening, coding & reconciliation Evidence Review Tool (ERT)

June 25, Overview of the coding process

June 25, Electronic searches by keywords Literature search

June 25, Study selection criteria Relevance of study by:  Topic  Population  Subject  Study design Randomized controlled trial (RCT) Quasi-experimental design (QED) with matched comparison group  Student achievement outcome  Measures and their psychometric properties  Time  Country

June 25, Prescreening stage  N = 1,343 “potentially relevant” studies Reasons for failing selection criteria (1)

June 25, Stage-1 coding  N = 132 relevant studies Reasons for failing selection criteria (2)

June 25, Stage-2 coding  N = 27 relevant studies eligible for quality ratings  18 failed to meet WWC evidence standards RCT – randomization, attrition, disruption, etc. QED – baseline equivalence, attrition, disruption, etc. Inter-rater reliability  4 met evidence standards without reservations  5 met evidence standards with reservations Reasons for failing evidence standards

June 25, Stage-3 coding  Effect size  Characteristics of PD Form, duration, contact hours Content Provider Participants (volunteers?) Information about the implementation of PD Replicability issue Documentation of studies

June 25, Results (1) Paucity of rigorous studies  Only 9 studies met evidence standards Distribution of 9 studies  By study design 5 RCT 4 QED  By content area Concentrated in reading  By grade level All focused on elementary school level

June 25, Results (2) Overall effect size  Average of 20 effect sizes drawn from 9 studies =.54 Of 20 effect sizes, 12 were not statistically significant. Nine of those 12, however, are substantively important according to WWC conventions. Effects by subject area  Fairly consistent across three subject areas Effects by form, duration, and intensity of PD  Some evidence of effect of intensive PD Effects by content of PD  No consistent pattern  Failed to replicate Kennedy’s (1998) finding

June 25, Suggestions (1) Matching units of assignment and analysis Increasing statistical power to detect effects Considering potential confounding effects of PD and other important instructional factors (e.g., curriculum)

June 25, Suggestions (2) Adequate documentation of PD & study  PD implementation  Sample and cluster, if any  group assignment  baseline equivalence  effect size (ES) calculating and reporting effect sizes (weighting ES, adjusting ES for multiple comparisons, and correcting ES for clusters) Use of structured abstract to facilitate research synthesis structured abstract (Mosteller et al, 2004) claim-based structured abstract (Kelly & Yin, 2007)

June 25, Contact info & links Kwang Suk Yoon  ( ) Link to our RELSW report on the IES website  id=70 Link to our presentation material  Thank you!