Michelle A. Duda, Ph.D. Melissa Van Dyke, LCSW Chris Borgmeier, Ph.D. Susan Davis, NC SPDG Margaret McGlinchey, MI SPDG Evidence-based Professional Development 2011 SPDG Regional Meeting Feb 2011 Washington, D.C.

A Few Questions:  What do we mean when we use the term, “professional development?”  What outcomes are we attempting to achieve?  What do we know about the professional development strategies that are likely to achieve particular outcomes?  How can we monitor and improve the quality of professional development over time?

Overcoming Barriers with  Common language  Common frameworks  Defined approach (best practices)  Common measures (commitment to continuous quality improvement) Common PD Challenges

COMMON LANGUAGE Overcoming Barriers :

Training  “the process of bringing a person to an agreed standard of proficiency, by practice and instruction”  “the acquisition of knowledge, skills, and competencies as a result of the teaching of vocational or practical skills and knowledge that relate to specific useful competencies.”knowledgeskillscompetenciesvocational

Adult Learning Defined  “a collection of theories and methods for describing the conditions under which the processes of learning are optimized (Merriam, 2001; Trotter, 2006; Yang, 2003).” Trivette, Dunst, Hamby, O’Herin, 2009, p. 1

Professional Development "Professional development... goes beyond the term 'training' with its implications of learning skills, and encompasses a definition that includes formal and informal means of helping teachers not only learn new skills but also develop new insights into pedagogy and their own practice, and explore new or advanced understandings of content and resources.” Modified from

Professional Development (cont.) “... [This] definition of professional development includes support for teachers as they encounter the challenges that come with putting into practice their evolving understandings about the use of new skills and technology.” Modified from

Reflection: Language Barriers  How has issues with terminology, definition or language impacted your work?

COMMON FRAMEWORKS Overcoming Barriers :

Effective Interventions Actual Supports Years 1-3 Outcomes Years 4-5 Every Teacher Trained Fewer than 50% of the teachers received some training Fewer than 10% of the schools used the CSR as intended Every Teacher Continually Supported Fewer than 25% of those teachers received support Vast majority of students did not benefit Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006 Longitudinal Studies of a Variety of Comprehensive School Reforms Implementation Science

What Works  Implementation Drivers  Common features of successful supports to help make full and effective uses of a wide variety of innovations

Core Implementation Components © Fixsen & Blase, 2008 Improved Outcomes for... Competency Drivers Organization Drivers Leadership Program/Initiative (set of practices) How: What: Why: Capacity to provide direction/vision of process Staff capacity to support students/families with the selected practices Institutional capacity to support staff in implementing practices with fidelity

Core Implementation Components © Fixsen & Blase, 2008 Improved Outcomes for... Competency Drivers Organization Drivers Leadership Program/Initiative (set of practices) How: What: Why: Capacity to provide direction/vision of process Institutional capacity to support staff in implementing practices with fidelity Professional Development

© Fixsen & Blase, 2008 Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Integrated & Compensatory Competency Drivers Organization Drivers Leadership AdaptiveTechnical Improved Outcomes for... Program/Initiative (set of practices)

Why Focus First on Teachers’ Behavior?  In education, the Teacher IS THE INTERVENTION  Build supports in relationship to what the Teacher needs to be competent  Create the conditions under which teachers can do the right thing for the right reason at the right time to maximize positive results  Wide ranging inputs (individuals with all their past history and current realities)  Build alignment of implementation practices and outcomes on what needs to happen at the point of the “learning exchange”

Improved outcomes for students Participants with improved knowledge and skills around identified practice(s) Think about those who are the participants in (receivers of) your professional development activities… What features would you suggest be in place around the Competency, Organization and Leadership drivers so that professional development is provided in a “host environment” to support sustainability and promote fidelity? Participants of Professional Development

Participants with improved knowledge and skills around identified practice(s) Professional development providers Think about those who are the providers of your professional development activities… What features would you suggest be in place around the Competency, Organization and Leadership drivers so that your providers conduct quality professional development? Providers of Professional Development

The NCSIP Process: Focus Of Work Research- Based Practice Staff Training On-site Fidelity Checks On-site Program Reviews Student Progress Evaluation

DEFINED APPROACH Overcoming Barriers: “Best Practices”

The BIG QUESTION: What outcomes are you hoping to achieve?

Use training “alone” strategies if…  You want to disseminate new information  You want to create “buy in”  You want to clarify “truths” and dispel “myths”  You are talking with a broad audience

Use training “alone” strategies …  To increase knowledge about the effective program or practice related to...  underlying theory of change  intervention components  rationales of key practices  To increase familiarity with the use of new skills  To increase awareness of what it feels like to begin to use new skills and to receive expert feedback

Training “Best Practices”  Theory grounded (adult learning)  Skill-based  Data-based (pre and post testing)  Feedback to Selection and Feed Forward to Supervision  Trainers have been trained and coached

SAMPLE: Reading Foundations Training Pre and Post Test Items  Some tips to consider:  Pre-post test items should be linked to training/teaching objectives  Data from pre-post tests should be used for training revisions, developing coaching service delivery plans, etc.  Presentation of pre-post test should emphasize the non-punitive nature of this type of assessment and that it will be shared back with the participants  Be a good model of using data to get better

Training “Best Practices”  Theory grounded (adult learning)  Skill-based  Feedback to Selection and Feed Forward to Supervision  Data-based (pre and post testing)  Trainers have been trained and coached What do we know about each of these “best practices?”

Adult Learning “Best Practices”  The most effective training includes learner experiences related to planning, application, and deep understanding  Use a diverse array of Adult Learning methods (“where 5 or 6 adult learning method characteristics were used, the average effect size was almost 1.25”)  Learners need to be engaged actively in the learning process  Multiple learning experiences, large doses of learner self-assessment and reflection, instructor facilitated learner assessment  Small numbers of participants, multiple occasions Trivette, Dunst, Hamby, O’Herin, 2009, p

Adult Learning “Best Practices” Trivette, Dunst, Hamby, O’Herin found best practices in each of the following categories:  Introducing Information  Illustrate/demonstrate  Practicing  Evaluation  Reflection  Mastery Trivette, Dunst, Hamby, O’Herin, 2009, p

Skill-based “Best Practices”  Behavior Rehearsals (vs. Role Plays)  Knowledgeable Feedback Providers  Practice to Criteria

Data Based “Best Practices”  Develop and use pre/post tests to determine to what extent knowledge and skill levels are being improved  Outcome data collected and analyzed  Fidelity measures collected and analyzed

Collins, S. R., Brooks, L.E., Daly, D.L., Fixsen, D.L., Maloney, D.M., & Blase, K. A. (1976) Staff Training

Feedback “Best Practices”  Feed Forward of pre/post data to Coaches/Supervisors  Feedback of pre/post data to Selection and Recruitment

Feedback “Best Practices” Of course, there are even more opportunities to benefit from feedback… SelectionTrainingCoaching Performance Assessment

© Fixsen & Blase, 2008 Performance Assessment (Fidelity) Coaching Training Selection Systems Intervention Facilitative Administration Decision Support Data System Integrated & Compensatory Competency Drivers Organization Drivers Leadership AdaptiveTechnical Improved Outcomes for training participants Effective Training Practices

Implementation-informed PD  If Professional Development is in support of a well-defined, effective intervention then Professional Development includes  Training  Coaching  Performance assessment  and Selection

Training, plus coaching  Some well-defined interventions (EBPs) will require additional support to teachers post- training:  Joyce and Showers (2002) data »With coaching = 95% use in classrooms »Without = 5% use in classrooms  Rogers, Wellens, & Conner (2002) data »About 10% of what is taught in business workshops actually is put into practice  Significant data across domains that feedback improves performance

Coaching “Best Practices”  Design a Coaching Service Delivery Plan  Use multiple sources of information for feedback – Direct observation is critical  Provide regular feedback to all “Drivers”  Develop accountability structures for Coaching – Coach the Coaches!  Regular review of adherence to Coaching Service Delivery Plan  Multiple sources of information for supervisor feedback

Performance Assessment Best Practices  Transparent Processes – Orientation  What, When, How, Why  Use of Multiple Data Sources  Context  Content  Competency  Tied to positive recognition – not used ‘punitively’

Selection “Best Practices”  Job description clarity about accountability and expectations  Pre-Requisites are related to “new practices” and expectations (e.g. basic group management skills).  Interactive Interview Process:  Behavioral vignettes and Behavior Rehearsals  Assessment of ability to accept feedback  Assessment of ability to change own behavior

EXAMPLE: Qualifications of Potential SIP II Foundation Trainers Successfully complete the Reading Foundations course including all of the tasks (level 2) and meet the passing criteria (80%) on the Structure of Language Test. Demonstrate mastery of the content of the Reading Foundations course as well as the ability to deliver this information to others. Mastery of content also includes knowledge of the texts identified as basic to the course. Provide appropriate review and feedback of tasks completed by participants in the Reading Foundations course.

Enhancing Professional Development  Promotes adult learning and changes adult behavior  Empowers individuals to improve their craft  Ensures that new skills are used and/or current skills are refined  Integrates selection, training, coaching, and performance assessment to promote the development of a highly effective workforce  Supports the creation of a “hospitable environment” to allow for these new behaviors to be used and improved

Core Implementation Components © Fixsen & Blase, 2008 Improved Outcomes for... Competency Drivers Organization Drivers Leadership Program/Initiative (set of practices) How: What: Why: Capacity to provide direction/vision of process Institutional capacity to support staff in implementing practices with fidelity Professional Development

Discussion Question To what extent are current approaches for professional development designed to develop the necessary competence in participants to skillfully use the most effective educational approaches to accomplish the goal of improved outcomes for students?

COMMON MEASURES Overcoming Barriers : “Commitment to Continuous Quality Improvement”

Integrating All We Know  A well-defined and effective intervention  Clarity about the “active ingredients”  Clear criteria to assess performance (that correlates with the desired outcomes)  An effective approach to training  An effective coaching system  An effective performance assessment process  Use of selection “best practices”

Improvement Cycles

Improving PD Effectiveness  Collect and use data (pre/post tests to determine to what extent knowledge and skill levels are being improved)  Develop the capacity of the trainers (establish clear performance criteria, based on “what works,” then select, train, coach, and assess trainers to criteria)  Improve the ways in which training participants are “prepared” for training (create readiness)

Questions to Consider 1. What do we mean when we use the term, “professional development?” 2. What outcomes are we attempting to achieve? 3. What do we know about the professional development strategies that are likely to achieve particular outcomes? 4. How can we monitor and improve the quality of professional development over time? 5. What are some resources and tools we can use to support sustainable and aligned professional development?

For More Information Michelle A. Duda Melissa Van Dyke Chris Borgmeier

For More Information State Implementation and Scaling up of Evidence-based Practices (SISEP) Dean Fixsen, Karen Blase, Rob Horner, George Sugai  Concept paper  Annotated bibliography  Data on scaling up  Scaling up Briefs

Evidence-based Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: Implementation Research: A Synthesis of the Literature

Thank You for your Support  Annie E. Casey Foundation (EBPs and cultural competence)  William T. Grant Foundation (implementation literature review)  Substance Abuse and Mental Health Services Administration (implementation strategies grants; national implementation awards)  Centers for Disease Control & Prevention (implementation research)  National Institute of Mental Health (research and training grants)  Juvenile Justice and Delinquency Prevention (program development and evaluation grants  Office of Special Education Programs (scaling up capacity development center)  Administration for Children and Families (child welfare reform; capacity development)  Duke Endowment (child welfare reform)