1. 1 Carol M. Trivette, Ph.D. Orelena Hawks Puckett Institute Asheville and Morganton, North Carolina Presentation Prepared for a Webinar with the State Personnel Development Grantees, U.S. Office of Special Education Programs, January 11, 2012

2. 2

3. 3 Participant Chat This “Wordle” highlights the word implementation. When considering your SPDG work, what do you think about when you see the word implementation?

4. 4 Purposes of the Presentation Describe the key characteristics of an evidence-based approach to implementation. Describe the impact of using trainee reflection during the implementation process. Describe the relationship between teacher classroom practices and trainer implementation fidelity.

5. 5 A Lesson Learned from More Than 40 Years of Experience No intervention practice, no matter its evidence base, is likely to be adopted and used if the implementation methods used to teach or train practitioners to use the practice are themselves ineffective. Therefore, concern for the characteristics of implementation practices that are associated with optimal learner and practitioner outcomes should be of paramount importance as part of implementation research.

6. 6

7. 7 Two Types of Evidence-Based Practices Evidence-Based Intervention Practices Early childhood intervention practices Evidence-Based Implementation Practices Adult learning methods

8. 8 Definition of Terms Implementation practices include methods and procedures used by implementation agents (trainers, coaches, instructors, supervisors, etc.) to promote interventionists’ use of evidence- based intervention practices. Intervention practices include methods and strategies used by intervention agents (teachers, therapists, clinicians, parents, etc.) to effect changes or produce desired outcomes in a targeted population or group of recipients (e.g., infants and toddlers with disabilities).

9. 9 Participatory adult learning Coaching Mentoring Just-in-time training Guided design Accelerated learning Early child contingency learning Interest-based child learning Natural environment practices Preschool classroom practices Communication and language learning Early literacy learning Family systems intervention practices Implementation PracticesIntervention Practices Relationship Between the Two Types of Practices

10. 10 Principles Evidence-based intervention practices are a necessary—but not sufficient—condition to ensure optimal outcomes for children and families. Practitioners must be trained on intervention strategies using evidence-based adult learning implementation practices.

11. 11 Participant Chat Implementation/Training Formats Thinking about professional development that targets classroom personnel, what training formats do you use most often?

12. 12

13. 13 Promoting Adoption of Family-Systems Intervention Practices 473 IDEA Part C early intervention practitioners 5 types of training (presentations, day and multi-day workshops, field-based and enhanced field-based) Participants were randomly assigned to complete the study outcome measure at one or six months after training The outcome measure included items on the usefulness of the training and the extent to which the training improved their abilities to work with families

14. 14 Six Characteristics Identified in How People Learn a Were Used To Code and Evaluate the Adult Learning Methods Planning Introduce Engage the learner in a preview of the material, knowledge or practice that is the focus of instruction or training Illustrate Demonstrate or illustrate the use or applicability of the material, knowledge or practice for the learner Application Practice Engage the learner in the use of the material, knowledge or practice Evaluate Engage the learner in a process of evaluating the consequence or outcome of the application of the material, knowledge or practice Deep Understanding Reflection Engage the learner in self-assessment of his or her acquisition of knowledge and skills as a basis for identifying “next steps” in the learning process Mastery Engage the learner in a process of assessing his or her experience in the context of some conceptual or practical model or framework, or some external set of standards or criteria a Donovan, M. et al. (Eds.) (1999). How people learn. Washington, DC: National Academy Press.

15. 15 Practice Characteristics Type of In-Service Training Conference Presentations Half-Day/ Full-Day Workshops Multi-Day Workshops Field-Based Training Enhanced Field-Based Training Trainer introduction of the practice ++++ Trainer illustration of use of the practice ±++++ Trainee application/use of the practice -±++++ Trainee evaluation of his/her use of the practice --±+++ Trainee reflection on his/her learning --±+++ Trainee assessment of learner mastery ---++ Multiple learning sessions--+++ NOTE. - = No activity or opportunity, ± = limited opportunity, + = multiple opportunities, and ++ = multiple and varied opportunities. Characteristics of the Different Types of In-Service Training

16. 16 Participants’ Judgments of the Benefits of the Five Types of Training

17. 17 Interaction Between Hours and Type of Training for Improved Family Systems Intervention Abilities

18. 18 Principles Training formats must be designed to ensure that evidence-based adult learning strategies can be used. More time spent using the most effective strategies increases the learning rate.

19. 19 After looking at these data, what training formats/strategies come to mind that you can use that might support changes in teacher practices? Participant Chat

20. 20

21. 21 Research Synthesis of Adult Learning Studies a Research synthesis of studies of accelerated learning, coaching, guided design, and just-in-time training 58 randomized control design studies 2,095 experimental group participants and 2,213 control or comparison group participants Combination of studies in university and non-university settings Learner outcomes included learner knowledge, skills, attitudes, and self- efficacy beliefs The influence of the adult learning methods on the learner outcomes was estimated by weighted Cohen’s d effect sizes for the differences on the post test scores for the intervention vs. nonintervention group participants a Dunst, C.J., Trivette, C.M., & Hamby, D.W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1), 91-112.

22. 22 Efficacy of the Adult Learning Methods Effect size for the intervention vs. nonintervention group comparisons is d =.42 (95% Confidence Interval =.36 to.47).

23. 23 Efficiency of the Four Adult Learning Methods

24. 24 Translational Findings Process for unpacking and unbundling the key characteristics of the adult learning methods

25. 25 Effect Sizes for the Six Adult Learning Characteristics Planning Application Understanding Average effect sizes and 95% confidence intervals for the relationships between the six adult learning method characteristics and the study outcomes.

26. 26 Most Effective Adult Learning Method Practices CharacteristicPracticeMean Effect Size IntroductionOut-of-class learner activities/self-instruction0.64 Classroom/workshop presentations0.63 Pre-class learner exercises0.54 IllustrationTrainer role playing/simulations0.55 Learner-informed input0.53 PracticingReal-life learner application0.94 Real-life learner application/role playing0.86 EvaluationSelf assessment of strengths/weaknesses0.94 ReflectionIdentify performance improvement goals1.27 Journaling/behavior suggestions0.82 MasteryStandards-based assessment0.86

27. 27 Cumulative Effects of Different Combinations of the Most Effective Adult Learning Method Practices Average Cohen’s d effect sizes and 95% confidence intervals for the relationship between different combinations (clusters) of adult learning methods characteristics and the study outcomes.

28. 28 PALS (Participatory Adult Learning Strategy) PLAN APPLICATION RECYCLE Active Learner Involvement Reflection and Mastery Practice and Evaluate Introduce and Illustrate Identify Next Steps in the Learning Process INFORMED UNDERSTANDING

29. 29 PALS PhasesTrainer RolesTrainee Roles IntroductionPreview learning topicComplete pre-training preview Describe key elementsPre-class/workshop exercises Provide examplesProvide input on the learning topic Include trainee inputIn-class/workshop warm-up exercises Illustrate application Demonstrate application ApplicationFacilitate applicationProvide examples of application Observe trainee applicationTrainee role playing, games, etc. Provide in vivo feedback/guidanceImplement/practice use of the subject matter Facilitate learner assessment of optionsEvaluate use of the knowledge or practice Informed Understanding Establish learning standardsStandards-based evaluation Engage learners in self-assessmentConduct self-assessment Provide guidance to learnersTrainer-guided learner reflection Provide behavioral suggestionsJournaling Group discussions of understanding Repeat Learning Process Joint planning Trainer guidanceIdentify needed information/experiences Trainer/trainee mentoring Trainer and Trainee Roles in the Different Phases of PALS

30. 30 Trainer/Coach Introduction of Targeted Practice

31. 31 Trainer/Coach Illustration or Demonstration of Targeted Practice

32. 32 Practitioner Application of the Targeted Practice

33. 33 Practitioner Evaluation of Use of the Targeted Practice

34. 34 Adult Learning Method Practices

35. 35 Practice Reflection Checklists Specify the practice by creating indicators for each project component Help teachers understand key characteristics of the practices Serve as the standards against which teachers examine and improve their practices

36. 36 Caregiver Responsive Teaching Checklist Center for Early Literacy Learning

37. 37 Participatory Adult Learning Strategy (PALS) Checklist

38. 38 Everyday Learning Activity Checklist Center on Everyday Child Language Learning

39. 39 Participant Chat How do you use checklists in training? Who completes them? Who provides feedback?

40. 40 Principles The more training strategies (introduce, illustrate, practice, etc.) used during training, the greater the change in practitioner practices. Encouraging practitioner self-reflection in all training phases increases the rate of change in practitioner practices. Checklists are effective tools for engaging practitioners in self-reflection practices.

41. 41

42. 42 Head Start Teacher Effectiveness and Child Outcomes 36 Classrooms in Early Head Start Programs:  18 children per classroom  2 full-time adults per classroom 49 teachers and teacher assistants:  Average teacher age was 40 years  Average years in current position were 5.1 (SD = 4.7) Collected the following data:  Background Demographics  Teacher Characteristics  Implementation Fidelity  Intervention Fidelity  Social Validity  Child Outcomes

43. 43 PALS (Participatory Adult Learning Strategy) PLAN APPLICATION RECYCLE Active Learner Involvement Reflection and Mastery Practice and Evaluate Introduce and Illustrate Identify Next Steps in the Learning Process INFORMED UNDERSTANDING

44. 44 Training Content Training focuses on two components of classroom practices known to promote positive child outcomes: Child Learning Practices Instructional Practices

45. 45 The key features of the professional development (PALS) model include: Active participation of teachers in their own learning The ability to respond to and individualize training to a teacher’s personal learning style and existing skills Use of early childhood classroom practices as standards against which teachers review their own practices Key Features

46. 46 Practice Reflection Checklist Specify the practice indicators for each project component  Child Learning Practices: 18 indicators  Instructional Practices: 16 indicators Help teachers understand key characteristics of the practices Serve as the standards against which teachers examine and improve their practices

47. 47 Training Tools Practice Scenarios Classroom Interest Assessment Tool Practice Checklists Child Observation Guides Planning Tools (Child Learning Practices/ Instructional Practices)

48. 48 Training Tools Practice scenarios are used to provide examples of (illustrate) what practices look like when they are implemented in the classroom.

49. 49 Training Tools The Classroom Interest Assessment Tool is used to provide teachers opportunities to practice identifying child interests and selecting interest- based activities.

50. 50 Training Tools Practice Checklists are used to help teachers examine their use of practices compared to the practice standards for each project component.

51. 51 Training Tools Planning Tools are used to engage teachers in planning how they will try out the classroom practices and the additional opportunities that will promote their learning.

52. 52 Training Tools Child Observation Guides are used to help teachers examine the outcomes of their practices and make decisions about how their practices should change.

53. 53 Relationship Among the Two Types of Fidelity and Child/Parent Outcomes Implementation Fidelity Intervention Fidelity Child/Parent Outcomes

54. 54 “Coaching” Characteristics Weekly sessions for 16 weeks Session length: M = 2.5 hours, SD = 1.5, Range =.25- 5.25 In classroom (93%) Trainer introduces new information (72%) Trainer provides illustration (94%) Observation of classroom (95%) Trainee practices specific skill (64%) Trainee reflects on practice (81%) Trainer provides verbal feedback (98%) Goal planning (83%)

55. 55 MeasureConstruct HSTDI Training RecordFrequency, length, and mode of training Teacher Engagement in Adult Learning Process Introduction, illustration, evaluation, mastery, teacher receptiveness—Coach’s assessment Teacher Training Feedback Form Exposure, adherence, responsiveness regarding training process—Teacher’s assessment Teacher Practice Feedback Form Exposure, adherence, responsiveness regarding training on project practices— Teacher’s assessment Implementation Fidelity Project Measures

56. 56 Relationship Between Differences in Implementation Fidelity and Teachers’ Classroom Practices Classroom Practices Implementation Fidelity Cohen’s d LowHigh MeanSDMeanSD Child Learning Practices Interest-Based Child Learning9.222.5910.801.81.71 Child Learning Activities29.006.3432.902.23.82 Increased Learning Opportunities 22.004.3923.902.92.51 Instructional Practices Child Engagement9.442.2410.601.35.63 Teacher Responsiveness20.445.5323.503.75.64 Response Elaboration18.784.6621.803.68.72 CLASS Emotional Support4.881.195.580.87.67 Classroom Organization4.750.685.010.68.32 Instructional Support1.700.732.090.99.45 a Number of subscale items. b One-tailed test.

57. 57 Relationship Between Differences in Training Implementation Fidelity and Follow-Up Observation of Teachers’ Classroom Practices Classroom Practices Implementation Fidelity Cohen’s d LowHigh MeanSDMeanSD Child Learning Practices Interest-Based Child Learning 10.221.7210.001.24-.15 Child Learning Activities 31.892.9332.502.91.21 Increased Learning Opportunities 22.113.6622.303.02.06 Instructional Practices Child Engagement 9.672.1210.101.52.23 Teacher Responsiveness 23.224.7124.803.26.39 Response Elaboration 20.333.6021.503.02.35 CLASS Emotional Support 5.460.975.820.72.42 Classroom Organization 5.160.865.130.96-.03 Instructional Support 1.960.721.870.73-.13

58. 58 What could/should we have done differently as part of our implementation practices? Participant Chat

59. 59 Principles Active participation of the learner in all phases of the training is essential. Individualize training to support learner mastery of the practice. Encourage self-reflection at every phase of the process (Specific checklists can be helpful). Hands-on tools are helpful to support teacher practice. Training must be implemented with fidelity. Implementation fidelity must be measured.

60. 60 Thank you for your participation!

61. 61 References Dunst, C. J., & Trivette, C. M. (2009). Let’s be PALS: An evidence-based approach to professional development. Infants and Young Children, 22, 163-175. Free download at http://dept.washington.edu/isei/iyc/22.3_Dunst.pdfhttp://dept.washington.edu/isei/iyc/22.3_Dunst.pdf Dunst, C. J., Trivette, C. M., & Deal, A. G. (2011). Effects of in-service training on early intervention practitioners’ use of family systems intervention practices in the USA. Professional Development in Education, 37, 181-196. doi:10.1080/19415257/2010.527779. Dunst, C. J., Trivette, C. M., & Hamby, D. W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1), 91-112. Retrieved from https://w5.hkuspcae.hku.hk/journal/index.php/ijcell/article/view/111. https://w5.hkuspcae.hku.hk/journal/index.php/ijcell/article/view/111 Trivette, C. M., Dunst, C. J., Hamby, D. W., & O’Herin, C.E. (2009). Characteristics and consequences of adult learning methods and strategies [Winterberry Research Syntheses, Vol. 2, No. 2]. Asheville, NC: Winterberry Press. Free download at www.wbpress.comwww.wbpress.com Website: www.puckett.org