1. Training is Not Enough: Intervention and Evaluation Supplements Christina J. Groark and Robert B. McCall University of Pittsburgh Office of Child Development

2. Overview—Intervention Training alone is a minimally effective intervention Need: −Hands-on technical assistance in situ −Supportive work environment −Continuous monitoring, positive supervision

3. Assessing trainee’s satisfaction and learning is a minimum evaluation Need: −Changes in trainee’s on-the-job behavior −Changes in the behavior/outcomes of clients of the trainees −Analyses to show trainees behavior change mediated the intervention's effect on clients’ behavior/outcomes Overview—Evaluation

4. Intervention Training alone is minimally effective Why? Trainees often do not change their behavior Need: On-the-ground technical assistance Supportive work environment Monitoring and positive supervision

5. Intervention On-the-ground technical assistance Observe (videotape) client on job Constructive, positive feedback Modeling with client on job Daily review

6. Intervention

7. Work environment Supports behaviors taught in training Hand washing—need a sink, waterless soap handy Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers

8. Chaotic Child Environment

9. Orderly Child Environment 9

10. Monitoring and positive supervision Sets expectations, standards, reminds, rewards Continuing system, supervisor, staff meetings Hand washing Early care and education—teachable moments, responsiveness to child- directed initiatives

11. Illustrative Example I Latin American Orphanage Intervention (birth-6 years) Training—sensitive, responsive interactions ◦6 one-day training sessions On-the-ground technical assistance ◦12 days more than planned because caregivers were not implementing

12. BUT – Limited environmental change ◦4 primary caregivers per ward ◦8-10 children per caregiver No continuing supervision

14. Mean Total ITERS/ECERS 1 5 7 Intervention Pre- Post- Results—Caregiving behavior/environment (ITERS/ECERS) 2 3 4 Younger Older 6 Typical USA Early Care

15. Battelle 60 70 80 Intervention Pre- Post- Results—Children’s development (Battelle) Younger Older 90 100 Typical Parent-Reared

16. Conclusion Needed hands-on technical assistance to promote implementation Limited supportive work environment No supervisory system Some improvement in caregivers, children, but NOT MUCH

17. Illustrative Example II St. Petersburg (Russia) Orphanage Intervention More intensive, comprehensive All major components

18. Minimum warm, sensitive, contingently- responsive caregiver-child interactions Perfunctory, business-like; little talking Caregiver directed Conformity 10/10/201518 St. Petersburg-USA Orphanage Research Team, 2005, 2008

20. 12-14 children/ward 9-12 caregivers/week Homogeneous age, disability groups Periodic graduations 60-100 caregivers before age 19 months 10/10/201520 St. Petersburg-USA Orphanage Research Team, 2005, 2008

21. Intervention Training ◦ Train Trainers ◦ 12 sessions, 2-3 hrs. ◦ Warm, sensitive, responsive ◦ “Love these kids…” Hands-on technical assistance

23. Intervention—Supportive Work Environment Reduced group size to 6-7 Assigned primary, secondary caregivers—6 vs 9 cgrs. Changed work schedules Integrated groups by age, disabilities Eliminated periodic graduations to new groups Family hour

24. Intervention—Monitoring and Positive Supervision Training for supervisors Trainer observed, coached both caregivers and supervisors Supervisors met periodically to problem solve, case reviews, mutual support

25. Three orphanages (Baby Homes), birth – 4 years T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision TO—Training Only NoI—No Intervention Evaluation

26. Evaluation—Trainee Learning

27. Longitudinal Sample Evaluation—Improved Caregiving T+SC

28. Evaluation – Children’s Development 40 50 60 70 80 90 First 4-9 mos 9+ mos Time in Intervention Developmental Quotient T+SC TO NoI 100 Typical Parent Reared

29. Evaluation--Mediation Analysis Intervention improved caregiving Intervention improved children’s development But was children’s developmental improvement associated with improved caregiving?

30. Evaluation—Mediation Analysis

31. Evaluation--Mediation Analysis Independent Variable T+SC vs NoI Intervention Outcome Battelle Total Effect Ĉ = 14.14(2.07)*** Mediator HOME Independent Variable T+SC vs NoI Intervention Outcome Battelle Effect of Mediator on Outcome Controlling for Indep. Variable Direct effect Controlling for Mediator Ĉ = 8.00(2.87)*** Mediator accounts for 43% of Total Effect Effect of Indep. Var. on Mediator  = 3.41(.22)***  = 1.80 (.59)**

32. Intervention ◦ Training alone is often minimally effective ◦Has greater effect if accompanied by  On-the-ground technical assistance  Supportive work environment  Monitoring and supervision Conclusion

33. Evaluation is improved if ◦Some training vs. No Training comparison ◦Measures of –Learning –Work-place behavior change –Outcome behavior of clients ◦Mediation analysis Conclusion

34. Evaluation Mediation Analysis: MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2), 144-158.