1. Keeping Fidelity in an Evidence-Based Program: A University-Community Agency Research Partnership Susan B. Stern Factor-Inwentash Faculty of Social Work Factor-Inwentash Faculty of Social Work University of Toronto Katina Watson Katina Watson YOUTHLINK YOUTHLINK Child Development Institute Child Development Institute Funded by SSHRC Institutional Grant, Royal Bank Fellowship

2. Overview Context Context Evidence-Based Parenting Program Evidence-Based Parenting Program IY Early Learning Centres in Toronto IY Early Learning Centres in Toronto Treatment fidelity - transporting evidence-based interventions to community practice Treatment fidelity - transporting evidence-based interventions to community practice Understand barriers/facilitators Understand barriers/facilitators Study implementation process as it unfolds Study implementation process as it unfolds

3. Why is Fidelity Important?

4. WHY EFFECTIVE PARENTING PROGRAMS ARE IMPORTANT In the absence of treatment, child conduct problems typically intensify In the absence of treatment, child conduct problems typically intensify The majority of chronic behaviour problems in school age children had an early onset The majority of chronic behaviour problems in school age children had an early onset If the difficulties continue after entry into school, the risk of peer rejection, later substance abuse and delinquency increases. If the difficulties continue after entry into school, the risk of peer rejection, later substance abuse and delinquency increases. Parent-child interaction key risk factor for early onset conduct problems (esp. harsh/inconsistent discipline, low nurturance). Parent-child interaction key risk factor for early onset conduct problems (esp. harsh/inconsistent discipline, low nurturance). Parent training is the most empirically supported intervention strategy for decreasing problem behaviours. Parent training is the most empirically supported intervention strategy for decreasing problem behaviours.

5. Incredible Years Model Emphasizes building positive parent-child interactions and breaking coercive cycles Emphasizes building positive parent-child interactions and breaking coercive cycles Increase parent self-efficacy/coping, and problem-solving skills Increase parent self-efficacy/coping, and problem-solving skills Increase parents positive communication skills and reduce anger and the use of criticism and unnecessary commands. Increase parents positive communication skills and reduce anger and the use of criticism and unnecessary commands. Strengths-based and goal focused Strengths-based and goal focused Learning through modeling and practice Learning through modeling and practice Action-oriented with home activities Action-oriented with home activities Creates support networks and opportunities to share a range of responses and experiences Creates support networks and opportunities to share a range of responses and experiences Flexible and collaborative, tailored to each childs temperament and family situation. Flexible and collaborative, tailored to each childs temperament and family situation. Focus on key principles behind strategies Focus on key principles behind strategies Led by skilled and knowledgeable leaders Led by skilled and knowledgeable leaders

6. Expert Model Collaborative Model Leader in charge, dispenses advice >Parents are active and invested Leader in charge, dispenses advice >Parents are active and invested Parents passive recipients of service >Highlight strength and change Parents passive recipients of service >Highlight strength and change Discusses problems and deficits > Reciprocal working relationship Discusses problems and deficits > Reciprocal working relationship Can increase guilt, self blame >Non-blaming, non-hierarchical Can increase guilt, self blame >Non-blaming, non-hierarchical Diagnoses problems, attempts > Mutual problem-solving, Diagnoses problems, attempts > Mutual problem-solving, to impose a general solution increases self-efficacy to impose a general solution increases self-efficacy >Can increase resistance > Climate of trust and openness

7. Incredible Years 6 randomized control group studies -Webster-Stratton 6 randomized control group studies -Webster-Stratton 5 independent replications 5 independent replications SAMHSA Model Strengthening Families Program SAMHSA Model Strengthening Families Program OJJDP Blueprints for Violence Prevention OJJDP Blueprints for Violence Prevention UK Home Office recommended EBP for antisocial children UK Home Office recommended EBP for antisocial children Clinic studies Clinic studies Clinically significant improvement in positive parenting/child behavior Clinically significant improvement in positive parenting/child behavior Head Start - universal prevention with disadvantaged high risk families Head Start - universal prevention with disadvantaged high risk families Significant differences in parenting practices and child behavior/affect Significant differences in parenting practices and child behavior/affect Maintenance at one year follow ups high Maintenance at one year follow ups high BASIC Preschool and School Age, 3-10 years

8. Research (contd) Parent Training group shown to be as effective as individual (family) training with high risk parents whose children had conduct problems. 70% maintained improvements at one year. Parent Training group shown to be as effective as individual (family) training with high risk parents whose children had conduct problems. 70% maintained improvements at one year. Many families stated preference for group over individual family treatment. Many families stated preference for group over individual family treatment. Children with high levels of baseline conduct problems and mothers with high levels of critical parenting benefited the most. Children with high levels of baseline conduct problems and mothers with high levels of critical parenting benefited the most. Participation and engagement rates high as many practical barriers are reduced Participation and engagement rates high as many practical barriers are reduced.

9. IY Key Dissemination Strategies Three day intensive training workshop for group leaders Three day intensive training workshop for group leaders Clear, detailed manual with session by session checklists outlining key principles Clear, detailed manual with session by session checklists outlining key principles Series of video taped vignettes illustrating parent-child interactions Series of video taped vignettes illustrating parent-child interactions Weekly home activities and summary lists of key concepts for parents Weekly home activities and summary lists of key concepts for parents Comprehensive book Comprehensive book Series of integrity checklists and leader evaluations Series of integrity checklists and leader evaluations

10. Research Foci Questions Barriers to adherence Barriers to adherence Reasons for departures Reasons for departures What facilitates adherence to IY protocol What facilitates adherence to IY protocol

11. Program Context 3 year project 3 year project Year 1-3 data Year 1-3 data 29 groups 29 groups 18 Group leaders 18 Group leaders MSW or Post-secondary education MSW or Post-secondary education Range of 4-20 years experience (M=10 yrs.) Range of 4-20 years experience (M=10 yrs.)

12. Parent Demographics Round 3 and 7 93% mothers, 7% both parents (n=44) 93% mothers, 7% both parents (n=44) Child age range, 6 mo - 15 yrs (n=43)* Child age range, 6 mo - 15 yrs (n=43)* 48% <=3yrs. 48% <=3yrs. 39% 4-7 yrs 39% 4-7 yrs 7% 8-11 yrs. 7% 8-11 yrs. Ethnicity (n=41) * Ethnicity (n=41) * 41% White 41% White 18% Black 18% Black 18% South Asian 18% South Asian 4.5 % Aboriginal, 4.5% Filipino, 2.3% Latin American, 4.5% Other 4.5 % Aboriginal, 4.5% Filipino, 2.3% Latin American, 4.5% Other 59% English as first language (n=43) 59% English as first language (n=43)

13. Parent Demographics (cont.) Income (n=41)* Income (n=41)* 34.1% < $15,000 34.1% < $15,000 18.2% $15,000 – $24,999 18.2% $15,000 – $24,999 40.9% >= $25,000 40.9% >= $25,000 50 % on social assistance (n=43) 50 % on social assistance (n=43) 71% high school completed/GED or higher (n=44) 71% high school completed/GED or higher (n=44)

14. IY Fidelity Measures Leader Session x Session Checklist Leader Session x Session Checklist Group Leaders Written Narratives Group Leaders Written Narratives Leader Rating Scale (Program Elements & Collaborative Process) Leader Rating Scale (Program Elements & Collaborative Process) Independent Supervisor Written Observations Independent Supervisor Written Observations Audiotapes of group leaders joint meeting Audiotapes of group leaders joint meeting

15. Group Leaders Focus Group To understand program implementation and adherence issues over the course of multiple IYPP groups To understand program implementation and adherence issues over the course of multiple IYPP groups Semi-structured by interview guide Semi-structured by interview guide Two focus group facilitators Two focus group facilitators Trained by one of the lead researchers Trained by one of the lead researchers

16. Focus Group Questions What barriers did you as group leaders face or perceive the parents to experience in implementing the IYPP. What barriers did you as group leaders face or perceive the parents to experience in implementing the IYPP. What facilitated implementing the program and/or overcoming barriers? What facilitated implementing the program and/or overcoming barriers? Given your perceptions of the parents, what else would have helped in the implementation of the program? Given your perceptions of the parents, what else would have helped in the implementation of the program? What do you think were the most effective intervention components and processes? What do you think were the most effective intervention components and processes? If IY was to be implemented here or in another organization, what modifications do you think the program needs, or parents have suggested, to improve the fit with a diverse and universal prevention population? If IY was to be implemented here or in another organization, what modifications do you think the program needs, or parents have suggested, to improve the fit with a diverse and universal prevention population? What organizational factors or other supports do you think would be necessary for program sustainability? What organizational factors or other supports do you think would be necessary for program sustainability?

17. Trustworthiness of the Data Dependability of the data through audio-taping and transcriptions Dependability of the data through audio-taping and transcriptions Open and axial coding for thematic analysis Open and axial coding for thematic analysis Credibility through persistent observation with participants (e.g. group meetings, observations of sessions) Credibility through persistent observation with participants (e.g. group meetings, observations of sessions) Prolonged engagement Prolonged engagement Negative case analysis Negative case analysis Multiple independent coders Multiple independent coders Member checking Member checking Confirmability through quotes Confirmability through quotes

18. IY Protocol Adherence All Rounds Session x Session Checklist Session x Session Checklist Overall adherence DID I Grand mean 83.88 % Overall adherence DID I Grand mean 83.88 % Role play Grand mean 59.94 % Role play Grand mean 59.94 % Buddy calls Grand mean 52.77 % Buddy calls Grand mean 52.77 % Vignette Grand mean 63.51% Vignette Grand mean 63.51% Group Leader Rating Scale Group Leader Rating Scale Overall adherence across sites 4.48 Overall adherence across sites 4.48 Leader group process skills 4.69 Leader group process skills 4.69 Showing vignettes 4.45 Showing vignettes 4.45 Practice and role plays 3.8 Practice and role plays 3.8 4 th session observation 3.83 4 th session observation 3.83 8 th session observation 3.71 8 th session observation 3.71

19. IY Protocol Adherence Contd Vignettes: Vignettes: Videotape modeling with core vignettes occurred in at least 6 (55%) and often all 11 (100%) sessions that called for vignettes (M= 9.7 or 88%). Role Plays: Role Plays: Role Plays occurred in as few as two (20%) but sometimes all ten (100%) sessions that called for at least one role play (M = 6.34 or 63%)

20. Negative Cases: Program Adherence Buddy calls Buddy calls Vignettes Vignettes Role plays Role plays

21. Balancing Parent Preferences with Structured Program Components Collaborative vs. Expert role Collaborative vs. Expert role Parent discussion preference Parent discussion preference Desire to keep parental involvement high Desire to keep parental involvement high Discomfort with new, unfamiliar (strange) activities & skills (parents & leaders) Discomfort with new, unfamiliar (strange) activities & skills (parents & leaders) Cultural preference for learning style e.g., lecture, education Cultural preference for learning style e.g., lecture, education

22. Balancing parent preferences with structured program components It is very hard to cover the agenda because of the large group and there is a lot of discussion. Often that gets into a pretty large story. Evaluations show that parents enjoyed group discussions and did not find role-play as helpful. Less vignettes were shown to make time for more discussion.

23. Balancing Parent Needs within the Groups Various educational levels Various educational levels Mixed socioeconomic levels Mixed socioeconomic levels Various learning styles (lecture, culture) Various learning styles (lecture, culture) Range of child ages Range of child ages Different perceived parent needs re: relevance of IY strategies (e.g., -need for play, time-out) Different perceived parent needs re: relevance of IY strategies (e.g., -need for play, time-out)

24. Parent Needs There were people with different levels of education and sometimes with some people it was pretty basic and others...so it was sometimes difficult to maintain a level where they dont get bored because you are repeating. There were people with different levels of education and sometimes with some people it was pretty basic and others...so it was sometimes difficult to maintain a level where they dont get bored because you are repeating. Having too much of a wide spectrum of parents with children from literally 6 months old to twelve years old in a group that was specifically designed for a specific age group was a bit challenging for me to do. Having too much of a wide spectrum of parents with children from literally 6 months old to twelve years old in a group that was specifically designed for a specific age group was a bit challenging for me to do.

25. Time Constraints Leader workload/preparation time Leader workload/preparation time Amount of content Amount of content Balance content/structured activity time with parent talk time Balance content/structured activity time with parent talk time Balancing individual parent needs with collective needs Balancing individual parent needs with collective needs Time for reviewing & setting up homework Time for reviewing & setting up homework Parent tardiness (time constraints in their own lives to get to group) Parent tardiness (time constraints in their own lives to get to group)

26. Language Issues Language (ESL) Language (ESL) Time to express self in 2 nd language Time to express self in 2 nd language Translation time Translation time Difficulties in understanding vignettes, written materials Difficulties in understanding vignettes, written materials Lingo Lingo Homework difficulties – reading/writing Homework difficulties – reading/writing ESL issues took precedence over cultural differences. ESL issues took precedence over cultural differences.

27. Language (ESL) Sometimes we have to be patient, as people are struggling with language and having trouble explaining themselves. (relates to issue of not enough time) We were worried that we have so much content to begin with this is going to take forever with two interpreters Some parents had difficulties understanding the video-tapes because of language. Parents who had difficulty with English could not do the written assignments.

28. Culture Spanking discussion was excellent because it became a discussion about cultural values, how people were raised. Spanking discussion was excellent because it became a discussion about cultural values, how people were raised. In many cultures its different, and I think the tendency is to get to the role of education In many cultures its different, and I think the tendency is to get to the role of education I find people from traditional cultures learn and push us to be more educational I find people from traditional cultures learn and push us to be more educational

29. Relevance Buddy calls Buddy calls Vignettes Vignettes Outdated Outdated Most focus on one child without siblings Most focus on one child without siblings Few examples of single parents, father only in some topics Few examples of single parents, father only in some topics Many parents had younger children-need to address attachment, developmental issues Many parents had younger children-need to address attachment, developmental issues Did not cover some common concerns, e.g., bullying, sibling conflict Did not cover some common concerns, e.g., bullying, sibling conflict

30. Relevance The vignettes do not match the teaching points of the topics. For example it [time out] should be used as a last resort yet vignettes show minor things to be timed out (e.g. not hanging up coat). Vignettes were an issue this week as in some sections there were not mother-child examples and some mothers had more trouble identifying with the father-child vignettes... Other sections were difficult because they show two-parent families. Some seem redundant, same topic. Although parents comment they enjoy them but they are dated and some seem staged.

31. Role Play Reluctance Anxiety Anxiety Self-consciousness Self-consciousness Parents did not think of or experience them as useful Parents did not think of or experience them as useful Childish Childish Prefer discussion Prefer discussion Lack of time Lack of time Leaders own discomfort Leaders own discomfort

32. Role Play Discomfort/Reluctance... We werent so great at role plays and neither were the parents... They preferred to have discussion.... We werent so great at role plays and neither were the parents... They preferred to have discussion. Because of discussion time didnt have time for role plays. We have more trouble fitting in the role plays because they are time consuming.

33. Concerns with Time out Confusion between home use and child care policy Confusion between home use and child care policy Does not work Does not work Values and cultures Values and cultures Safety issues Safety issues Inappropriate, not relevant for their family Inappropriate, not relevant for their family Child temperament Child temperament Parents own style Parents own style Concerns about abandonment, letting kids scream Concerns about abandonment, letting kids scream

34. Structure of IY materials, manual Hard to follow at times – not user friendly Hard to follow at times – not user friendly Mismatch Mismatch Manual instructions to use Time out for serious behaviors/ vignettes show minor behavior (e.g., not hanging up coat) Manual instructions to use Time out for serious behaviors/ vignettes show minor behavior (e.g., not hanging up coat) One session had 2 homework assignments One session had 2 homework assignments Mixing 2 topics in one session Mixing 2 topics in one session

35. Disconnect Between IY procedures for selected/ indicated population and universal preventive one Parent reactions to specific IYPP strategies (e.g., depth of teaching time-out, buddy calls) Parent reactions to specific IYPP strategies (e.g., depth of teaching time-out, buddy calls) Group leader discomfort/boundaries teaching vs. clinician role (e.g., follow up calls) Group leader discomfort/boundaries teaching vs. clinician role (e.g., follow up calls) I think because the program was developed for high risk situations and they did put in all of these things to help and maybe it isnt appropriate for a parenting group

36. Organizational Barriers Communication Communication Commitment to the program Commitment to the program Outreach to parents Outreach to parents Clear expectations between child care centres and group leaders Clear expectations between child care centres and group leaders

37. Organizational Barriers Its a three-way relationship, you have us, the parents, and the childcare centre and all parts of it have to be strong in order for it to work. I think that if there are expectations and boundaries about what the childcare centre is supposed to do and what we are supposed to do, we need that, for some groups that wasnt very clear. Its challenging and goes back to the buy-in from the centre and I think we have to screen very carefully who were going to be doing [the program] with.

38. Barriers to Protocol Adherence Balancing Parent Preferences with Structured Program Components Balancing Parent Preferences with Structured Program Components Time Constraints Time Constraints Language (ESL) Language (ESL) Relevance Relevance Role Play Reluctance Role Play Reluctance Confusion/Concerns with Time Out Confusion/Concerns with Time Out Structure of the Materials and Manual Structure of the Materials and Manual Disconnect between High Risk/Clinical and Universal Preventive IY Groups Disconnect between High Risk/Clinical and Universal Preventive IY Groups Lack of Control Lack of Control Cultural Cultural Balancing Parent Needs Balancing Parent Needs Organizational Organizational

39. Facilitators/Strategies to Overcome Barriers Highlighting and reinforcing fidelity Highlighting and reinforcing fidelity Validating Validating Provides rationales [with group leaders & parents] Provides rationales [with group leaders & parents] Collaboration Collaboration Problem solving barriers Problem solving barriers Focusing on goal Focusing on goal

40. Highlighting and Reinforcing Fidelity It is (role plays) for many intimidating. I remember in training lots of us hesitated. But we also found out we needed them because some skills needed practicing and it was hard to remember everything. Reiterating the importance of vignettes in general - (research) shows they are helpful even if parents dont like them.Reiterating the importance of vignettes in general - (research) shows they are helpful even if parents dont like them.

41. Facilitators of Adherence Supervisor responsiveness, facilitator of organizational support, concrete resources Supervisor responsiveness, facilitator of organizational support, concrete resources 1 st session preparation 1 st session preparation Elicits ongoing feedback Elicits ongoing feedback Follow through Follow through Mediator between IY group leader and the CC centre Mediator between IY group leader and the CC centre Definitely [supervisor] makes it easier for us What are your thoughts and ideas about what would be helpful right now? Is there anything more I could be doing? IY structure, materials (manual, vignettes, book) IY structure, materials (manual, vignettes, book)

42. Monitoring Fidelity and Problem Solving Barriers Accountability for monitoring/commitment Accountability for monitoring/commitment Understanding reasons for departures Understanding reasons for departures Ensuring programs responsiveness to parent needs Ensuring programs responsiveness to parent needs Tenor of problem solving collaboratively Tenor of problem solving collaboratively Weekly evaluations [parent] and follow up calls give us a way to make sure we are being responsive to parents, and our own leader checklists will give us feedback about what we are and arent doing. Please note which [vignettes] you used on leader checklist and why if skipped or added vignettes. Lets make notes of barriers to doing home activities which are core to the group.

43. Monitoring Fidelity/PS Barriers When I observe groups, it will be in the same spirit, to learn and understand the needs of each group and how and why the group leaders have made any modifications to meet the needs of their particular parents Our motto always was, that any departure was OK, they could trust their judgment or suggest something, as long as they had a rationale or a clinical reason for it. Attention to adherence & procedures embedded in agency, feasibility for adherence monitoring in social work community settings

44. Group Leader Fit & Belief in IYPP Core Components & Collaborative Process You might find that people self select, cause I agree with you, when I came to it, I really liked the material, and agreed with most of it – with the foundation, so I choose to continue doing it, where others might not like it, find it to rigid, so you find people that will adhere to it because they like it. I think you do things that you believe in, and I think you do things that you believe in, and most of us are philosophically similar.

45. Key Contributors to Adherence Overall, adherence was good with the exception of role plays, # of vignettes shown, & buddy calls Overall, adherence was good with the exception of role plays, # of vignettes shown, & buddy calls Therapists belief in the importance of the programs core components and collaborative process Therapists belief in the importance of the programs core components and collaborative process Emphasis on accountability and ongoing session monitoring Emphasis on accountability and ongoing session monitoring Therapists sense of ownership and responsibility for quality control Therapists sense of ownership and responsibility for quality control Supervisor understanding of IY and underlying research, commitment to high quality implementation Supervisor understanding of IY and underlying research, commitment to high quality implementation Supervisor commitment to supporting group leaders and follow- through Supervisor commitment to supporting group leaders and follow- through Training/structure of program and materials Training/structure of program and materials Concrete resources Concrete resources

46. Limitations Fidelity data either self report or supervisor observation – possibility of systematic bias Fidelity data either self report or supervisor observation – possibility of systematic bias Supervisors dual role – potential conflict of interest Supervisors dual role – potential conflict of interest Validation of quantitative adherence measures Validation of quantitative adherence measures Triangulation of findings with clear and consistent patterns across different methods & data sources, and stability over time of seven IYPP cycles, increases confidence in the results.

47. Next Steps

48. Upcoming Article Stern, S.B., Alaggia, R., Watson, K. & Morton, T.R. (in press). Implementing an Evidence-Based Parenting Program With Adherence in the Real World of Community Practice. Research on Social Work Practice. Stern, S.B., Alaggia, R., Watson, K. & Morton, T.R. (in press). Implementing an Evidence-Based Parenting Program With Adherence in the Real World of Community Practice. Research on Social Work Practice.