1. The Importance of Treatment Fidelity in Delivering Parent Programmes Presentation for the Incredible Years Wales Conference Llandrindod Wells, 7th February 2008 Catrin Eames Bangor University c.eames@bangor.ac.uk Catrin Eames Bangor University c.eames@bangor.ac.uk Supervisory committee: Dr Judy Hutchings, Dr Carl Hughes, Dr David Daley Funded in collaboration by ESF/Objective 1 fund and Incredible Years Wales

2. Overview of presentation The importance of treatment fidelity History of treatment fidelity research Current Ph.D. research The importance of treatment fidelity History of treatment fidelity research Current Ph.D. research

3. What is fidelity? Fidelity may be referred to as the degree to which intervention delivery adheres to the original intervention protocol (IOM, 2001).

4. Fidelity research terminology Adherence Treatment delivery Treatment differentiation Treatment exposure Treatment integrity Participant responsiveness Adherence Treatment delivery Treatment differentiation Treatment exposure Treatment integrity Participant responsiveness

5. Effective delivery Adequate evidence base Methods used Means of assessing fidelity Adequate evidence base Methods used Means of assessing fidelity

6. History of treatment fidelity research Poor methodologies Surge of treatment manuals Manuals alone do not guarantee fidelity Few studies reported any fidelity issues Of those that did, no reference to the development of valid fidelity measures Poor methodologies Surge of treatment manuals Manuals alone do not guarantee fidelity Few studies reported any fidelity issues Of those that did, no reference to the development of valid fidelity measures

7. The importance of fidelity It is necessary to implement all components of a programme fully in order to preserve the behaviour change mechanisms that made the original programme model a successful one (Arthur & Blitz, 2000; Mihalic et al., 2002) Can offer a template of service delivery Further resources/supervision/training Ensure adherence to delivery of intervention models Programme drift most common reason for poor treatment outcomes (Mills & Ragan, 2000; Mowbray et al, 2003) It is necessary to implement all components of a programme fully in order to preserve the behaviour change mechanisms that made the original programme model a successful one (Arthur & Blitz, 2000; Mihalic et al., 2002) Can offer a template of service delivery Further resources/supervision/training Ensure adherence to delivery of intervention models Programme drift most common reason for poor treatment outcomes (Mills & Ragan, 2000; Mowbray et al, 2003)

8. Current fidelity measures Typically, manuals or checklist format Measure programme content Threats to validity Typically, manuals or checklist format Measure programme content Threats to validity

9. Benefits of observation Precise account of behaviour as it unfolds Bias reduced with independent observation Continuous sampling, quantitative Precise account of behaviour as it unfolds Bias reduced with independent observation Continuous sampling, quantitative

10. Rationale for current research IY : Internationally delivered and evaluated Over 10,000 trained in the BASIC Numbers of independent replications increasing Quantitative observational measure of process skills Ensure measure is reliable and valid Relate leader fidelity scores to changes in parent and child behaviour from the Sure Start Study IY : Internationally delivered and evaluated Over 10,000 trained in the BASIC Numbers of independent replications increasing Quantitative observational measure of process skills Ensure measure is reliable and valid Relate leader fidelity scores to changes in parent and child behaviour from the Sure Start Study

11. Leader Observation Tool (LOT) Objective frequency count Continuous time sampling at 10-minute intervals throughout the 2-hour session Both leaders coded 18 behaviour categories forming five skills subgroups: Listening Empathy Physical Encouragement Positive Behaviour Other/Negative Objective frequency count Continuous time sampling at 10-minute intervals throughout the 2-hour session Both leaders coded 18 behaviour categories forming five skills subgroups: Listening Empathy Physical Encouragement Positive Behaviour Other/Negative

13. Data set 12 groups, two leaders per group, 2 hour sessions Course split up into : Play and relationship building Praise and reward Effective limit setting Handling misbehaviour Four sessions selected for coding for each group Of these, 30% randomly selected for second coding Coders blind to behaviour change scores to avoid bias 12 groups, two leaders per group, 2 hour sessions Course split up into : Play and relationship building Praise and reward Effective limit setting Handling misbehaviour Four sessions selected for coding for each group Of these, 30% randomly selected for second coding Coders blind to behaviour change scores to avoid bias

14. LOT reliability and validity Internal reliability at both p<.05 and p<.01 84% average inter-rater agreement Intra-class correlations of each category: mean of.92 and.95 for code-recode and inter-rater reliability respectively Concurrent validity with parent and leader reports of the group intervention process LOT identified variability between groups, highlighting the heterogeneity in process skills and treatment fidelity Internal reliability at both p<.05 and p<.01 84% average inter-rater agreement Intra-class correlations of each category: mean of.92 and.95 for code-recode and inter-rater reliability respectively Concurrent validity with parent and leader reports of the group intervention process LOT identified variability between groups, highlighting the heterogeneity in process skills and treatment fidelity

15. LOT scores and behaviour change Leaders attributed total score per subgroup Parent data = 86 completers Each parent attributed a leader fidelity exposure score Predict that probability of parent behaviour change greater when attending a group delivered with high fidelity Greater change in parent behaviour in turn predict greater change in child behaviour Leaders attributed total score per subgroup Parent data = 86 completers Each parent attributed a leader fidelity exposure score Predict that probability of parent behaviour change greater when attending a group delivered with high fidelity Greater change in parent behaviour in turn predict greater change in child behaviour

16. Measures Leader Parent Child Leader: LOT Parent: Parenting Scale, Positive Parenting Child: ECBI, Child positives, Compliance Leader Parent Child Leader: LOT Parent: Parenting Scale, Positive Parenting Child: ECBI, Child positives, Compliance

17. Results Leader ParentParent Child Leader Positive Behaviour Positive Parenting (ß =.28, p<.01; R 2 =.08) Change in Positive Parenting Child Positive (ß =.60, p<.01; R 2 =.36) Change in Positive Parenting Compliance to Direct Commands (ß =.23, p<.05; R 2 =.05) Leader Physical Encouragement Parenting Scale (ß =.26, p<.01; R 2 =.07). Change in Parenting Scale ECBI-I (ß =.23, p<.05; R 2 =.05) Change in Parenting Scale ECBI-P ( ß =.34, p<.01; R 2 =.12)

18. Therefore…. In order to change child behaviour, need to change parenting behaviour Leader treatment fidelity of process skills significantly predict both observed change in parenting behaviour as well as parent self reported behaviour In turn, predict the change in child behaviour In order to change child behaviour, need to change parenting behaviour Leader treatment fidelity of process skills significantly predict both observed change in parenting behaviour as well as parent self reported behaviour In turn, predict the change in child behaviour

19. Conclusion Need to report fidelity criteria in treatment outcome studies, especially considering the number of manual based interventions that are replicated in the community today Poor implementation fidelity is the most common factor for poor treatment outcome (Mills & Ragan, 2000) Valid fidelity measurement can predict treatment outcome, as well as document treatment differentiation in multi-centre research Need to report fidelity criteria in treatment outcome studies, especially considering the number of manual based interventions that are replicated in the community today Poor implementation fidelity is the most common factor for poor treatment outcome (Mills & Ragan, 2000) Valid fidelity measurement can predict treatment outcome, as well as document treatment differentiation in multi-centre research

20. Thank you for listening For further information please contact: Catrin Eames School of Psychology College of Health and Behavioural Sciences Bangor University Gwynedd LL57 2DG 01248 38 2651 c.eames@bangor.ac.uk