1 What works in the real world? Evaluating social-emotional programmes, PATHS, and Incredible Years Louise Morpeth &Tracey Bywater Inaugural IEE Conference: Social Research Unit at Dartington

What is ‘real world’ evaluation? Efficacy trials – does it work under optimal conditions? Effectiveness trials – does it work under real world conditions? Dissemination trials – what happens when the programme is ‘taken to scale?

What is PATHS? A school based programme for primary schools age children that facilitates the development of self control, emotional awareness and interpersonal problem solving skills. It seeks to reduce aggression and behaviour problems, and enhance emotional development and the educational process. PATHS lessons of minutes are taught 3- 5 times a week throughout primary school. Detailed lesson plans with teaching instruction are supplemented with materials (feelings cards, puppets etc) Training is provided to all teachers and head teachers. Fidelity is maintained by a PATHS coach. PATHS Promoting Alternative Thinking Strategies

How do Social and Emotional Learning programmes work? From CASEL (

Evaluation questions Does PATHS meet the needs of and improve outcomes for the children that participate? Are programmes implemented efficiently and effectively with fidelity? Do programmes offer value for money and net benefit to the local authority, national government, society and participants?

Evaluation design PATHS (60 schools) cluster randomised trial stratified by size & % FSM 1:1, Reception & Year 1 classes 3 data main collection points (Oct - June 2010, June 2011) A wait-list, intention to treat, randomised controlled trial

PATHS measures  SDQ – child behaviour  Paths Teacher Rating Scale - child behaviour  Teacher background & school climate  Attendance records – teacher & pupils  Academic grades  Observations in class (Oct & April both academic years) Preliminary results show…..

A great deal has been written about looked after children, or children in state care. Much recent legislation A great deal has been written about looked after children, or children in state care. Much recent legislation. 1 million people 30% Black, Minorities, and Ethnic 23% under 16 yrs of age 30% single parents Birmingham London context 62 children’s centers 60,000 professionals working with children and families

A great deal has been written about looked after children, or children in state care. Much recent legislation A great deal has been written about looked after children, or children in state care. Much recent legislation. 1 million people 30% Black, Minorities, and Ethnic 23% under 16 yrs of age 30% single parents Birmingham London context 62 children’s centers 60,000 professionals working with children and families

 No longer pushing PATHS, there is a pull coming from schools  Implementation extended from 27 - >60 schools out of 299  Working on a sustainable costing model (£6000 per school), while maintaining fidelity Next steps Find image of anxiety – nail biting

Parenting Overview 1.What is parenting and what are the benefits? 2.What does a parent programme look like, and what does it do? 3.Do parent programmes work? 4.What are the issues around recruitment and programme implementation? 5.What are the associated costs of the programme? 6.Is there a benefit to link parent programmes to school/early learning environments? 12

1. What is parenting and what are the benefits?... the process of promoting and supporting the physical, emotional, social, and intellectual development of a child from infancy to adulthood. Parenting refers to the activity of raising a child rather than the biological relationship.physical emotionalsocialintellectual developmentinfancyadulthood Davies, Martin (2000). The Blackwell Encyclopaedia of Social Work. Wiley- Blackwell. p ISBN ISBN

14 Parenting factors Harsh and ineffective Poor monitoring Low stimulation Child factors Poor conflict control Impulsivity, ADHD Learning delays Poor social skills Contextual factors Poverty Parental crime, substance abuse or mental illness Life stressors School factors Classroom aggression Deviant peers Poor teacher response Peer rejection Conduct Problems Most influential & easiest to change

15 Sainsbury Centre for Mental Health, 2009 The chance of a lifetime: Preventing early conduct problems and reducing crime

2. What does a parent programme look like, and what does it do? 1 to 1, practitioner & parent DIY, parent is given material and learns at own pace Group format – typically hr weekly sessions, 12 parents Targeted or universal, for specific group or open to everybody Has structured content, with a manual and handouts 16

Components of effective programmes New parenting skills must be modelled and rehearsed Home-based practice or ‘homework’ Parenting programmes should be collaborative & emphasise principles rather than prescribe techniques Non-violent sanctions for negative behaviour Relationship building, praise, fun & play, reinforcement 17

Incredible Years suite of programmes This one has been trialled in Wales, Ireland & England in RCTs

19

3. Do parent programmes work? 1. Wales - Sure Start Trial Delivered by Sure Start staff Children 3-4 years at risk of conduct disorder 2. Ireland - National Evaluation of Incredible Years Delivered by Archways staff in disadvantaged areas Children 5-8 years at risk of conduct disorder 3. England – part of Birmingham’s Brighter Futures Initiative Delivered across the city by children centre and local service staff Children 3-4 years at risk of conduct disorder 20

Measures used in the trials About the family: 1.Demographic questionnaire 2.Service Utilisation Questionnaire (Chisholm et al., 2000) 3.Dyadic Parent-Child Interaction Coding System (DPICS; Eyberg & Robinson, 2000) About the child: 1.Eyberg Child Behavior Inventory (ECBI; Eyberg & Ross, 1978) 2.Strengths & Difficulties Questionnaire (SDQ; Goodman, 1997) 3.Conners Parent Rating Scale (Conners, 1994) 4.Kendall Self Control Rating Scale (SCRS; Kendall & Wilcox, 1979) 5.Social Competence Scale (Fast Track Project) About the parent: 1.Beck Depression Inventory (BDI-II; Beck et al., 1961, 1996) 2.Parenting Stress Index (PSI; Abidin, 1990) 3.EQ-5D Health-Related Quality of Life Questionnaire (Kind et al.,1995) 4.The Parenting Scale (Arnold, O’Leary, Wolff & Acker, 1993) 21

Child Behaviour – parent report (ECBI) 22 Programme delivery between baseline & 6 months

4. What are the issues around recruitment and programme implementation? “ Success is a science: If you have the conditions you get the results” Oscar Wilde ( ) PROCESS EVALUATION: Was engagement, recruitment and retention successful? Is the programme implemented with fidelity in all fidelity categories? 23

Parent engagement 24 Referral methodNumber referrals Eligible (& could be contacted) Participated (from contactable, eligible, consented) Wales (3-4yrs)Health visitors24074%93% Ireland (5- 8yrs) Health service waiting lists, schools, community-based agencies, self-referrals 23364%100% England (3- 4yrs) Initially children centres, then nursery nurse, family support, schools, social workers 299 (early 2010) 17-40% variable across 6 areas Approx 80%

Key Themes linked to participant engagement: 25 1.Organisation readiness 2.Target group & accuracy of demand analysis 3.Recruitment procedures 4.Publicity & referral materials 5.Context of an RCT 6.Staff workload 7.Cultural context

26

Delivery & Implementation Fidelity is the programme delivered as designed? are all the core components present? to the right population? with appropriately trained staff? using the right protocols, techniques & materials? in the right context? 27

28 5. What are the associated costs of the programme? Or - do the programmes offer value for money? What are the delivery costs? Is the programme cost effective? What are the longer term costs & benefits?

29 Micro-costing exercises establish delivery costs Wales & Ireland studies: £ per child for 12 children, (approx. £14,500 including facilitator training, and staff time) Cost-effectiveness Wales & Ireland: £70-80 per point improvement on the ECBI Cost benefits Ireland: Cost and benefit data yields an internal rate of return for the programme of 11% (assuming that individuals with conduct problems spend an additional 5 months in unemployment, and that the savings from reductions in imprisonment occur only once and at the age of 30)

30 6. Is there a benefit to link parent programmes to school/early learning environments? Early Years Foundation Stage Nursery staff trained in IY parent programmes can show less stress, with improved children behaviour Parent programmes delivered in schools - IY school readiness FAST – Families & Schools Together SPOKES - Supporting Parents on Kids Education in Schools (Helping Children Achieve) combines a parent & a literacy programme School based social/emotional programmes e.g. IY Dina increases parent input to child’s schooling

Lessons Learned to inform policy & practice 1.Know what outcomes/change are required 2.Select an evidence-based programme for the target population 3.Develop a strategy for recruiting the target population 4.Address relevant service access issues 5.Staff training and ongoing support 6.Reference/stakeholder groups/steering committee 7.Ensure implementation fidelity (process & delivery) 8.Evaluate programme delivery, costs, and outcomes LT (to be embedded within services) 31

Thank you for your participation