1. + Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle (UCD School of Economics & UCD Geary Institute) Conference on Irish Economic Policy 1 st February 2013

2. + Importance of Early Childhood Investment Targeted early intervention programmes effective way of reducing socio- economic inequalities in childrens skills Well-designed interventions can generate a return to society ranging from $1.80 to $17.07 for each dollar spent YET primarily US-based evidence only on the effectiveness & cost- effectiveness of such interventions Argument for early intervention in more generous welfare systems is not rooted in strong empirical evidence

3. + Irish Context Prevention & Early Intervention Programme (The Atlantic Philanthropies) & Government, DCYA) initiated in 2004 Aim: to support the development, implementation and evaluation of strategies for disadvantaged children All programmes evaluated using experimental or quasi-experimental design Little tradition of social experimental evaluation in Ireland Comparing the effectiveness of multiple programmes will enhanced resource allocation and policy design

4. + Preparing for Life Programme Community-led initiative: operated by Northside Partnership in highly disadvantaged area of Dublin, Ireland Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives Evidence of need: Children scored below the norm on cognitive & non- cognitive skills on school entry (Doyle & McNamara, 2011) Aim: Improve levels of school readiness by assisting parents in developing skills to prepare their children for school Evaluation Design: RCT & embedded quasi-experimental design

5. + Design of Preparing for Life

6. + PFL Evaluation Impact Evaluation Aim: To determine if the programme/service is effective PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths, 24mths, 3yrs, 4yrs, school entry Implementation Evaluation Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity to the model PFL data collection: 1. Implementation data 2. Focus groups with participants 3. Semi-structured interviews with programme staff

7. + Recruitment Eligibility Criteria: Cohort of pregnant women residing in PFL catchment area between Jan 2008-August 2010 (32 months) Includes nulliparous and multiparous women Recruitment: Maternity hospital at first booking visit (b/w 12-26 weeks) Within the local community Population-based recruitment rate, based on all live births during the recruitment phase, was 52%

8. + Randomisation Procedure Individual-level randomisation Unconditional probability randomisation strategy Design procedure that preserves the integrity of randomization ie. no opportunity for recruiter to intentionally influence assignment Randomisation process After informed consent obtained, participant pressed key on tablet laptop to allocate her assignment condition Evaluation & Implementation team automatically received an email with the assigned ID number and treatment condition Thus no opportunity to compromise the randomisation process

9. + Baseline Analysis Determine the effectiveness of the randomisation procedure Baseline assessment conducted post randomisation, pre intervention Randomisation worked! PFL Baseline Assessment Proportion of Measures Not Significantly Different at Baseline TopicPFL Low – PFL HighPFL – Comparison group Family Socio-demographics33/33 (0%)27/33 (82%) Maternal Well-being24/24 (0%)18/24 (75%) Maternal Health & Pregnancy35/35 (0%)26/35 (74%) Parenting & Childcare10/13 (74%)6/13 (46%) Social Support17/18 (94%)9/9 (0%) Total NOT Statistically Different119/123 (97%)86/114 (75%)

10. + Baseline Differences in Selected Socio- Demographics Low Treatment – High Treatment Instrument N (n LOW / n HIGH) M LOW (SD) M HIGH (SD) p Effect Size (d) Mothers Age 205 (101/104) 25.30 (5.99) 25.46 (5.85) ns.03 First-time Mothers 205 (101/104) 0.50 (0.50) 0.54 (0.50) ns.09 Mother Married 205 (101/104) 0.18 (0.38) 0.14 (0.35) ns.09 Mothers with Junior Certificate Qualification or Lower 205 (101/104) 0.40 (0.49) 0.34 (0.47) ns.12 Mothers with Primary Degree 205 (101/104) 0.03 (0.17) 0.03 (0.17) ns.01 Mothers Unemployed 205 (101/104) 0.41 (0.49) 0.43 (0.50) ns.05 Fathers Unemployed 198 (97/101) 0.31 (0.46) 0.43 (0.50) ns.24 Residing in Social Housing 204 (101/103) 0.55 (0.50) 0.55 (0.50) ns.00 In Possession of a Medical Card 205 (101/104) 0.66 (0.47) 0.60 (0.49) ns.14

11. + Methodology Estimate treatment effects at birth, 6 months, 12 months Permutation based hypothesis testing (Heckman et al. 2010) Classical hypothesis tests unreliable when the sample size is small and the data are not normally distributed Permutation test are distribution free, thus suitable in small samples Based on the assumption of exchangeability between treatment conditions under the null hypothesis Stepdown procedure (Romano & Wolf, 2005) Ignoring the multiplicity of tests may lead to the rejection of too many null hypotheses Test multiple hypotheses simultaneously by controlling overall error rates for vectors of hypotheses using the family-wise error rate (FWER) as a criterion Less conservative & more powerful than other methods as takes account of statistical dependencies between tests

12. + Analysis of Maternity Records Maternity recordsN (n HIGH / n LOW) M HIGH (SD) M LOW (SD) Individual Test p Infant Outcomes Apgar 5180 (95/85) 9.79 (0.48) 9.69 (0.63) ns Birthweight (grams)181 (95/86) 3281 (664) 3294 (614) ns Low BW (<2500gr)181 (95/86) 0.10 (0.30) 0.05 (0.23) ns High BW (>4000gr)181 (95/86) 0.09 (0.29) 0.12 (0.33) ns Labour Outcomes Weeks at gestation174 (92/82) 39.60 (2.51) 39.56 (2.01) ns Prematurity (<37 wks)174 (92/82) 0.07 (0.26) 0.06 (0.24) ns Labour onset (Spontaneous=0, Induced=1) 178 (93/85) 0.31 (0.46) 0.43 (0.49) p=0.056 Caesarean section178 (93/85) 0.15 (0.35) 0.25 (0.44) P=0.036 Data Collection Summer 2012 accessed maternity records for PFL participants who attended Rotunda Hospital (85%) & NMH (15%) 283 parents gave consent to access records - available records for 272

13. + PFL Maternity Records – Summary No programme impact on birth weight or gestational age, but programme altered the birth experiences of the participating families Significantly fewer mothers in the high treatment group had a c-section compared to those in the low treatment group Result remains when the incidence of a previous c-section and pariouness are controlled for Programme particularly effective in preventing high treatment mothers from having a repeat c-section Result possibly linked to Tip Sheets which specifically focus on preparing the mother for the birth and being aware of the birthing process and possible labour outcomes

14. + Analysis of 6 Month Results PFL 6 Month ResultsProportion of Measures Significantly Different at 6 Months PFL Low – PFL HighIndividual Tests Multiple Hypothesis Tests Child Development0% (0/13)0% Child Health10% (3/30)0% Parenting23% (5/22)20% Home Environment & Safety36% (8/22)50% Maternal Health5% (1/20)25% Social Support38% (5/13)0% Childcare & Service Use7% (1/14)0% Household Factors & SES0% (0/26)0% Total Statistically Different14% (23/160)12% (3/25) Test for treatment effects across 160 measures & 25 Step-down categories Sample: PFL High/Low treatment groups: 84/90

15. + ASQ Scores & Difficult Temperament N (n HIGH / n LOW) M HIGH (SD) M LOW (SD) Individual Test p 1 Step Down Test p 2 Effect Size d ASQ Gross Motor Score 173 (83/90) 40.78 (11.93) 38.50 (12.99) ns 0.18 ASQ Communication Score 173 (83/90) 53.07 (7.84) 51.78 (8.49) ns 0.16 *Difficult Temperament 173 (83/90) 11.70 (5.71) 12.21 (5.50) ns 0.09 ASQ Personal Social Score 172 (82/90) 46.52 (12.09) 45.94 (13.57) ns 0.05 *ASQ Social-Emotional 173 (83/90) 14.76 (10.68) 15.17 (13.75) ns 0.03 ASQ Fine Motor Score 173 (83/90) 50.84 (9.46) 51.39 (10.17) ns 0.06 ASQ Problem Solving 173 (83/90) 51.87 (9.39) 52.56 (9.92) ns 0.07 Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ns indicates the variable is not statistically significant. E XAMPLE OF R ESULTS : C HILD DEVELOPMENT @ 6 M ONTHS

16. + Positive Treatment Effects @ 6 Months Child Development: none Child Health: Appropriateness & frequency of eating; Immunizations Parenting: Parent-child dysfunctional interactions; Baby comparison score; Parental hostile- reactive behaviour; Interaction with baby Home Environment: HOME sub-scales – variety, childcare, toys & books, physical environment, learning materials; Electrical socket covers, Framingham safety survey Mothers Health: Mother less hospitalisation for complications after birth Social support: Frequently sees grandparent; Discuss programme with partner & others; Knows children same age as baby; Meet friends frequently Childcare & Service use: Voted in last election Household & SES: none

17. + Analysis of 12 Month Results Test for treatment effects across 140 measures & 23 Step-down categories Sample: 12 Months: PFL High/Low treatment groups: 83/82

18. + Positive Treatment Effects @ 12 Months Child Development: ASQ Fine Motor skills; BITSEA Social Competence cut-off Child Health: Chest Infections; Immunizations; Appropriateness eating Parenting: none Home Environment: none Mothers Health: Drank alcohol Social support: Meet friends frequently; Voted in last elections Childcare & Service use: ~ Household & SES: none Results consistent with the home visiting literature

19. + Attrition & Dosage Attrition/disengagement Attrition may bias evaluation results if non-random 29% of high treatment & 29% of low treatment dropped out/disengaged between randomisation & 12 months Few individual characteristics predict attrition – IQ more important Dosage/engagement Variation in dosage, may lead of variation in effect Collected data on number/duration of home visits On average, high treatment group received 21 home visits of ~1 hr in duration between pregnancy & 12 months 58% of prescribed visits delivered based on bi-weekly visits Higher maternal IQ associated with higher engagement

20. + Conclusions to date Many of the results are in hypothesized direction Few significant effects, yet consistent with literature Significant findings on birthing experiences, parenting, home environment, child health, social support Corresponded directly to information on the PFL Tip Sheets No significant impact on many key domains such as child development, birth weight, breastfeeding, maternal well-being etc. Attrition & disengagement between 6 & 12 months very low Engagement below prescribed dosage Participate satisfaction is relatively high (focus groups)

21. + Update on PFL Oldest PFL child is 4 years and old & youngest is 18 months 24, 36 & 48 month surveys are currently in the field 18 month results available very soon Over 1,600 interviews conducted as part of impact evaluation Conducted five school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011, 2012 Evaluation will continue until all children start school Programme website: www.preparingforlife.comwww.preparingforlife.com Evaluation website: http://geary.ucd.ie/preparingforlife/ttp://geary.ucd.ie/preparingforlife/