1 Making the most of evaluation a provider perspective on the outcomes of Brighter Futures Andrew Anderson, The Benevolent Society

2 The presentation Background to Brighter Futures and The Benevolent Society (TBS) Background to the Brighter Futures evaluation Agreement between TBS, Social Policy Research Centre (SPRC) and Community Services (CS) Benefits and constraints of the partnership Implications for future evaluation practice

3 The Benevolent Society Provides support to more than 40,000 children and adults each year Child and family services range from childcare to child protection programs More than 650 staff and 800 volunteers 145 programs across 60 locations in NSW and Queensland Revenue in 2011 was $78m Evaluation is a strategic priority at The Benevolent Society.

4 Background to Brighter Futures & TBS Early Intervention Program in NSW Funded by CS – delivered in partnership between CS and 14 NGOs For vulnerable families with children (0-8) at risk of abuse or neglect. Includes: Childcare, parenting support and structured home visiting Aim to prevent further progression within the child protection system TBS delivers program in 5 areas – last 12 months approx 700 families

5 Background to evaluation Carried out by SPRC Included results, process and economic evaluation Results evaluation included family survey administered to all families by their caseworker on entry, six months and exit. Survey included data on carer wellbeing, parenting practices and child behaviour NGO lead agencies had no access to the results of the family survey for their client groups Evaluation also drew heavily on minimum client and service data provided to DoCS by lead agencies

6 The agreement Brighter Futures priority for evaluation at TBS and didn’t want to duplicate evaluation efforts Agreement between TBS, SPRC and CS for TBS to get access to results of family survey for their clients SPRC undertook additional analysis of family survey data for TBS clients Included six monthly reports with breakdown by service areas Baseline family survey data for 344 families (41% response rate) and follow up data for 129 families

7 Benefits of the arrangement Legitimised and encouraged administration of Family Survey i.e. staff knew they would get something back Provided access to new information including: –more detailed information about client characteristics and needs –Comparable data across our different sites –Data on the nature and number of child protection reports

8 Employment status

9 Length of time on program

10 Report data

11 Benefits of the arrangement Provided affirmation for staff of positive work they were doing Provided useful information for program development and advocacy: –highlighted multiple and complex needs of clients and implications of this for service delivery model –identified issues with participation and retention of clients especially Aboriginal and Torres Strait Islander clients Comparison to overall evaluation reports provided comparative data outside of TBS

12 Constraints of the evaluation No comparison group i.e. don’t know if changes were due to the program Time lag in getting the results of the data – sometimes 12 months and caught up in delays with CS reporting Limited information about change in outcomes over time - response rate and attrition in the Family Survey No process evaluation data

13 Implications for future evaluation policy Program providers should be involved in the design of program evaluations including tools and measures to be used Where possible evaluations of multi-provider programs should be designed to provide feedback to individual providers on their results This will increase buy in from providers to the evaluation and increase their participation Program evaluations that focus on what works for whom under what circumstances are more useful in informing future program development than those that only focus on whether the program works on average

14 Contact Andrew Anderson Tel