1 External influences shaping the evaluation of a tertiary child protection program Andrew Anderson, The Benevolent Society

2 The presentation Background to The Benevolent Society (TBS) and its child protection program Background to the evaluation Lessons learned Implications for future evaluation practice

3 The Benevolent Society Provides support to more than 40,000 children and adults each year Child and family services range from childcare to child protection programs More than 650 staff and 800 volunteers 145 programs across 60 locations in NSW and Queensland Revenue in 2011 was $78m Evaluation is a strategic priority at The Benevolent Society.

4 Background to child protection program Three locations in Sydney since early ’90s Child under 12 where abuse or neglect has been confirmed and child at high risk of removal Detailed assessment of families and a range of interventions for children and parents Work with families to ensure safety of child and to address issues that led to abuse and neglect.

5 Background to the evaluation TBS’ first formal internal evaluation Identification of outcomes, development of evaluation framework, development of service model Outcomes included: –Improved child development and child behaviour –Improved parent-child relationship –Improved parenting –Increased safety for children within their families –Increased supportive connections with family/friends and community Suite of outcome measures developed – collected by staff using six monthly survey of clients from January 2007 – June 2010 Qualitative interviews with staff and analysis of administrative data

6 Lesson 1 Choose an appropriate methodology

7 Do your homework about the program

8 What we learned Importance of looking at administrative data and questioning assumptions about the program –decreasing number of referrals to the program –short term nature of the program for many families Outcome data is important but doesn’t mean a lot on its own Don’t be afraid to change approach if it isn’t working A mixed method approach can help avoid these pitfalls

9 Lesson 2 Choose the right outcomes

10 What we learned Outcomes need to be achievable and linked to clients’ length of time on the program Short term and medium term outcomes are important to measure as well as long term outcomes Importance of undertaking a program logic that doesn’t only focus on outcomes but also program theory

11 Lesson 3 Take account of the internal and external influences on the program

12 What we learned Interviews with program managers and staff revealed a number of external and internal influences that had affected the program: –Difficulties with staff recruitment and staff turnover –Changing nature of referrals to the program –Changes to child protection system Importance of collecting this contextual information about the program to inform interpretation of results

13 Lesson 4 Need processes in place for organisation to act on evaluation results

14 What we learned Unless processes are in place to actively monitor and implement evaluation results then they are likely to have little impact New processes and structures have improved this at the The Benevolent Society –focus on evaluation at operations meetings –development of performance improvement planning process and link to business plans –Senior Manager research to practice role –increased focus on using data

15 What’s happened as a result Review of child protection programs with Australian Centre for Child protection Involved refining evaluation framework and revisiting service model Child Protection Outcomes Child is safeChild has secure and stable relationships

16 New approach to evaluation Phase 2 Evaluation planning and piloting Phase 3 Implementation Phase 4 Reporting and dissemination Phase 5 Implementing evaluation recommendations Phase 1 Program clarification

17 "The only mistake in life is the lesson not learned.“ Albert Einstein Thank you!

18 Contact Andrew Anderson Tel