Challenges in evaluating child & family welfare services Robyn Parker, Myfanwy McDonald, & Jenny Higgins Association of Children’s Welfare Agencies Conference, 2-4 August 2010, Sydney

Context Evaluation of programs becoming more of a reality – Clients’ right to receive high quality services – Growing need to justify expenditure Only way to know is through evaluation Not new – we do it every day, just not in a formal & systematic way AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Context This session not about actually doing evaluation; more about principles and avoiding derailment – Sticking to the rules – Easing the way – Looking to the future AUSTRALIAN INSTITUTE OF FAMILY STUDIES

What is evaluation? A systematic process to determine whether objectives met – did the client change or improve in the way we wanted them to, and was the program responsible for that change? Involves making a judgment – is this program worth pursuing/funding, is the investment justified, to what extent is it doing what it was intended to do? AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Why evaluate? Did participants change significantly in the way the program logic suggested? i.e. did parenting confidence improve, is the child making progress, are newbie parents better equipped to cope with parenthood, do parents of toddlers employ positive parenting skills, does the child exhibit fewer challenging behaviours, etc... AND To what extent was the program responsible for the change? AUSTRALIAN INSTITUTE OF FAMILY STUDIES

General process Describe program in detail – objectives, methods, etc Develop evaluation questions Decide what to measure, when, where, how, who will be ‘measured’, & who will do the ‘measuring’ Put procedures in place Collect & analyse data Interpret results, report & act on findings AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Challenges (I) Ruling out other explanations – Integrity of the evaluation – sticking to the rules – Evaluation design (pre/post test; comparison/control groups) – Program objectives – evaluation questions – Break it down – what is the pathway to results for participants? – What does the outcome “look like”? “If program effective, client will...” AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Challenges (II) Avoiding “wouldn’t it be nice to know...” items (i.e. research) Collect only what you need, use all that you collect Integrating data collection into day-to-day practice Synthesising data from different sources AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Challenges (III) Practice-based evidence requires input of practice-based professionals Building the capacity of practice-based professionals to conduct evaluation Building capacity requires investment – time, support &/or funding AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Challenges (IV) The data currently being collected may not produce the outcomes data you need (to say “it works!”) Avoiding the influence of other agendas Be prepared to be flexible AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Strategies Getting to “how we do things here”: – Planning; gradual introduction – Participation, skill development – Briefings & training in protocols – Reinforce through ongoing, active management support – Provides updates, interim reports – Acknowledge & accommodate impact on workload – Try starting small AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Ethics – what an ethics committee will want to know 4 basic principles: – Merit & integrity; beneficence; justice; respect. Harm to participants – Minimising, dealing with distress Obtaining informed consent – Parent, child, both? Privacy & confidentiality – Limits (can’t always guarantee) Disconnect in assessment of above - derailment AUSTRALIAN INSTITUTE OF FAMILY STUDIES

Further resources: [look for new resource published later this year] [Evaluation Resource sheet] [National Statement...] AUSTRALIAN INSTITUTE OF FAMILY STUDIES

FINAL TIPS PLAN REVIEW PLAN PLAN REVIEW PLAN PLAN... RINSE AND REPEAT AUSTRALIAN INSTITUTE OF FAMILY STUDIES