The golden thread of evaluation in effective programs and practice Alicia McCoy, Research & Evaluation Manager Jo Cavanagh OAM, CEO

About Family Life

Research & Evaluation at Family Life o Research and Evaluation Unit since 2009 o Developed ‘organically’ for first 3 years o Evaluation Capacity Building approach o Program Planning and Evaluation Framework o Training, consultation and support o Family Life Knowledge Hub o Research partnerships o Outcomes Framework (currently being developed) Where are we now?

The facts Creating an organisational environment for successful evaluation and social impact measurement requires consideration of: Culture Leadership Communication Systems and structures References: Preskill, H. & Torres, R. (1999) Evaluative Inquiry for Learning in Organisations, SAGE McCoy, A., Rose, D. & Connolly, M. (2014) Approaches to evaluation in Australian child and family welfare organizations, Evaluation and Program Planning, 44, McCoy, A., Rose, D. & Connolly, M. (2013) Developing evaluation culture in human service organisations, Evaluation Journal of Australasia, 13, 15-20

How to we get to here! An initial thought… “Qualitative evidence is the ‘bones’, qualitative evidence is the ‘flesh’, and evaluative reasoning is the ‘vital organs’ that brings them both to life” -- E. Jane Davidson (2014) Davidson, E.J. (2014) How “beauty” can bring truth and justice to life. In J.C. Griffith B. Montrosse-Moorhead (Eds.), Revisiting truth, beauty and justice: Evaluating with validity in the 21 st century. New Directions for Evaluation, 142,

Environment Influencing factors: increased scrutiny articulate impact of services on reducing/influencing social issues in an austerity environment Federal and state governments reviews interest in new ways of funding services; outcome-based funding Evidence-informed programs and implementation vs social innovation Impact measurement e.g. Social Return on Investment Collective Impact Strategic philanthropy Impact investing reliant on evidence of effectiveness; Social Benefit (Impact) Bonds

Untangling the golden thread… Evidence- informed practice Individual learning Professional ethics Professional standards Organisational ethics Organisational learning Leadership Culture Processes Evaluative inquiry Communicatio n

Levels of evaluation use McCoy, A., Rose, D. & Connolly, M. (2014). Approaches to evaluation in Australian child and family welfare organizations. Evaluation and Program Planning, 44, [Adapted from Patton 2008] Entirely external Minimal ad hoc internal evaluation Occasional internal evaluation Part-time internal evaluator Full-time internal evaluator Routine internal evaluation Fully integrated and highly valued internal evaluation

How organisations use evaluation

Considerations for organisations Meeting organisational needs Cost Utilisation Strategy Expertise Organisational understanding Relationship-building and management Bridging the “research-practice gap” Objectivity and credibility

Family Life’s journey Entirely external Minimal ad hoc internal evaluation Occasional internal evaluation Part-time internal evaluator Full-time internal evaluator Routine internal evaluation Fully integrated and highly valued internal evaluation

Family Life’s journey (cont’d) Internal research and evaluation position appointed 2011 Program Planning and Evaluation Framework introduced Research and Evaluation Manager joins Executive Managemen t Team 2 questions in agency- wide staff engagemen t survey First research and evaluation climate survey undertaken Outcomes Framework developed Research and evaluation consultation days commence at all service centres Regular presentation s at team meetings, all agency days External research and evaluation partnership s with universities Second research and evaluation climate survey undertaken 2014

Family Life’s approach Evaluation Capacity Building … “is a context-dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organisations/programs/sites” -- Stockdill et al (2002) Stockdill, S.H., Baizerman, M. & Compton, D.W. (2002) Toward a definition of the ECB process: a conversation with the ECB literature, New Directions for Evaluation, 93, 7-25

Staff climate survey

Challenges Mainstreaming evaluation meaningfully e.g. ‘anything is not good enough’ Organisational growth impact. Convincing practitioners evaluation is of programs and services, not of individuals Subcultures may undermine the broader effort to create an organisational-wide culture (critics and problem generators Vs curious questioning) Research and evaluation is work for all – not a job title

What we have learnt There is a tipping point. Meet practitioners where they are at. Different disciplines, evidence-informed practice and evaluation. A sense of safety in learning - constructive feedback is welcome and beneficial Know your evaluation champions.

Trust in practitioners’ knowledge. Provide a ‘hook’ – make evaluation interesting and relevant Be comfortable with diverse views and focus efforts strategically What is paid attention to, and publicly valued is what will get done. What we have learnt (cont’d)

Personal relationships are important Attitudes toward evaluation are not binary. Market evaluation with this in mind Make the evaluation message noticeable but not overwhelming Make messages about staff roles and responsibilities clear and consistent Leaders are the ones that drive culture Communication is key

A final reflection “Evaluation must become part of the culture, building a common language. If these things happen, evaluation behaviour will follow. Changes in values lead to changes in behaviour… We are asking people to mainstream a frame of mind, to ask good questions, be sceptical, use answers to bring about change, to continue to move toward excellence in practice” -- James Sanders (2001) Sanders, J.R. “Reflections on mainstreaming evaluation”. Comments presented during Session 790 at the annual meeting of the American Evaluation Association, St Louis, November

Thank you Alicia McCoy Research and Evaluation Manager Jo Cavanagh OAM CEO