1. Evaluation and Out of Home Care Iain Matheson ACWA Conference 2008

2. 1.What is Evaluation Research? 2.Overseas Out of Home Care Examples 3.Commissioning Evaluation 4.Some Issues for Out of Home Care 5.Developing an Evaluative Management Culture

3. Evaluation Research – a Definition Professional evaluation is defined as the systematic determination of the quality or value of something (Scriven, 1991).

4. What is evalu(e)ation research?  Summative and formative  What can be evaluated?  Evaluation questions  Specific logic, methodology and techniques  Logic models  Same data and analysis methods as applied research  Evaluation v research standards for quality

5. Evaluation as Knowledge Production ActionBasic Evaluation Applied

6. Evaluation to Inform Decisions Monitoring Evaluation Research Reviews Audit of Standards Inspection

7. Some Evaluation Approaches  CIPP model (Context, Input, Process and Product)  Empowerment evaluation  Experimental and quasi-experimental  Fourth Generation evaluation  Goal-free evaluation  Indigenous-led research/evaluation

8. More Evaluation Approaches  Participatory evaluation  Rapid reconnaissance  Stakeholder evaluation  Strategic evaluation  Theory-based evaluation  Utilisation-focused evaluation

9. Overseas Out of Home Care Examples Context  Cashmore & Ainsworth (2004)  Bromfield et al (2005)  Not alone

10.  Current  Pilot of Oregon Social Learning Centre model  York U and Manchester U  Funded by Dept. for Education and Skills  15 evaluation sites  Quasi-experimental Treatment Foster Care in Eng. Project

11. Youth Horizons Trust SCD Bridging Programme - NZ  2005  SCD residential programme  Centre for Social Research and Evaluation  Funded by Ministry of Social Development  Literature review, document analysis, observation & interviews

12. Community Alternative Placement Scheme - Scotland  2002  Action for Children (NCH) project - family placements alternatives to secure care.  Moira Walker, Malcolm Hill and John Triseliotis  Funded by Scottish Executive  Document/data analysis and interviews – longitudinal with control group

13. Tri-County Treatment Foster Care Program - Canada  Current  Tri-County Treatment Foster Care Program - hard- to-serve children  Centres of Excellence for Children’s Well-being  Document/data analysis and interviews – longitudinal with control group

14. Foster Care to College Partnership - US  Current  Completion of schooling by yp in foster care and take up post-compulsory education  Washington State Institute of Public Policy  Mason Burley  Administrative data and statistical analysis

15. Commissioning Evaluation  Scoping  Budget  Resources/commitment – and stakeholders  Clarity on what is being evaluated  Methodological robustness  A matter of...timing  Dissemination

16. Other Issues for Out of Home Care  Difficult area to research  Complex intervention logic  Pilots, pilots, pilots  What counts as good evidence?  Data v information v evaluative judgements  Still not widely used in out-of-home care

17. Developing an Evaluative Management Culture All forms of learning activity that aim to improve:  Focus  Impact  Outputs and  Outcomes

18. Thanks! Iain Matheson iain@mathesonassociates.co.nz References Alkin, M. (2004) Evaluation Roots: Tracing Theorists’ Views and Influences, Thousand Oaks, Sage Beddoe, L., Fouche, C., Harington, P., Light, G., Lunt, N., & Yates, D. (2007). Growing research in practice; The story so far. Social Work Review, 2007(Autumn), 39 – 50. Bromfield, L. M., Higgins, D. J., Osborn, A., Panozzo, S.,& Richardson, N. (2005). Out-of-home care in Australia: Messages from Research. Melbourne: National Child Protection Clearinghouse, Australian Institute of Family Studies. Burley, M. (2007). Foster care to college partnership evaluation: Program overview and research design. Washington DC: Washington State Institute of Public Policy. Cashmore, J. and A. F. (2004). Audit of Australian out-of-home-care research. Sydney: Association of Child Welfare Agencies. Centre of Excellence for Children’s Well-being (2008). "Intervention with attachment disordered children in the child welfare system." Retrieved on 28 July from http://www.cecw-cepb.ca/projects/attachment_disorders_e.ht Davidson, E. J. (2005). Evaluation Methodology Basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. Fox, M., Martin, P., & Green, G. (2007). Doing practitioner research. London: Sage. Patton, M. Q. (2002). Qualitative research and evaluation methods (3 ed.). Thousand Oaks, Calif. : Sage Publications. Saville-Smith, K. (2005). Evaluation of the Youth Horizons Trust SCD bridging programme report. Ministry of Social Development, Wellington. Scriven, M., (1991) Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage. Walker, M., Hill, M. & Triseliotis, J. (2002). Fostering and secure care: An evaluation of the community alternative placement scheme (CAPS), Edinburgh: Scottish Executive.