Evaluation – the why’s, the what’s and the how’s 2014 Dr Basma Ellahi (PI) Cat Crum (PhD Student)

What is evaluation? Evaluation is a process that critically examines a program. It involves collecting and analysing information about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions (Patton, 1987).

Why evaluate? Answer the objectives of the scheme/project Demonstrate change due to the scheme/project Determine effectiveness i.e. causality or association Inform improvements

‘dull but important When compared with the creative and exciting process of conceiving and initiating a project, evaluation can often be forgotten and be perceived as dull! However, some form of evaluation or formative feedback is the only thing that will show the effectiveness of the project

Evaluations fall into one of two broad categories: formative and summative. Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program. Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals.

Type of EvaluationPurpose Formative 1. Needs Assessment Determines who needs the program, how great the need is, and what can be done to best meet the need. A needs assessment can help determine what audiences are not currently served by programs and provide insight into what characteristics new programs should have to meet these audiences’ needs. 2. Process or Implementation Evaluation Examines the process of implementing the program and determines whether the program is operating as planned. Can be done continuously or as a one-time assessment. Results are used to improve the program. A process evaluation of a program may focus on the number and type of participants reached and/or determining how satisfied these individuals are with the program. Summative 1. Outcome Evaluation Investigates to what extent the program is achieving its outcomes. These outcomes are the short-term and medium-term changes in program participants that result directly from the program. For example, outcome evaluations may examine improvements in participants’ knowledge, skills, attitudes, intentions, or behaviours. 2. Impact Evaluation Determines any broader, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the entire school, community, organisation, society, or environment. Adapted from Norland (2004), Pancer and Westhues (1989) and Rossi et al. (2004).

Aims of evaluation How has the pilot been received by the different participants in each of the health communities/ What have been the successes and the issues – lessons learned? How successful has the pilot been in moving the health communities towards working on the basis of the 5 key principles? How successful has the pilot been in helping the health communities meet their objectives?

Evaluation Objectives of the Malnutrition Prevention Pilot Programme Confirming outcomes and impact indicators Inputs and activities Appraising progress Identify limiters Identify unintended consequences of implementation Generate evidence base Use self generated data

Methodology Impact Evaluation Framework Theory of change Logic Model Linkages between inputs, activities, outputs and outcomes

Evaluation Plan Why? What? Who? How? When? Where?

Complex Evaluation Range of activities -One-off information and/or taster session -Promotional stand -Promotional Campaigns -Health days -Demonstrations -Cooks and eat -Community Enterprise -Training courses Combination of evaluation methods Qualitative Quantitative

The tools of the evaluator Quantitative Monitoring Information Questionnaire/survey Experimental evaluation – RCT, case control, cohort etc Qualitative Observation Interviews Focus Groups Case study Documentation

Methods Semi-structured interviews (telephone) Action Learning sessions Audit - baselines and MUST Sampling strategy – purposive sampling Pluralistic model Ethics Frame of reference

What does good look like? Good evaluation is tailored to your program and builds on existing evaluation knowledge and resources. Good evaluation is inclusive Good evaluation is honest. Good evaluation is replicable and its methods are as rigorous as circumstances allow.

Common dilemmas Intellectual property rights / Data protection and data sharing Be conscious of multiple roles Follow informed-consent rules Respecting confidentiality and privacy Ethics Complexity of data collection sites

How do I make evaluation an integral part of my program? Making evaluation an integral part of your program means evaluation is a part of everything you do. You design your program with evaluation in mind, collect data on an on-going basis, and use these data to continuously improve your program.

To build and support an evaluation system:  Couple evaluation with strategic planning.  Revisit and update your evaluation plan and logic model to make sure you are on track.  Build an evaluation culture

What are the benefits?  better understand your target audiences' needs and how to meet these needs  design objectives that are more achievable and measurable  monitor progress toward objectives more effectively and efficiently  learn more from evaluation  increase your program's productivity and effectiveness

10 reasons to evaluate your project 1.So you know whether it’s working 2. So you can be adaptable 3.To know how things are working 4.So you’re aware of unintended outcomes 5.To be able to better communicate the value of your work

10 reasons to evaluate your project cont. 6. To focus your work 7. To help look after the people you work with 8. Build organisational resilience 9. Know why things are working 10. Life is complicated

Make evaluation part of your program; don’t tack it on at the end!

Resources Magenta Handbook - m/uploads/attachment_data/file/220542/mage nta_book_combined.pdf m/uploads/attachment_data/file/220542/mage nta_book_combined.pdf The Magenta Book: guidance notes for policy evaluation and analysis - content/uploads/2011/09/the_complete_mage nta_book_2007_edition2.pdf content/uploads/2011/09/the_complete_mage nta_book_2007_edition2.pdf

Contact details Centre for Ageing Studies Faculty of Health and Social Care University of Chester Riverside Campus Chester Dr Basma Ellahi – Reader in Food and Nutrition Professor Paul Kingston – Director of The Centre for Ageing Studies, Professor of Mental Health and Ageing. Cat Crum (PhD Student – sponsored by Age UK South Staffordshire)