Workshop Aim To provide participants with useful background information and a practical step -by-step guide to planning your evaluation approach and developing a simple and effective outcome monitoring questionnaire for your project
STEP 1: Digging Around Use simple Google searches to do some research into your area of interest What evaluation tools and techniques are similar organisations using? Some common systems are available including Outcome Stars and Quality of Life Indicators: check if these could be appropriate Take stock each time you start a new project Talk to other people doing similar work
STEP 2: Set your Outcomes “Outcomes are the difference your project can make to your community or beneficiaries” Big Lottery
SMART Outcomes Specific Measurable Achievable Realistic Time-bound –Significant –Meaningful –Attainable –Relevant –Resourced –Results-based –Timely Outcomes need to be SMART before you can begin to work out how to measure whether you have achieved them or not!
Example 1 OUTCOME Increased confidence SMART OUTCOME Older people who participate in the programme will evidence an increase in their confidence to speak up for themselves, which will be shown immediately following their participation, and will still be evident 6 months later
Example 2 OUTCOME Increased use of recreational facilities SMART OUTCOME The number of people using the recreational facilities will be evidenced to have increased by 10% in the first year of the project and by 20% by the end of the project Or The number of people who report using more than one of the recreational facilities at the Centre will have increased from 15% to 20% in the first year of the project and will reach 25% by the end of the project
STEP 3: Evaluation Questions Turn your SMART outcomes into evaluation questions Remember, evaluation is COMPARATIVE by nature past with present project 1 with project 2 method a) with method b) before and after
GOOD evaluation questions Easy to understand Written in plain language (no jargon or abbreviations) Not too long Only ask one thing per question Don’t ask ‘leading’ questions Offer mutually exclusive and exhaustive categories Can be closed or open-ended Can use recognised ‘scales’
Ordinal Scales (LIKERT) ExcellentVery goodFairPoor Strongly agree AgreeNeither agree nor disagree DisagreeStrongly disagree AlwaysOftenSometimesRarelyNever NoneVery mildMildModerateSevere Completely satisfied Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Completely dissatisfied
Developing Evaluation Questions SMART Outcome: Older people who participate in the programme will evidence an increase in their confidence to speak up for themselves, which will be shown immediately following their participation, and will still be evident 6 months later
…continued Evaluation Metric: What % of beneficiaries report an improvement in their confidence to speak up for themselves?
…continued Evaluation Question: To make this evaluation question comparative you must ask it at least twice: BEFORE and AFTER your project intervention (in this case the question must also be asked 6 months later as well)
STEP 4: Develop your Evaluation Plan Develop an evaluation plan which details all of your intended project outcomes and evaluation questions, and sets out how you will collect the data to monitor your achievement of these, alongside wider service monitoring and evaluation
Evaluation Plan: Contents (i) Data reporting deadlines Service monitoring and evaluation questions (RBA ‘How much are we doing’ and ‘How well are we doing it’?) SMART outcomes (RBA ‘Is anyone better off’?) Outcome evaluation questions
Evaluation Plan: Contents (ii) Use of monitoring and evaluation data To meet funding requirements? To influence social policy? To gain further project funding? To scope a particular issue or problem? Evaluation participants (who you will collect the data from)? Beneficiaries? Volunteers? Stakeholders?
Evaluation Plan: Contents (iii) The type of data you will collect The data collection methods you will use
Evaluation Plan: Contents (iv) Analysis How you will analyse your results Reporting What kind of reports you will need to produce from your results Dissemination How your findings will be shared to have the greatest impact Unfortunately there won’t be time today to go into these areas in more detail
Quantitative / Qualitative Quantitative data is essentially NUMBERS Qualitative data is essentially WORDS
Quantitative Learning little from many: can make generalisations Standardised responses Non-deliberative Numerical Based on sample sizes Learning a lot from a few: hard to make generalisations Open responses; conversational and flexible Deliberative Non-numerical; ideas; beliefs; meaning Based on experience Qualitative
What kinds of data can be collected? “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted” Albert Einstein
Data Collection Methods Different types of data require different data collection methods to be used Quantitative Survey Routine monitoring data collection (stats) Qualitative Interview Focus group Digital story
Ensuring Quality Data Making sure you collect the right data at the right time and in a standardised way is essential to making sure the data you collect, analyse and share is good quality Different ‘methods’ use different ‘tools’ to do this, such as a questionnaire, a structured interview schedule or a focus group discussion guide
What is an Outcome TOOL? An outcome tool is something designed to ensure you can assess or describe change in a consistent and standardised way. This will produce data which is of a better quality Good quality data makes it possible to draw together information on a number of beneficiaries and give an overview of the change achieved for a whole project or group of projects
Developing an Outcome Tool for your Project Measuring outcomes for beneficiaries
Simple and Effective Make the TOOL as useful as possible Simple design which is easy to follow Gather data which answers your evaluation questions Use recognised SCALES where appropriate to ensure quality Think about when and how the TOOL is used
Service Quality Take the opportunity to assess the QUALITY of your service along with beneficiary outcomes
Free Text Space It is always useful to include free text space for beneficiaries to say more about something Whilst not everyone will complete free text space, those who do will often have something constructive or positive to say You can get great quotes for future use!
Capture project ‘champions’ Make sure there is space for beneficiaries to fill in their contact details if they wish Take the opportunity to ask if they would mind being interviewed or contributing further to your project evaluation User voice is exceptionally powerful so don’t risk losing it!
Simple Structure Order the questions well according to any natural timelines Move from familiar to unfamiliar Don’t ask sensitive questions near the beginning Limit the number of questions End with easy questions Make the completion instructions clear
Data Collection Points “Timing has a lot to do with the outcome of a rain dance” Cowboy Proverb
Data Collection Points Baseline Progress End of Intervention Impact You must ask exactly the same questions each time when trying to evaluate outcomes. Service monitoring and quality questions can be one-off
Completion Method Face to face By post By the client alone or with support By a proxy (for people with dementia) By a volunteer Standardising your data collection method helps ensure the data gathered is high quality
Useful Resources www.ces-vol.org.uk http://www.biglotteryfund.org.uk/er_eval_explaining_the_difference.pdf Link to a document called ‘Explaining the difference your project makes’ http://www.evaluationtrust.org/evaluation/evaluate http://www.worldbank.org/ieg/ipdet/presentation/M_05-Pr.pdf Power point module on writing good evaluation questions http://www.keene.edu/crc/forms/designingsurveysthatcount.pdf Slides on good practice in writing evaluation questions http://www.homelessoutcomes.org.uk/resources/1/Guide%20to%20Outcomes %20Tools%20Second%20Edition.pdf Really useful publication describing outcome tools (written for the homelessness sector but more widely applicable)