Evaluation tools training BBO Shropshire Evaluation tools training

Introductions Who you are Three things about your project: Who is it for? What will they do? How will they benefit?

Evaluation objectives To obtain feedback from participants, partners and stakeholders on what is working well and less well, in order to inform changes to the delivery model as the project progresses, To share good practice and improve the way partners work together to improve employment prospects of those in need To provide evidence of impact which will help inform future funding bids In addition to these objectives, the evaluation should be: Meaningful Objective, accurate and high quality Be embedded in the delivery of the project

Primary and secondary research Evaluation framework Primary and secondary research Analysis and Report Learning Events Self-evaluation (data you collect) Independent evaluation (data we collect)

Outcomes Questions Tools Analysis and Report

Outcomes Short Medium Long Increased skills Better partnership working Reduced social exclusion Improved self-esteem and relationships Improved wellbeing and quality of life Improved employment prospects Long Less long term dependency on benefits Improved long term prosperity and quality of life

Methods Participant monitoring data – who, what, when etc… Partner research - e-surveys and interviews Participant research: E-survey (process) Interviews/focus groups Entry and exit surveys (impact) Outcome star Qualitative tools – choose from a selection Your own tools Us You

Surveys Entry – at start of involvement in BBO Exit – at end of involvement in BBO Doesn’t matter if they complete the exit survey more than once Outcome star – entry, exit and every 6 months

Market Research Society Guidelines Give consent to take part Be honest Be clear why you are doing it Respect confidentiality Ensure that respondents are not harmed or adversely affected Balance the needs of individuals, clients and the research Carried out by people who have been trained

Do’s and Don’ts Do’s and don’ts Common issues: Mis-reporting Mis-understanding Reluctance to complete Literacy issues Mental capacity issues Other scenarios – what do you do?

Completing the survey Designed for people to complete themselves But be available to provide support Reassure them what the data will be used for Make it entirely optional Practice session – ask each other the questions. Anything they don’t understand or want clarification about? Any changes needed?

Completing the survey Before you start: Do you understand the questions? What type of answers are expected? Be prepared to answer questions

Sampling Why is this important? Aim is to achieve a confidence level of 95%, with a margin of error of +/- 5% The smaller the population, the greater % of returns you need Total participants Surveys needed % 1000 278 28% 1500 306 20% 2000 322 16%

Practice Ask each other the survey Any issues?

Outcome star Complete at start, end and every 6 months

Qualitative tools Quantitative tools (surveys) will tell you WHAT you have achieved Qualitative tools tell you WHY and HOW You get to choose! Observation Interview Stories

Observation About the Tool The person leading or observing the sessions tracks progress of individuals or the group Advantages It is easy to do and requires little time from participants in the group. It allows the observer to consider changes in individuals and the way the whole group interacts. They work well with groups that struggle with surveys and with participants who may not recognise the changes in themselves Disadvantages You are making a judgement on the change that has happened in people and so it is your opinion rather than the participants. The observer needs to be systematic so it takes time from the group leader. It can take time to record and analyse data.

Interviews About the Tool You ask people a set of questions about involvement in the project. Advantages It is easy to do and gives you key insights into why or how your project is working. They provide lots of information. You can use interesting tools like video or pod casts to collect and then share what participants say. They can be a great way of involving volunteers in the project. Disadvantages They are generally conducted one to one and so this can be time consuming and take the group leader away from the session. They generate a lot of information that needs to be analysed.

Stories About the Tool People tell you about their experience of being involved in your project or activity Advantages Stories are brilliant. They allow participants to tell you in their own way what they have got out of a project. You can use writing, drawing, film or audio to collect peoples feedback. You can do it formally or informally at the end of sessions. They can generate information you can use in case studies and to showcase with funders as well as giving you insight into your project. Disadvantages If you are collecting lots of stories it can generate lots of information for you to analyse.

Your choice…. Pick a tool! Every quarter, complete either… 3 interviews 3 stories, or 3 observations

Evaluation timetable Next week – final versions of tools and guidance issued Data you have collected – send to us by end November ’17 Sep-Nov ’17 – we will: E-survey/interviews with partners E-survey/interviews with participants Year 1 report – end December ‘17 Learning to Action workshop – Jan ‘18