1. Evaluation tools training
BBO Shropshire
Evaluation tools training

2. Introductions Who you are Three things about your project:
Who is it for?
What will they do?
How will they benefit?

3. Evaluation objectives
To obtain feedback from participants, partners and stakeholders on what is working well and less well, in order to inform changes to the delivery model as the project progresses,
To share good practice and improve the way partners work together to improve employment prospects of those in need
To provide evidence of impact which will help inform future funding bids
In addition to these objectives, the evaluation should be:
Meaningful
Objective, accurate and high quality
Be embedded in the delivery of the project

4. Primary and secondary research
Evaluation framework
Primary and secondary research
Analysis and Report
Learning Events
Self-evaluation (data you collect)
Independent evaluation (data we collect)

5. Outcomes
Questions
Tools
Analysis and Report

6. Outcomes Short Medium Long Increased skills Better partnership working
Reduced social exclusion
Improved self-esteem and relationships
Improved wellbeing and quality of life
Improved employment prospects
Long
Less long term dependency on benefits
Improved long term prosperity and quality of life

7. Methods Participant monitoring data – who, what, when etc…
Partner research - e-surveys and interviews
Participant research:
E-survey (process)
Interviews/focus groups
Entry and exit surveys (impact)
Outcome star
Qualitative tools – choose from a selection
Your own tools Us
You

8. Surveys Entry – at start of involvement in BBO
Exit – at end of involvement in BBO
Doesn’t matter if they complete the exit survey more than once
Outcome star – entry, exit and every 6 months

9. Market Research Society Guidelines
Give consent to take part
Be honest
Be clear why you are doing it
Respect confidentiality
Ensure that respondents are not harmed or adversely affected
Balance the needs of individuals, clients and the research
Carried out by people who have been trained

10. Do’s and Don’ts Do’s and don’ts Common issues:
Mis-reporting
Mis-understanding
Reluctance to complete
Literacy issues
Mental capacity issues
Other scenarios – what do you do?

11. Completing the survey Designed for people to complete themselves
But be available to provide support
Reassure them what the data will be used for
Make it entirely optional
Practice session – ask each other the questions. Anything they don’t understand or want clarification about? Any changes needed?

12. Completing the survey Before you start:
Do you understand the questions?
What type of answers are expected?
Be prepared to answer questions

13. Sampling Why is this important?
Aim is to achieve a confidence level of 95%, with a margin of error of +/- 5%
The smaller the population, the greater % of returns you need
Total participants
Surveys needed %
1000
278
28%
1500
306
20%
2000
322
16%

14. Practice
Ask each other the survey
Any issues?

15. Outcome star
Complete at start, end and every 6 months

16. Qualitative tools
Quantitative tools (surveys) will tell you WHAT you have achieved
Qualitative tools tell you WHY and HOW
You get to choose!
Observation
Interview
Stories

17. Observation About the Tool
The person leading or observing the sessions tracks progress of individuals or the group
Advantages
It is easy to do and requires little time from participants in the group. It allows the observer to consider changes in individuals and the way the whole group interacts. They work well with groups that struggle with surveys and with participants who may not recognise the changes in themselves
Disadvantages
You are making a judgement on the change that has happened in people and so it is your opinion rather than the participants. The observer needs to be systematic so it takes time from the group leader. It can take time to record and analyse data.

18. Interviews About the Tool
You ask people a set of questions about involvement in the project.
Advantages
It is easy to do and gives you key insights into why or how your project is working. They provide lots of information. You can use interesting tools like video or pod casts to collect and then share what participants say. They can be a great way of involving volunteers in the project.
Disadvantages
They are generally conducted one to one and so this can be time consuming and take the group leader away from the session. They generate a lot of information that needs to be analysed.

19. Stories
About the Tool
People tell you about their experience of being involved in your project or activity
Advantages
Stories are brilliant. They allow participants to tell you in their own way what they have got out of a project. You can use writing, drawing, film or audio to collect peoples feedback. You can do it formally or informally at the end of sessions. They can generate information you can use in case studies and to showcase with funders as well as giving you insight into your project.
Disadvantages
If you are collecting lots of stories it can generate lots of information for you to analyse.

20. Your choice…. Pick a tool! Every quarter, complete either…
3 interviews
3 stories, or
3 observations

21. Evaluation timetable
Next week – final versions of tools and guidance issued
Data you have collected – send to us by end November ’17
Sep-Nov ’17 – we will:
E-survey/interviews with partners
E-survey/interviews with participants
Year 1 report – end December ‘17
Learning to Action workshop – Jan ‘18