1. intandem evaluation

2. intandem evaluation Presentation contents Quick exercise
The intandem approach to data collection - evaluation
The intandem approach to monitoring
Recording the data
External evaluation

3. intandem evaluation – quick exercise
In pairs (or in threes) identify five different groups who would be interested in knowing the impact the service is having – and explain why they would be interested.

4. intandem evaluation – quick exercise
Children
How much progress are they making?
Where might support help them?
Volunteers
How much difference are they making for their mentee?
Are they addressing the child’s needs?
Co-ordinators
Are the volunteers making a difference?
Are there some mentors or mentees who need more support?
Do we need to review how we train/support our mentors?
Service managers
Is the mentoring service making a difference?
Are there ways of improving the service?
Charity Board
Are we delivering what we set out to deliver?
Is the mentoring service effective?
Inspiring Scotland
Is the mentoring service effective right across the portfolio?
Where there are differences, do we understand them?
Scottish Government
Is the funding achieving what was expected?
Does the impact measured justify funding for a wider group?

5. intandem evaluation – five core outcomes
Children have increased confidence
Children have improved self-esteem
Children have improved social relationships
Children are more engaged with the local community
Children experience reduced isolation 1 2 3 4

6. intandem evaluation – confidence and self-esteem
Confidence is a belief in being able to do things – and confidence can vary from one area of your life to another. I am confident I can cycle 40 miles in a day, but I am not confident to speak in front of people without notes.
Self-esteem is how you see yourself in the world, what value you place on yourself. Self- esteem tends to be more consistent across different areas.

7. intandem evaluation – confidence indicators
I know the things I am good at and enjoy doing them
I enjoy trying to do new things
I am good at asking for help
I can cope well in new and unexpected situations
I am comfortable meeting new people
All scored on the basis of Strongly Agree (4), Agree (3), Disagree (2) and Strongly Disagree (1), and on the assumption that we are talking about how the child feels most of the time.

8. intandem evaluation – self-esteem (Rosenberg)
I feel that I am as good as other people
I think there are lots of good things about me
Sometimes, I feel like I am a failure

9. intandem evaluation – self-esteem (Rosenberg)
I am able to do things as well as most other people
I like being me
I feel good about being me
Sometimes I think I am no good at all
I feel I do not have much to be proud of
Sometimes I feel useless
I wish I liked myself more

10. intandem evaluation – self-esteem (Rosenberg)
I feel that I am as good as other people
I think there are lots of good things about me
I am able to do things as well as most other people
I like being me
I feel good about being me
Sometimes I feel like a failure
Sometimes I think I am no good at all
I feel I do not have much to be proud of
Sometimes I feel useless
I wish I liked myself more
What do you think? Reliability, validity and appropriateness or fewer questions and softer statements.

11. intandem evaluation – social relationships and isolation
People who care about me
People I can talk to
about anything
People I really like
being with
People I talk to and
play games with

12. intandem evaluation – community engagement
Four levels of engagement:
0. Zero awareness
Awareness
Access
Participation
Ownership / leadership
Across five areas of community:
Sports facilities
In school (out of hours) activities
Community centres/youth club/libraries
Medical and dental services
Local facilities – shops, McDonald’s, cinema

13. intandem evaluation – community engagement
Sports
Out of school
Entertainment
Health
Shops 1 2 3 4

14. intandem evaluation – when and who?
Baseline data: as soon as practical after the referral has been accepted by you: just knowing that they will be getting a mentor could have an impact.
So, at first home visit, by the co-ordinator.
Difference measurement: every six months – from the date of the baseline data.
So, within the mentoring relationship, by the volunteer mentor.

15. intandem evaluation – recording the data
Use your own systems initially
Small numbers, so no great administrative issue
Always keep the source paper work until information transferred
We are reviewing options for a web-based approach to recording this data
Keep a note of those children whose baseline data is captured before the web- based approach is available

16. intandem evaluation – other data
Recording progress against individual objectives
Significant SHANARRI indicators e.g. safe
Monitoring data for the children
Monitoring data about the mentors
What do you think? Should we track achievement of individual objectives? If so, how should we do this?

17. intandem evaluation – monitoring data: mentees
Age
Gender
Time as before matching
Local authority
Post code
Living in: rural/village, small town, large town/city
Number of parents at home; number and age of siblings at home
Outcome: matched or not matched (if not matched, reason for this)
Match history: start date, end date, name of mentor, typical meeting time, activities undertaken
Education: attendance records and instances of exclusion; educational attainment during period of match

18. intandem evaluation – monitoring data: mentors
Age
Gender
Employment status: full-time, part-time, student, unemployed, retired
Local authority
Post code
Initial commitment (in months)
Additional training attended
Outcome: matched or not matched
Match history: start date, end date, age/gender of mentee, travel time to mentee, transport used to meet mentee

19. intandem evaluation – external evaluation
In April we will be discussing with potential evaluators what is required for a successful evaluation of intandem.
What we already know is we need:
Fidelity (all mentors volunteers, same frequency of meetings)
Consistency in data collection
What do you think? How can we ensure that the data is collected consistently if we use the volunteers to be involved in collecting it? And, what do you need from us?

20. intandem evaluation
Questions and Comments……………..
……………then lunch.

21. intandem evaluation – Rosenberg self-esteem

22. intandem evaluation – summary

23. intandem evaluation – external evaluation