Care to Share? Capturing learning from the Short Breaks Fund Graeme Reekie, Evaluation Support Scotland

Background Learning: Funded Groups Shared Care Government Carers and cared-for “What works, for whom, in what circumstances?”

The Plan! Quick reminder on outcomes Measuring soft outcomes –(using indicators) Evaluation tools, including participatory Evidence Reporting Planning for action

Reminder Outcomes: The change or difference That ‘comes out’ of your activities Tips: Use change words (e.g. increase, reduce) Ask ‘So What?’

Indicators Want to: Involve users? Spread the burden? Strengthen your evidence? Make better use of evaluation tools? Here’s how…!

Soft outcomes Need to break down into proxy indicators  Take each outcome. What would it look like if we achieved it? Pick 2-3 most important, most likely, easiest to collect Draw out the methods

Indicators must be … Specific Simple Recognisable to your service users Able to be measured more than once and show change Ideally you need a baseline in order to measure progress or change.

Writing from the perspective of the user If it helps, write indicators from the perspective of the user: I feel … I do… I can ….We can…They can

5 I don’t feel sad as much as I used to4 I can keep my temper when I need to3 I find it easier to deal with the tough stuff going round my head 2 I am less scared about things than I used to be1Young people are better able to manage their anger and frustration and anxiety B If I have to I find it easier to stand up to people who are mean to me. 5 I have made new friends.4 I can take part in activities.3 I make suggestions2 I feel surer of myself and less nervous about doing stuff1Young people feel more confident A Outcome indicatorOutcome

Common sources of information on indicators Outcome They tell you Third party tells you Records Hard evidence Observed behaviour changes Assessment

Common methods They tell you Third party tells you Behaviour changes Records Hard evidence Assessment Record casual feedback, Questionnaires, Interviews, Focus groups, Video diaries, Exercises, mapping and visual progress tools Observation notes Attendance sheets, Record of activities, Use of services Tracking what happens, Comparing statistics (eg illness, employment, debt, truancy) Badges, checklists, standards

Traditional methods People tell you: Questionnaire Interview Focus group Either stand alone OR Part of activities

Questionnaires Your questions must relate to your outcomes and indicators. Short and simple - one piece of paper. No leading questions! (‘How good is the service?’). Think about closed or open questions. Think about scales: numbers, smiley & sad faces etc Stamped addressed envelopes and give a deadline. Offer rewards for replying. Do you need to get information from everyone or can you select a sample. Try it out first to check it makes sense (piloting).

Interviews Tips for group interviews: Your topic guide must relate to your outcomes and indicators. If possible use a colleague to facilitate if you need very personal feedback. Give info in advance about the times, venue, expenses. Provide refreshments, introductions, ice breakers. Keep the discussion moving on when issues have been addressed. Do not let one person dominate the group, invite less vocal people to offer their views. Include group interview as part of an existing group activity?

Other methods Observation Group discussion (set and measure indicators) Diaries & video diaries Third party feedback (referrers, family members etc) Visual representations of progress – eg trees, wheels, dart boards, ladders Body maps Time lines More info on our website

Relationship map (social network) At the start of the projectAt the end of the project X X Liz Mum Carol Tom Mum Liz Carol Tom Dad Callum Di Mary Susan Andrew

Body maps: Example 1 Before participation

After participation

Balanced Wheels 1.Ask people to shade portions of a circle 2.Could represent the importance of different things to them 3.Or time spent

Exercise The Short Breaks Fund generates increased understanding of the role of short breaks and respite in supporting caring relationships. What would that look like?

How do you like your evidence? Poached? Scrambled? Hard boiled? Fried? All in one basket?

Jigsaw or mosaic? Evidence comes from a variety of sources Your can build pictures using those different sources So that you can identify: what outcomes for whom in what circumstances and why

Where do you get evidence for your decisions? The last person you spoke to? The Esoteric Journal of Perpetual Enquiry, vol 5349?

Take your evidence and TREBL it! Transparency: methods clear, limitations acknowledged Relevant: up to date, appropriate Enough: strength of evidence v proportionality Believable: accurate, representative, reliable Legitimate: coming from the right sources

What goes in a report? Original aims Planned outcomes and activities Headline achievements Main facts about activities Main facts about outcomes What went wrong and why? Case studies Learning and changes for the future Financial information Appendices

Scotland Funders’ Forum The best reports come from organisations that can evidence their work tell their story use reporting to reflect on what they have achieved and learned (Scotland Funders’ Forum: Harmonising Reporting Working Group 2010)

Summary Outcomes: What ‘comes out’? So What? Indicators: What would that look like? Tools: Traditional, maps, wheels. Evidence: TREBL Reports: Telling the story More help on our website: