Into the wilds – using creative evaluation methods with young people in outdoor and experiential settings. Kaz Stuart & Lucy Maynard

Aims of the session: Overview of research and evaluation Reveal the contrast between practice and contextual demands Experiment with a range of tools Discuss how to analyse and present data

Introductions Who are you? What is your practice? What do you already do r.e. research & evaluation? Why do you do what you do?

Semantics? What are research, evaluation, monitoring and reviewing?

Research This involves finding data to generate new knowledge and insights into open questions e.g. Are there gender differences in outdoor learning?

Evaluation This entails answering criteria based questions on efficacy for a range of stakeholders e.g. What was the impact of the X programme / course?

Monitoring This is the regular standardised measurement of performance e.g. How many young people with which needs did we work with this year?

Reviewing Bringing out responses to experiences in ways that enhance their value, their impact and their benefits. Alternative terms for 'reviewing' are 'debriefing', 'reflection' and 'processing'. (Greenaway, 2010). e.g. What do you think that you have gained from this session?

Why do research and evaluation? Enhanced learning for young people Practice development Organisational learning Knowledge management Fundraising / commissioning Marketing and advertising Sharing and disseminating expertise Influencing policy …..because we have to?

The Context “The public spotlight on third sector organisations and their effectiveness has become intense. It has never been more important for third sector organisations to acquire an understanding of monitoring and evaluation and the skills associated with its practice” (Ellis and Gregory 2008:v)

Select Committee Report (2012) We accept that the outcomes of individual youth work relationships can be hard to quantify ….. …….‘you know good youth work when you see it’… However, with a tight spending settlement and an increase in commissioning of youth services at a local level, it is essential that publicly funded services are able to demonstrate what difference they make to young people.

Spending reviews Less cash Less services Payment by results

AHOEC, 2012 Outdoor education centres used by over 320,000 children each year are under threat. A dozen centres have already closed and the future of another 60 is being discussed. They contribute to a range of important societal targets across education, health, anti-social behaviour and community cohesion…… an approach to teaching and learning which has clear and incontestable evidence verifying its effectiveness. Alistair D. Cook, National Chair of the Association of Outdoor Education Centres

Catalyst outcomes framework and tools

AHOEC materials Pre and post surveys Log books Activities

Tensions? Do these create any tensions for you?

Our Dilemma Internal valuesContextual values Respect for the individual Learning and experience Primacy of the youth work process Human centred - fulfilling potential Anti-oppressive and anti- discriminatory Social justice Evidence based Outcome focussed Quality Indicators of success Quantitative data Value for money

How do you measure this….?

Our solution… Participative Action Research Mixed Methods Theory of Change Qualitative & quantitative creative toolkit

Our Approach Pose questions Investigate (collect data) Interpret the data Take actions External evidence and theory informing practice Practice generating external evidence and theory

The tools A: Quantitative: ‘Standard’ range of questions that could be answered in any format Forms Shields Line outs Outcome stars

Forms

Shields

Line Outs

Outcome stars

The tools contd. B: Qualitative: ‘open ended’ creative tools Journey Maps Your Story Photo Elicitation

Journey Maps

Your story I was dealing when I was 13, I got into some bad stuff, so I was not in school a lot of the time. I had plenty of money, didn’t give a fuck. I didn’t care about education, and had no reason to. Teachers couldn’t understand why I was like that, they just thought I couldn’t be arsed. I did join college, for two years on a mechanics course. I passed, but the teachers there looked down on me. I didn’t feel I belonged. The invitation to find out about this course came through the post, it was at Brown’s night club. I remember when I arrived at Brown’s that day. There were a few other people there, sitting around really quietly, looking awkward! I just thought, I can show the real me here! I walked in and started being chatty and straight away got a new feeling. I’d been quiet before, inside myself. I’d learned that. But when I met the others…I could see why people were quiet, I wanted to show others you can be yourself. The highs for me…well it’s changed me, changed my life. It’s been the time for me to grow up, I know we’re not just going to be hand-fed, it’s been a time for all of us to mature really, do you know what I mean?

Photo ellicitation I remember being so cold and miserable, and yet so chuffed too that I had got to the top of that hill…. I could do what I thought was impossible…even though it hurt!

Group work What are the strengths of the tool? What are the weaknesses of the tool? How relevant is to your work? What are the benefits of the data? What are the limitations of the data? How relevant is this for your stakeholders?

Feedback 10 minutes each…

Close Could you apply any of these in your practice? Will these help you with the tensions? What will you take away? What are your next steps?

Brathay Research Hub evaluation/ evaluation/