1 Aims for the input 1.To explain the concept of evidence in a youth work context 2.To distinguish between the terms ‘evidence-based’ and ‘evidence-informed ‘ 3.To discuss the possibilities afforded by an evidence informed approach to youth work, involving a theory of change, and a logic model 4.To set out the challenges involved for different actors
2 Why evidence? The Age of RESULTS and ACCOUNTABILITY Key question: What difference does youth work make?
3 1.Systematic reviews Objective? 2.Randomised control trials 3.Quasi-experimental study 4.‘Before’ and ‘after’ designs 5.Retrospective designs (no baseline) 6.Independent large scale evaluations 7.Traditional literature review 8.Independent small scale evaluations 9.Independent single studies 10.Opinion pieces – peer reviewed journals 11.Expert panel/committee 12.Internal evaluation – large or small scale 13.Practice wisdom – policies, standards, manuals 14.Personal experience – reflective practice Subjective? Is there a hierarchy of evidence? 1-4 ‘evidence based’ = proof of ‘what works’?
4 Evidence based programmes
5 An evidence informed approach Professor Sandra Nutley, University of Edinburgh CES Practice Seminar Series, October 2010 Keynote address Evidence-informed practice: Using research to improve services for children and young people seminar-series-practice-issues
6 Know-about (problems): e.g. the nature, formation, and interrelations of social problems. Know-why (requirements of action): explaining the relationship between values and policy and practice. Know-how (to put into practice): e.g. pragmatic knowledge about service and programme implementation. Know-who (to involve): e.g. care teams; building alliances for action. Sandra Nutley (2010) Knowledge required is more than ‘what works’
7 An approach that helps people and organisations make well-informed decisions by putting the best available evidence at the heart of practice development and service delivery. Sandra Nutley (2010) Evidence informed is...
8 Good outcomes depend on effective action Local decision-making, and the harnessing of local initiative, imagination and adaptation can be pre-eminent, but are much more likely to lead to effective action when combined with a broad collection of accessible knowledge that is drawn from practice and theory as well as research (Schorr, L., 2003: 21-21).
9 Constructing a theory of change Consultation Practice Policy Research and Theory
10 Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed Evidence from practice report-2011_4th-1.pdf
11 Psychology: Erikson Psychosocial development Education: Piaget Schemas, stage theory Multiple intelligences Gardner Experiential Learning: Kolb Learning cycle Learning styles Communication: Berne Transactional analysis Constructivism Bandura Reciprocal determinism Action learning Revans L=P+Q Political theory: Marx Class struggle Theory of knowledge: Habermas Communicative action Social change: Freire Praxis Evidence from theory – a selection
12 Evidence from research…a selection McKee, V., Oldfield, C. and Poultney, J. (2010) The Benefits of Youth Work. Lifelong Learning UK and Unite. Wiggins, M., Bonell, C., Burchett, H., Sawtell, M., Austerberry, H., Allen,E. and Strange, V. (2008) Young People’s Development Programme Evaluation: Final Report. Social Science Research Unit, Institute of Education and University of London with the London School of Hygiene and Tropical Medicine. Bielby, G., Purdon, S., Agur, M., Gardiner, C., George, N., Golden, S. and Taylor, E. (2009) Empowering Young People Pilots (EYPP) Evaluation, Final Report. National Foundation for Educational Research. Lewis-Charp, H. Hanh C.Y., Sengouvanh S. and Lacoe, J. (2003) Extending the Reach of Youth Development Through Civic Activism: Outcomes of the Youth Leadership for Development Initiative. Innovation Center for Community and Youth Development. Eccles, J. and Appleton Gootman, J. Editors (2002) Community Programs to Promote Youth Development Committee on Community-Level Programs for Youth, National Research Council and Institute of Medicine. Washington, DC: National Academy Press. Fullerton, D. And Burtney, E. (2010) Young People’s Health Community Based Approaches, A Rapid Review. Report to the Sexual Health Centre Cork. Insights Health and Social Research. Institute of Education, London. (March, 2012) International review of research literature on youth development Commissioned by Centre for Effective Services on behalf of Department of Children and Youth Affairs, Dublin.
13 Evidence from policy
14 “I think you should be more explicit here in Step Two.” A common problem is that activities and strategies often do not lead to the desired outcomes. This is where evidence from consultation comes in…
15 Outcomes Goals Values Needs and issues Consultation: participants peers experts Evidence from consultation What are the needs, issues and interests? What changes would people like to see? What difference have we made, and has anybody benefitted?
16 Proximal outcomes Outcome AreaSample outcome statements Attitudes, beliefsParticipants are more ready to take on new and more diverse experiences Enhanced confidence, self-esteem, awareness (personal and social) More open to people from diverse backgrounds KnowledgeIncreased knowledge of local area, of society Increased knowledge of rights, social issues, politics More informed about health, sexuality, the law, careers and formal education SkillsEnhanced capacity regarding: public speaking, problem solving, self-efficacy, making decisions, critical thinking Enhanced interpersonal abilities in relation to: teamwork, group work, communications In creased abilities in arts and creativity regarding: music, dance, drama, writing Increase in physical competence in relation to: sports, games, outdoor activities BehaviorMore engagement in structured and constructive activities Increased involvement in decision-making Enhanced positive and pro-social behaviour and diminishing negative and anti-social behaviour Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press
17 Distal outcomes Outcome AreaSample outcome statements EducationDeeper engagement with formal education and alternative routes to qualifications Relationships with adults Improved communications and relationships between young people and adults within communities HealthSafer and more enjoyable sex, reduction in alcohol and substance use Social conditionsIncreased community cohesion, improvement in local conditions and amenities, developing local leadership Economic conditionsIncreased self-sufficiency SafetyReduction in violence, vandalism, offending rates Service provisionIncreased support for homeless young people, Improved cooperation and coordination between local services Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press
18 ShortMediumLong-Term Young people recognise the benefits of engaging in informal learning, and spend more time in structured developmental activities Enhanced sense of capabilities regarding community involvement, education, training Learning skills acquired and transferred to other areas of life eg work, family life, politics Young men more able to articulate their needs and wants Enhanced political skills through their involvement in decision making in youth work settings and the wider community Young people friendly environment, and safer community for residents Chain of outcomes - example
19 Outcomes and indicators - examples Well defined information which shows whether something is happening.
20 Outcomes and indicators - sources
21 Monitoring and evaluation Strategies Inputs Process Outputs Outcomes Evidence Goals Values Needs and issues INDICATORS INDICATORS
22 Logical Chain of Connections Showing What the Program is to Accomplish 22 Situation Analysis Goals derive from Aim, mission, vision, values, ideology Goals inform analysis Populations and target groups Problems and causes, Situations and issues Current provision: strengths, weaknesses, gaps Socio-economic and organisational context affect outcomes Policy, legislation, funding, politics, economy, culture, local conditions Outputs are key areas of work that enable desired outcomes Who we will reach What we will do Where, when and how it will happen Targets and numbers to be reached Frequency of activities Standards to be achieved Benchmarks Outcomes are cumulative changes Short term Gains in knowledge, skills, awareness Changes in attitudes Changes in individual and group behaviour – social action Medium Term Organisational development Change in local situation and circumstances Change in policy, decision- making, practice Long term Social change – health, education, civic, environment... Implementation Resources Activities Strategies distil and give focus to goals Outline of main ways to achieve outcomes INDICATORS INDICATORS Monitoring and Evaluation Collecting and interpreting information about implementation and results, and reporting findings These processes take external factors into account and inform analysis, planning and implementation Targets and Indicators that can provide signs of progress or achievement, may derive from standards and benchmarks Evidence informs all aspects of the logic model Taking ideas from social scientific research, literature, practice wisdom, policy and consultation processes, to inform understandings of problems, situations and issues, as well as ideas about work that can enable desired outcomes and ways of monitoring and evaluating the work Inputs (matched to strategies) Funding Buildings Technology Partners Staff and volunteers Process Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed
Sample logic model 23 Intentions Organisational purpose To develop young people personally, socially, educationally and recreationally in a high-quality, safe environment that respects its users. Analysis (includes consultation) Issues to be addressed: 1. Young people’s psycho-social development 2. Social change 3. Sexual health Inputs (matched to strategies) Staff Volunteers Funding Buildings Technology Partners Process Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed Outputs Adventure: 8 linked weekend challenge courses over 18 months (includes personal and group learning inventory) for 20 participants Social action project: Over 12 months on an issue of importance to young people Series of monthly films, talks and discussions on social issues Information, advice, training: Trial (x2) Teen Talk programme (2 evenings per week for 2-3 hours, over 6 weeks) Plan for further work Outcomes Participants are more ready to take on new and more diverse experiences (S) Enhanced interpersonal skills (M) Learning skills acquired and transferred to other areas of life: work, family life, politics (LT) Increased involvement in decision- making processes (S) More informed decision-making (M) Local services for young people are more user friendly (LT) More aware of harmful behaviours to self and others (S) Risk factors reduced (M) Safer sexual practices (LT) Implementation Resources Activities Results Short Medium Long term Monitoring and Evaluation Evidence Strategies 1. Adventure activities 2. Social action project 3. Information, advice, training
24 Challenges of an evidence informed approach Youth workers Managers Policy makers Researchers
25 Challenge for youth workers Assumptions underlie much of what we do Faulty assumptions are often the reason for poor results – e.g. ‘process is all there is’ Evidence helps us make our assumptions explicit so that we can check and test them: espoused versus theory-in-use Reflective practice (Schön, 1983)
26 Challenge for line managers Trust versus command and control Monitoring and reporting versus support, training and development Creating spaces for reflection Promoting a culture of evidence Underpinning own practice with evidence
27 The challenge for policy makers Fixsen, D. et al (2011)
28 Challenge for researchers Providing policy makers with usable ideas and information that addresses current imperatives Developing user friendly frameworks and tools to evaluate the process and the outcomes Collaborating with practitioners in addressing pressing issues and problems Offering a critical and objective view that addresses the realities of practice
29 Centre for Effective Services W.K. Kellog Foundation Together 4 all Wisconsin online Information about logic models Information about theories of change Wisconsin templates Blueprint model programmes Information about evidence informed approaches CyferNet Search University of Arizona ActKnowledge AECF Community Change AECF Guide Learning for sustainability EPPI Centre Campbell Collaboration PROSPER Partnerships youngballymun Information about evidence-based and informed programmes Archways Information about programmes in Ireland and Northern Ireland Lifestart Foundation Evidence Network Centre for Evidence-Based Intervention CDI Tallaght Preparing for Life SAMSHA resources Routes to evidence in youth work?
30 John Bamber Principal Fellow – Community Development Centre for Effective Services