Assessing the impact of research and KTE activities Sandra Nutley.

Slides:



Advertisements
Similar presentations
Question-led systematic reviews: implications for searching Methods Festival 2006 Sandy Oliver Institute of Education, University of London.
Advertisements

Types and models of research impact Sandra Nutley University of St Andrews.
Tools for Policy Influence. RAPID Programme SMEPOL, Cairo, February, Practical Tools.
Introduction to the unit and mixed methods approaches to research Kerry Hood.
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
Key Messages National Riparian Lands Research & Development Program Assessing Community Capacity for Riparian Restoration.
Making Evidence-Based Education Policy Ontario Research Chairs in Public Policy Symposium Carol Campbell Ontario Institute for Studies in Education, University.
OECD/INFE High-level Principles for the evaluation of financial education programmes Adele Atkinson, PhD OECD With the support of the Russian/World Bank/OECD.
Knowledge translation tool: A workbook for the contextualization of global health systems guidance at the national or subnational level _ CPHA, Toronto.
Good Evaluation Planning – and why this matters Presentation by Elliot Stern to Evaluation Network Meeting January 16 th 2015.
Regional Trajectories to the Knowledge Economy: A Dynamic Model IKINET-EURODITE Joint Conference Warsaw, May 2006.
Role of RAS in the Agricultural Innovation System Rasheed Sulaiman V
Connections and issues in the European evaluation context Professor Murray Saunders President of the European Evaluation Society.
UKCDS/DFID/IDRC Workshop: Evaluation of Research Impact Margaret Macadam, Principal Impact Evaluation Manager, ESRC.
Deanne Gannaway Facilitating Change in Higher Education Practices.
Knowledge Translation Curriculum Module 1: An Introduction to KT Lesson 1 - Knowledge Translation: The Basics.
The evidence base on lifelong guidance A guide to key findings for effective policy and practice Tristram Hooley (Reader in Career.
V MEASURING IMPACT Kristy Muir Stephen Bennett ENACTUS November 2013.
Evaluating Socio-Economic Development - the ever-evolving EVALSED Guide Elliot Stern Presentation to Conference: Evaluating Public Interventions.
Knowledge Translation: A View from a National Policy Perspective KU-02 Conference Oxford, England July 2, 2002.
Isabel Vogel for ESPA. Isabel Vogel for ESPA, 24 January 2012 What is theory of change? Structured technique for understanding how and why a programme.
Bond.org.uk The Bond Effectiveness Programme: developing a sector wide framework for assessing and demonstrating effectiveness July 2011.
Urban-Nexus – Integrated Urban Management David Ludlow and Michael Buser UWE Sofia November 2011.
Impact on the research dance floor Line dance, tango or ceilidh? Drs. Astrid Wissenburg Economic and Social Research Council / Research Councils UK.
Assistive Technology Clinical Outcomes Research Management System (AT-CORMS) Tool Utilizing the International Classification of Functioning (ICF) Cognitive.
From Evidence to Action: Addressing Challenges to Knowledge Translation in RHAs The Need to Know Team Meeting May 30, 2005.
The Adaptation Policy Framework Bill Dougherty Stockholm Environment Institute – Boston Center Manila April 2004 An overview of the new UNDP-GEF product.
Building Capacity and Capability to Translate Knowledge into Action Dr Ann Wales Programme Director for Knowledge Management
1 OPHS FOUNDATIONAL STANDARD BOH Section Meeting February 11, 2011.
KT-EQUAL/ CARDI Workshop: ‘Lost in Translation’ 23 June 2011 Communicating research results to policy makers: A practitioner’s perspective.
Canadian Cancer Society Manitoba Division: Knowledge Exchange Network (KEN) & CancerCare Manitoba Manitoba Integrated Chronic Disease Primary Prevention.
Evaluation methods and tools (Focus on delivery mechanism) Jela Tvrdonova, 2014.
Accountability in Health Promotion: Sharing Lessons Learned Management and Program Services Directorate Population and Public Health Branch Health Canada.
1 Promoting Evidence-Informed Practice: The BASSC Perspective Michael J. Austin, PhD, MSW, MSPH BASSC Staff Director Mack Professor of Nonprofit Management.
ESPON 2020 Cooperation Programme EU Stakeholder consultation 27 March 2013 ESPON renewal, refocus and upgrade: Mission, Priority Axes, Specific Objectives,
A cluster randomised controlled trial to assess the effectiveness of knowledge translation strategies for obesity prevention Elizabeth Waters, Boyd Swinburn,
WHAT IS YOUNG LIVES? Young Lives is an international research project that is recording changes in child poverty over 15 years and the factors affecting.
NIPEC Organisational Guide to Practice & Quality Improvement Tanya McCance, Director of Nursing Research & Practice Development (UCHT) & Reader (UU) Brendan.
Monitoring and Evaluation of GeSCI’s Activities GeSCI Team Meeting 5-6 Dec 2007.
Integrating Knowledge Translation and Exchange into a grant Maureen Dobbins, RN, PhD SON, January 14, 2013.
A translational routemap for public health research Peter Craig Programme Manager, MRC PHSRN Knowledge Transfer Scotland, Heriot Watt University, 23 April.
Maria E. Fernandez, Ph.D. Associate Professor Health Promotion and Behavioral Sciences University of Texas, School of Public Health.
Understanding and improving research use in social services Sandra.Nutley.
Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26.
Training Resource Manual on Integrated Assessment Session UNEP-UNCTAD CBTF Process of an Integrated Assessment Session 2.
Hosted by: Funded by: Research into Reality Overcoming the challenges of knowledge exchange Foundation for the Future Research Conference 26 th – 28 th.
Fife Partnership…Strengthening Fife’s Future Research can make a difference! Workshop on communicating and disseminating research Facilitation - Chris.
Thinking about research impact contributing to society and the economy Tara Fenwick (reflecting ESRC influence but not representing it!)
Faisal Naru Head of Better Regulation DAI Europe Ltd November 2007 Washington London Johannesburg Ramallah RIA – An Art and not a Science.
Workshop on health systems research in low and middle income countries: the role of global health funders in the UK The Wellcome Trust, Gibbs Building,
Transforming Patient Experience: The essential guide
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Developing a Framework In Support of a Community of Practice in ABI Jason Newberry, Research Director Tanya Darisi, Senior Researcher
Department of Labour Presentation of ILO 93 rd Conference to the Portfolio Committee on Labour 8 November 2005.
Possible role of TNAs in the context of the implementation of the Technology Mechanism Wytze van der Gaast UNFCCC workshop on technology needs assessment.
Kathy Corbiere Service Delivery and Performance Commission
Technology Needs Assessments under GEF Enabling Activities “Top Ups” UNFCCC/UNDP Expert Meeting on Methodologies for Technology Needs Assessments
Strategies for making evaluations more influential in supporting program management and informing decision-making Australasian Evaluation Society 2011.
How to measure the impact of R&D on SD ? Laurence Esterle, MD, PhD Cermes and Ifris France Cyprus, 16 – 17 October L. ESTERLE Linking science and.
UNEP EIA Training Resource ManualTopic 14Slide 1 What is SEA? F systematic, transparent process F instrument for decision-making F addresses environmental.
John N. Lavis, MD, PhD Professor and Canada Research Chair in Knowledge Transfer and Exchange McMaster University Program in Policy Decision-Making McMaster.
Global Partnership for Enhanced Social Accountability (GPESA) December 19, 2011 World Bank.
SEL1 Implementing an assessment – the Process Session IV Lusaka, January M. Gonzales de Asis and F. Recanatini, WBI
Scottish Improvement Science Collaborating Centre Strengthening the evidence base for improvement science: lessons learned Dr Nicola Gray, Senior Lecturer,
IPP Network Meeting London, 7-8 November 2005 A report of the discussions Bob Ryder Department for Environment, Food & Rural Affairs (UK)
PHE Knowledge and Library Services – building an interactive evidence base Anne Brice Head of Knowledge & Library Services Public Health England.
ISSUES & CHALLENGES Adaptation, translation, and global application DiClemente, Crosby, & Kegler, 2009.
By Dr. Talat AnwarAdvisor Centre for Policy Studies, CIIT, Islamabad Centre for Policy Studies, CIIT, Islamabad
HEALTH IN POLICIES TRAINING
the role of global health funders in the UK
Presentation transcript:

Assessing the impact of research and KTE activities Sandra Nutley

Agenda for this session 1.Key approaches to assessing the use and impact of research and knowledge mobilisation activities 2.Common challenges and methodological issues 3.Summary of findings from existing assessment activities

Why assess research impact? Addressing accountability Assuring value for money Setting priorities Assisting learning Improving outcomes Summative or formative purposes?

Impact of research Impact of initiatives to increase research use Research use in ‘user’ communities Systematic reviews Centres promoting research use 1. Key approaches to assessing research impact Target KTE interventions Organisational research use Single studies Research programmes Policymakers Practitioners Media Forward tracking Tracking back Evaluation of initiatives

Forward tracking from research projects and programmes to use and impact Two common methods: Describe and quantify impacts of research - E.g. Payback model 5 categories of possible impact: –Knowledge production –Research capacity building –Policy or product development –Sector benefits –Wider societal benefits Describe and map networks and flows of knowledge and the effects of any interactions

Tracking back from decisions or behaviours to research influences on these Bibliometric studies of policy documents and practice guidelines User panels – surveys and workshops Ethnographic studies of communities of practice Economic impact evaluations – estimate economic impact of a policy and estimate extent of research influence on that policy

What methods are used in practice? Systematic review by Boaz et al 2008

Evaluations of initiatives to increase research impact Standard programme evaluation strategies: –Qualitative investigation of processes and perceived effects –Action research –Experimental and quasi-experimental methods Importance of strong theories of change & models of KTE to design and test interventions Need shared taxonomy of KTE strategies and interventions

KTE models Ward et al 2009: 63 different theories or models of KTE Distilled down to 5 components of the KTE process Connection between them is seen as interactive and multi-directional Aim of framework is to provide a foundation for gathering evidence about knowledge into action processes

Knowledge/Research Context Barriers/Supports ProblemInterventionsUtilisation Attributes of knowledge (relative advantage, complexity and compatibility) How it is synthesised and adapted Recognition of a need for action Organisational, individual, environmental and structural Distribution-type interventions Linkage-type interventions Process of using intervention: cycles of selection, tailoring, implementation & evaluation Conceptual, instrumental, political, procedural

Taxonomies of KTE strategies E.g. Mitton et al 2007 – 8 key strategies Face-to face exchange between researchers and ‘users’ Education sessions for users Networks and communities of practice Facilitated meetings between researchers and users Interactive, multidisciplinary workshops Capacity building in user organisations Web-based information, electronic communications Steering committees for research projects/programmes

RURU taxonomy Intervention types and underlying mechanisms, with focus on the latter Five key mechanisms: –Dissemination –Interaction –Social influence –Facilitation –Incentives and reinforcement

2. Common challenges and methodological issues What types of use/impact are of interest? When to assess impact? Importance of context – assessing actual or potential impacts? Dealing with attribution and additionality – constructing a convincing impact narrative Getting away from linear models research use/impact

What uses/impacts are of interest? Instrumental, conceptual or symbolic research use? Reach/awareness, impact on behaviour & service delivery, outcomes for service users? Positive and dysfunctional consequences?

When to assess impact? Impact often occurs far down the line, but impact trails grow fainter over time

Tools for analysing context Accounting for the difference between actual and potential impacts E.g. Conducive policy environments - research more likely to be used when information, ideology and interests coincide and institutions can support developments (Weiss)

Constructing a convincing impact narrative Dealing with attribution – the potential of contribution analysis

Slide source: Wimbush 2010

Conclusions on methods Case study methods to take account of differing types of research and contexts for impact Combination of quantitative and qualitative methods and indicators within case study approach Need for research impact theory/model to guide data collection and analysis Limited benefits from studying the impacts of individual research projects Dangers of generalising from case sampling – because distribution of research impact is likely to be uneven

3. Summary of findings from the UK ESRC’s impact assessment activities The most important drivers of impact are: –Established relationships and networks with user communities –Involving users at all stages with research –Well-planned user-engagement and KTE strategies –Portfolios of research activity that build reputations with research users –Good infrastructure and management support –The involvement of intermediaries and knowledge brokers as translators, amplifiers, network providers

RURU’s conclusions on generic features of effective KTE practices  Research must be translated - adaptation of findings to specific policy and practice contexts  Enthusiasm- of key individuals - personal contact is most effective  Contextual analysis - understanding and targeting specific barriers to, and enablers of, change  Credibility - strong evidence from trusted source, inc. endorsement from opinion leaders  Leadership - within research impact settings  Support - ongoing financial, technical & emotional support  Integration - of new activities with existing systems and activities

Implications for future research on research use and impact Much that we still need to know We need to move away from: –Poor documentation and under-evaluated KTE activities –Studies that focus only on the instrumental use of research –An assumption that research is used and applied mainly by individual practitioners –Studies that result only in a now familiar listing of barriers and enablers, especially where these are the barriers/enablers experienced by individual practitioners

Key messages Need to understand more about how research-based knowledge enters and flows through diverse policy, organisational and practice settings Need realistic assumptions about the nature and processes of research use and impact– these are many and complex No single model of research use is likely to be sufficient for all situations when impact is to be assessed Need to make choices about where and how to look for use and impact based on the purpose of the assessment

References Main reference Davies HTO and Nutley SM (2008) ‘Learning More about How Research- Based Knowledge Gets Used Guidance in the Development of New Empirical Research’, Working Paper for the WT Grant Foundation, New York Other useful references Boaz et al (2008) ‘Assessing the impact of research on policy: A review of the literature’, Kings College London/ PSI ESRC (2009) Taking Stock: A summary of ESRC’s work to evaluate the impact of research on policy and practice, ESRC (2010) Branching out: New direction in impact evaluation from the ESRC’s Evaluation Committee Meagher L, Lyall C and Nutley S (2008) ‘Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research’ Research Evaluation 17(3): Ward et al (2009) ‘Developing a framework for transferring knowledge into action: a thematic analysis of the literature’ Journal of Health Services Research and Policy 14(3):

For further information The following three slides summarise the main research questions identified by Davies and Nutley (2008) when they were asked to define the emerging research agenda on research use and impact for the WT Grant Foundation. These questions are categorised under the three headings of: Knowledge source, presentation and integration Context and connections Strategies and process

Knowledge source, presentation and integration What models of research supply and synthesis might better support knowledge integration by potential users? How do different kinds of messaging and messengers affect the use of research knowledge? What is the role of the web in providing access to existing research? To what extent are policymakers and service managers conducting their own in-house research using administrative and local data? How is new knowledge integrated into current ways of thinking and models of practice?

Context and connections What communities are involved in the field of interest? How are these connected? Are policy and practice communities networked and does research-based knowledge flow across these networks? How, where and under what circumstances do practitioners source new knowledge? What models of research brokerage or intermediary activities have the best potential for fostering research use? How can lay people, service users, and others contribute more fully to evidence-informed discussions? How are education and CPD connected to and supportive of knowledge accumulation and integration?

Strategies and processes Do different models of research use co-exist in different practice settings? What knowledge management strategies are used across the sector? What models of push, pull and linkage-exchange are in place? How are these evolving and connecting? What strategies for increasing research use and impact have the best evidence in support of them, and how might they be tested to consolidate our knowledge base in this area? How can we get new knowledge on the research- action process to influence the future actions of researchers, funders, intermediaries, policy/decision- makers, and practitioners/end users?