Presentation on theme: "Real world approaches to assessing the impact of research on policy"— Presentation transcript:
1 Real world approaches to assessing the impact of research on policy Annette BoazKing’s College London23rd September 2010My interest in the use of research knowledge – interest in critical appraisal of qualitative research – searching for a study - experience at DH with research at bottom of a filing cabinet.Background in politics big believer in conspiracy – not now. Politics has a significant role to play, but lots we can do to improve impact
2 OutlineModels of the relationship between research policy and practiceRationales for assessing impactWhat can we learn from the literature?10 research impact assessment case studiesConsider the implications for SSH researchWanted to come at translational research from a slightly different angle
3 Models Relationship – two communities Linear – relay race Engineering DiffusionBroadly speaking three different types of model of the relationship between research, policy and practice:The first, a linear model (dissemination, diffusion of innovations etc.)The second, one in which there are two communities that aim to come together around, for e.g. research advisory groupsThe third, where researchers work with potential users to produce research togetherSystems – co-production
4 Activities designed to promote research impact Research briefingsAdvisory groupsKnowledge brokersCo-locating researchers and policy makersParticipation in NetworksJoint research projectsWe tend to be engaged in a wide range of activities which are designed to promote use – here are the most common onesSeminars/ presentations
5 Models Relationship – two communities Linear – relay race Advisory groupsModelsResearch briefingsRelationship – two communitiesLinear – relay raceJoint research projectsSystems – co-productionThese activities can be mapped onto the models. For e.g.
6 Effective knowledge transfer depends onthe nature of the research and its utilitythe personal characteristics of both researchers and usersthe links between researchers and usersthe context for the application of the researchSee Nutley, Walter and Davies, 2007, pp 81-83,See also Overseas Development Institute
7 Why assess the impact of our research? Moderating between stakeholdersLearningAccountabilityWhy assess the impact of our research?In its quest for excellence, the European Commission must and will encourage, promoteand support every effort to understand and monitor the quality of research at universities. 1Promoting organizational achievementsPerformance1Commissioner Janez Potočnik, Assessing Europe’s University-Based Research, 2010
8 Review question: How do you (best) evaluate the impact of research on policy? Explored the websites of 30 organizationsHand search of selected journals and reference listsSearch of SPRU and IDS library cataloguesSearch of 10 research databasesUK and international sourcesAn example is the work done by the Health Economics Research Group based at Brunel UniversityInnovating in terms of methods for analysis and theoretical frameworksFamous study of a prominent Diabetes researcher George Alberti – tracking his papers from 1981 and interviewing members of the team. Found much from going beyond bibliometrics. e.g. The great things his students and other co-investigators went on to achieve.
9 Initiatives to increase research use Focusing the review on policy useResearch UsersResearchSingle studyPolicy makersResearch programmesSystematic reviewsInitiatives to increase research use
10 Other includes: Brazil, Denmark, Spain and Norway
11 Methods for measuring research impact Recently conducted a review for Defra of methods for assessing research impact on policy – drawing on 156 papers.A few points to note:Main methods: semi structured interviews, case studies, documentary analysisTo note: people talk about bibliometric analysis a lot more than they do itPanel reviews: popular in Europe for their consensus building qualitiesLots of health research in the area – most famous Knowledge into action, but no systematic reviews and meta-analyses, randomised controlled trials, Cohort studies, Case-control studies, cross sectional surveys
12 How do people evaluate the impact of research programmes? Wide range of methods and approaches (often in combination):· Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations· Quantitative methods: surveys, bibliometrics and patent/new technology tracking· Panels and peer review· Workshops and focus groups· Process tracking
14 Ten impact assessment case studies The Foundation for Research, Science and Technology (New Zealand)French Ministry of Ecology, Energy, Sustainable Development and the SeaLand and Water AustraliaCGIARStockholm Environment InstituteTen impact assessment case studiesThe Irish Environmental Protection AgencyDefraEnvironment Agency (England and Wales)Finnish Environment InstituteSwedish Environmental Protection Agency
15 The Irish Environmental Protection Agency An example is the work done by the Health Economics Research Group based at Brunel UniversityInnovating in terms of methods for analysis and theoretical frameworksFamous study of a prominent Diabetes researcher George Alberti – tracking his papers from 1981 and interviewing members of the team. Found much from going beyond bibliometrics. e.g. The great things his students and other co-investigators went on to achieve.
16 Finnish Environmental Institute Embedded approach: Ten year independent evaluation, four year strategic evaluations, internal annual evaluations, annual researcher self evaluation Goal: how to increase impact Methods: multi-method
17 Key themes: Diverse of objectives, scope and scale Types of approach Generic challengesChallenges specific to environmental research
18 What are the implications for SSH research in Latin America and Europe?
19 8 key questions (when designing an impact evaluation) 1. What is your conceptual framework?. What are the outcomes of interest?. What methods will best explore the outcomes of interest?. How do you address attribution?. What is the direction of travel for the evaluation?. Is this a mixed method approach, providing scope for triangulation?. Will the methods selected capture context and the complexity?. When might be the best time to conduct the evaluation?These need to be considered in the light of the time, skills and resources available for the evaluation.See Boaz et al 2009
20 My contact details: Annette.firstname.lastname@example.org Some useful references and websites:Boaz, A, Fitzpatrick, S and Shaw, B (2009) Assessing the impact of research on policy: a literature review. Science and Public Policy, 36(4):Nutley S, Walter I, and Davies H (2007) Using evidence: How research can inform public services. Bristol: The Policy Press.O’Leary, G, Boyle, R, Donlon, B and Sheils, L (2008) Providing a framework for accountability and learning in environmental research. Administration. 5 (4):-The Overseas Development Institute RAPID programme:- Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J (2007) Policy and practice impacts of research funded by the ESRC. A case study of the Future of Work programme, approach and analysis. Cambridge: RAND Europe.My contact details:
21 Managing Editors: Annette Boaz, Division of Health and Social Care Research, King's College London, UK and David Gough, Institute of Education, University of London, UK Volume 6, 2010, 4 issues: January, May, August and November