Presentation on theme: "Tools to Promote the Use of Research-Based Evidence in Policy"— Presentation transcript:
1 Tools to Promote the Use of Research-Based Evidence in Policy ODI / INASP Research-Policy Symposium,Oxford, 16th/17th Nov 2006John Young
2 Overview Definitions Policy processes Researchers, Policy Makers & NGOsSome theoryA useful framework?How to do itSome examplesFurther information
3 DefinitionsResearch: “any systematic effort to increase the stock of knowledge”Policy: a “purposive course of action followed by an actor or set of actors”Evidence: “the available information supporting or otherwise a belief or proposition”Evidence-based Policy: “public policy informed by rigorously established evidence”.
4 Policy Processes Identify the problem Commission research Analyse the resultsChoose the best optionEstablish the policyImplement the policyEvaluation
5 Monitoring and Evaluation Policy Implementation Policy ProcessesCabinetDonorsMonitoring and EvaluationAgendaSettingDecisionMakingPolicy ImplementationPolicy FormulationParliamentCivil SocietyMinistriesPrivate Sector
6 Chronic Poverty in Uganda Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004 (
7 …in reality…“The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies 1”“Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2”“Research is more often regarded as the opposite of action rather than a response to ignorance”31 Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy inAgricultural and Rural Development, Heineman Educational Books, London2 Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges,International Service for National Agricultural Research (ISNAR) Research Report No 213 Surr (2003), DFID Research Review
8 But Policy makers are……practically incapable of using research-based evidence because of the 5 Ss…SpeedSuperficialitySpinSecrecyScientific IgnoranceVincent Cable – Lib. Democrat MP & Shadow Minister of FinanceMore at:
9 Factors influencing policy making Experience & ExpertisePragmatics & ContingenciesJudgementEvidenceLobbyists & Pressure GroupsResourcesHabits & TraditionValues and Policy ContextSource: Phil Davies Impact to Insight Meeting, ODI, 2005
10 Different Notions of Evidence ‘Scientific’ (Context free)Proven empiricallyTheoretically drivenAs long as it takesCaveats and qualificationsResearchers’ EvidenceColloquial (Contextual)Anything that seems reasonablePolicy relevantTimelyClear MessagePolicy Makers’ EvidenceSource: Phil Davies Impact to Insight Meeting, ODI, 2005
13 Key problems and solutions ExternalDifficult Political ContextsCampaignsBoomerangsPolicy PilotsInternalWeak understanding of political contextsRigorous context assessmentsWeak engagementBetter strategies for engagement at all parts of the policy cycleInadequate use of evidenceCollecting the right evidence for each situation (qualitative vs quantitative etc)Weak communicationBetter communication: publications, events, face-to-faceIsolationCollaboration with other CSOs, donors and government agencies: NetworksCapacity constraints“Systemic” capacity-building: of organisations and networks within their contexts
14 Existing theory X Linear model Percolation model, Weiss Tipping point model, Gladwell‘Context, evidence, links’ framework, ODIPolicy narratives, RoeSystems model (NSI)External forces, Lindquist‘Room for manoeuvre’, Clay & Schaffer‘Street level bureaucrats’, LipskyPolicy as social experiments, RondinelliPolicy Streams & Windows, KingdonDisjointed incrementalism, LindquistThe ‘tipping point’, GladwellCrisis model, Kuhn‘Framework of possible thought’, ChomskyVariables for Credibility, BeachThe source is as important as content, GladwellLinear model of communication, ShannonInteractive model,Simple and surprising stories, Communication TheoryProvide solutions, Marketing IFind the right packaging, Marketing IIElicit a response, KottlerTranslation of technology, VolkowEpistemic communitiesPolicy communitiesAdvocacy coalitions etc, ProssNegotiation through networks, SebattierShadow networks, KlickertChains of accountability, FineCommunication for social change, RockefellerWheels and webs, Chapman & Fisher
15 Existing theory – a short list Policy narratives, RoeSystems of Innovation Model, (NSI)‘Room for manoeuvre’, Clay & Schaffer‘Street level bureaucrats’, LipskyPolicy as social experiments, RondenePolicy streams and policy windows, KingdonDisjointed Incrementalism, LindblomSocial Epidemics, GladwellThe RAPID Framework
16 An Analytical Framework External InfluencesSocio-economic and cultural influences,donor policies etcThe political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.The links between policyand research communities – networks, relationships, power, competing discourses, trust, knowledge etc.The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc
17 Case Studies Detailed: Summary Sustainable Livelihoods Poverty Reductions Strategy ProcessesEthical Principles in Humanitarian AidAnimal Health Care in KenyaDairy Policy in KenyaPlant Genetic ResourcesSummaryGDN x 50CSPP x 20Good news case studies x 5Mental health in the UK
18 Animal Healthcare in Kenya 1970s1980s1990s2000sProfessionalisation of Public Services.Structural Adjustment → collapse of services.Paravet projects emerge.ITDG projects.Privatisation.ITDG Paravet network.Rapid spread in North.KVB letter (January 1998).Multistakeholder WSs → new policies.Still not approved / passed!Professionalisation of Public Services.Structural AdjustmentPrivatisationITDG Paravet network and change of DVS.KVB letter (January 1998).Multistakeholder WSs → new policies.International ResearchITDG projects – collaborative research.The Hubl StudyDr Kajume
19 A Practical Framework External Influences political context evidence Politics and PolicymakingCampaigning, LobbyingPolicy analysis, & researchMedia,Advocacy, NetworkingScientific information exchange & validationResearch, learning & thinkingevidencelinks
20 The Key Questions The external environment: Who are the key actors? What is their agenda?How do they influence the political context?The evidence:Is it there?Is it relevant?Is it practically useful?Are the concepts new?Does it need re-packaging?The political context:Is there political interest in change?Is there room for manoeuvre?How do they perceive the problem?Links:Who are the key actors?Are there existing networks?How best to transfer the information?The media?Campaigns?
21 What you need to do What you need to know What you need to do How to do itPolitical Context:EvidenceLinksGet to know the policymakers.Identify friends and foes.Prepare for policy opportunities.Look out for policy windows.Work with them – seek commissionsStrategic opportunism – prepare for known events + resources for othersWho are the policymakers?Is there demand for ideas?What is the policy process?Build a reputationAction-researchPilot projects to generate legitimacyGood communicationEstablish credibilityProvide practical solutionsEstablish legitimacy.Present clear optionsUse familiar narratives.What is the current theory?What are the narratives?How divergent is it?Get to know the othersWork through existing networks.Build coalitions.Build new policy networks.Build partnerships.Identify key networkers, mavens and salesmen.Use informal contactsWho are the stakeholders?What networks exist?Who are the connectors, mavens and salesmen?
23 Policy Engagement Tools Overarching Tools- The RAPID Framework- Using the Framework- The Entrepreneurship QuestionnaireContext Assessment Tools- Stakeholder Analysis- Forcefield Analysis- Writeshops- Policy Mapping- Political Context MappingCommunication Tools- Communications Strategy- SWOT analysis- Message Design- Making use of the mediaResearch Tools- Case Studies- Episode Studies- Surveys- Bibliometric Analysis- Focus Group DiscussionPolicy Influence Tools- Influence Mapping & Power Mapping- Lobbying and Advocacy- Campaigning: A Simple Guide- Competency self-assessment
24 Groundwater in Indiato maximise impact of DFID forest/ ground water research project in IndiaResearchers, policy makers and activistsUsed framework to analyse factors in water sector in IndiaDeveloped strategy for final phase:Less researchMore communicationDeveloping champions in regional and national governmentLocal, Regional & National advocacy campaign
26 Policy Processes in DFID Looking at internal policy processes – what works in DFID.Small, informal workshop with 7 staff.Participatory pair-wise ranking of factors influencing the success of 8 policy processes.Worked quite well.In DFID - agendas and processes rather than documents are key
27 Organisational Capacity Knowledge, Attitudes and Practices:The entrepreneurship questionnaireTraining & mentoring etcKnowledge ManagementOrganisational developmentFinance, admin & personnel systemsStrategic (action & business) planningFundraising & reportingBuilding an organisational profileCommunications, Public Affairs and the MediaStruyk, 2002, Local Governance Institute, Open Society Network, Budapest
28 Learning & KM Learning before, during & after Collaboration Tools Peer AssistReflective InquiryAfter-Action ReviewCollaboration ToolsE-discussionsShared workspacesInformation management toolsIntelligent search enginesIncentives
29 Increasing Research Use Increasing the pull for evidenceRequire the publication of the evidence baseRequire spending bids to provide evidence baseSubmit government analysis to external expert scrutinyProvide open access to informationFacilitating better evidence useEncourage better collaboration across analytical servicesCo-locate policy makers and internal analystsIntegrate analytical staff at all stagesLink R&D strategies to departmental business plansCast external researchers more as partners than as contractorsSecond more university staff into governmentTrain staff in evidence useSource: Abstracted from PIU 2000, Bullock et al (2001)
30 UK Government “Tools” Overview and Checklist 1. Impact Assessment and Appraisal: guidance checklist for policy makers.Strategy and Policy Evaluation2. Strategy Survival Guide3. Magenta Book: Guidance notes on Policy Evaluation4. Green Book: Appraisal and evaluation in Central Government5. Regulatory Impact Assessment (RIA)Ensuring Key Perspectives are Included6. Incorporating regional perspectives into policy making toolkit (Subnational)7. International Comparisons in Policy Making Toolkit8. Gender Impact Assessment: a framework for gender mainstreaming9. Managing risks to the public: Appraisal GuidanceTesting Policy Ideas10. Policy PilotsPublic-orientated Tools11. Concern Assessment Tool12. Community Engagement How to Guide13. Connecting with Users and CitizensGetting Better Advice and Evidence14. Expert Advisory Bodies for Policymakers15. Improving Standards of Qualitative Research
31 Regulatory Impact Assessment Aims to improve causality between evidence and adviceA process that must be completed for all proposed policy changes:Purpose / intended effectPolicy problemOptions & evidenceImpact & evidenceResults of consultationPublishedMore at: office.gov.uk/regulation/ria/ria_guidance/index.asp
32 Assessing Qualitative Research A framework developed by the Cabinet Office / National Centre for Social ResearchBased on review of 29 existing frameworks (esp from medical/health)Four principles. Research should be:contributory;defensible in design;rigourous in conduct;credible in claim.18 Questions, with criteriaRecognises need for:Policymakers to have necessary expertiseNew approaches to research
33 Assessing Research Quality How credible are the findings?How has knowledge or understanding been extended by the research?How well does the evaluation address its original aims and purpose?How well is the scope for drawing wider inference explained?How clear is the basis of evaluative appraisal?How defensible is the research design?How well defended are the sample design/target selection of cases/documents?How well is the eventual sample composition and coverage described?How well was the data collection carried out?How well has the approach to and formulation of analysis been conveyed?How well are the contexts of data sources retained and portrayed?How well has diversity of perspective and content been explored?How well has detail, depth and complexity of the data been conveyed?How clear are the links between data, interpretation and conclusions - i.e how well can the route to any conclusions be seen?How clear and coherent is the reporting?How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the evaluation?What evidence is there of attention to ethical issues?How adequately has the research process been documented?More at:
34 Conclusions From whichever side, you need Clarity of intent A systematic approachTo establish the right incentives / cultureTo establish the right systemsTo spend moreTo engage with all stakeholdersTo produce the right products for the right people at the right time
35 Further Information ODI – www.odi.org.uk RAPID - www.odi.org.uk/rapid PublicationsWorking PapersBriefing PapersBooksCase StudiesWorkshops and SeminarsTools and Toolkits
36 Mini-WorkshopsTools to understand the political and policy context and engage with policy makers. (Downstairs)Research and project tools to generate evidence which will convince policymakers.Ways in which local content reaches policy makers throughout the information chain.