1. Tools to Promote the Use of Research-Based Evidence in Policy
ODI / INASP Research-Policy Symposium,
Oxford, 16th/17th Nov 2006
John Young

2. Overview Definitions Policy processes
Researchers, Policy Makers & NGOs
Some theory
A useful framework?
How to do it
Some examples
Further information

3. Definitions
Research: “any systematic effort to increase the stock of knowledge”
Policy: a “purposive course of action followed by an actor or set of actors”
Evidence: “the available information supporting or otherwise a belief or proposition”
Evidence-based Policy: “public policy informed by rigorously established evidence”.

4. Policy Processes Identify the problem Commission research
Analyse the results
Choose the best option
Establish the policy
Implement the policy
Evaluation

5. Monitoring and Evaluation Policy Implementation
Policy Processes
Cabinet
Donors
Monitoring and Evaluation
Agenda
Setting
Decision
Making
Policy Implementation
Policy Formulation
Parliament
Civil Society
Ministries
Private Sector

6. Chronic Poverty in Uganda
Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004 (

7. …in reality…
“The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies 1”
“Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2”
“Research is more often regarded as the opposite of action rather than a response to ignorance”3
1 Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in
Agricultural and Rural Development, Heineman Educational Books, London
2 Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges,
International Service for National Agricultural Research (ISNAR) Research Report No 21
3 Surr (2003), DFID Research Review

8. But Policy makers are…
…practically incapable of using research-based evidence because of the 5 Ss…
Speed
Superficiality
Spin
Secrecy
Scientific Ignorance
Vincent Cable – Lib. Democrat MP & Shadow Minister of Finance
More at:

9. Factors influencing policy making
Experience & Expertise
Pragmatics & Contingencies
Judgement
Evidence
Lobbyists & Pressure Groups
Resources
Habits & Tradition
Values and Policy Context
Source: Phil Davies Impact to Insight Meeting, ODI, 2005

10. Different Notions of Evidence
‘Scientific’ (Context free)
Proven empirically
Theoretically driven
As long as it takes
Caveats and qualifications
Researchers’ Evidence
Colloquial (Contextual)
Anything that seems reasonable
Policy relevant
Timely
Clear Message
Policy Makers’ Evidence
Source: Phil Davies Impact to Insight Meeting, ODI, 2005

11. How CSOs influence Policy

12. Obstacles for CSO Research

13. Key problems and solutions
External
Difficult Political Contexts
Campaigns
Boomerangs
Policy Pilots
Internal
Weak understanding of political contexts
Rigorous context assessments
Weak engagement
Better strategies for engagement at all parts of the policy cycle
Inadequate use of evidence
Collecting the right evidence for each situation (qualitative vs quantitative etc)
Weak communication
Better communication: publications, events, face-to-face
Isolation
Collaboration with other CSOs, donors and government agencies: Networks
Capacity constraints
“Systemic” capacity-building: of organisations and networks within their contexts

14. Existing theory X Linear model Percolation model, Weiss
Tipping point model, Gladwell
‘Context, evidence, links’ framework, ODI
Policy narratives, Roe
Systems model (NSI)
External forces, Lindquist
‘Room for manoeuvre’, Clay & Schaffer
‘Street level bureaucrats’, Lipsky
Policy as social experiments, Rondinelli
Policy Streams & Windows, Kingdon
Disjointed incrementalism, Lindquist
The ‘tipping point’, Gladwell
Crisis model, Kuhn
‘Framework of possible thought’, Chomsky
Variables for Credibility, Beach
The source is as important as content, Gladwell
Linear model of communication, Shannon
Interactive model,
Simple and surprising stories, Communication Theory
Provide solutions, Marketing I
Find the right packaging, Marketing II
Elicit a response, Kottler
Translation of technology, Volkow
Epistemic communities
Policy communities
Advocacy coalitions etc, Pross
Negotiation through networks, Sebattier
Shadow networks, Klickert
Chains of accountability, Fine
Communication for social change, Rockefeller
Wheels and webs, Chapman & Fisher

15. Existing theory – a short list
Policy narratives, Roe
Systems of Innovation Model, (NSI)
‘Room for manoeuvre’, Clay & Schaffer
‘Street level bureaucrats’, Lipsky
Policy as social experiments, Rondene
Policy streams and policy windows, Kingdon
Disjointed Incrementalism, Lindblom
Social Epidemics, Gladwell
The RAPID Framework

16. An Analytical Framework
External Influences
Socio-economic and cultural influences,
donor policies etc
The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.
The links between policy
and research communities – networks, relationships, power, competing discourses, trust, knowledge etc.
The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc

17. Case Studies Detailed: Summary Sustainable Livelihoods
Poverty Reductions Strategy Processes
Ethical Principles in Humanitarian Aid
Animal Health Care in Kenya
Dairy Policy in Kenya
Plant Genetic Resources
Summary
GDN x 50
CSPP x 20
Good news case studies x 5
Mental health in the UK

18. Animal Healthcare in Kenya
1970s
1980s
1990s
2000s
Professionalisation of Public Services.
Structural Adjustment → collapse of services.
Paravet projects emerge.
ITDG projects.
Privatisation.
ITDG Paravet network.
Rapid spread in North.
KVB letter (January 1998).
Multistakeholder WSs → new policies.
Still not approved / passed!
Professionalisation of Public Services.
Structural Adjustment
Privatisation
ITDG Paravet network and change of DVS.
KVB letter (January 1998).
Multistakeholder WSs → new policies.
International Research
ITDG projects – collaborative research.
The Hubl Study
Dr Kajume

19. A Practical Framework External Influences political context evidence
Politics and Policymaking
Campaigning, Lobbying
Policy analysis, & research
Media,
Advocacy, Networking
Scientific information exchange & validation
Research, learning & thinking
evidence
links

20. The Key Questions The external environment: Who are the key actors?
What is their agenda?
How do they influence the political context?
The evidence:
Is it there?
Is it relevant?
Is it practically useful?
Are the concepts new?
Does it need re-packaging?
The political context:
Is there political interest in change?
Is there room for manoeuvre?
How do they perceive the problem?
Links:
Who are the key actors?
Are there existing networks?
How best to transfer the information?
The media?
Campaigns?

21. What you need to do What you need to know What you need to do
How to do it
Political Context:
Evidence
Links
Get to know the policymakers.
Identify friends and foes.
Prepare for policy opportunities.
Look out for policy windows.
Work with them – seek commissions
Strategic opportunism – prepare for known events + resources for others
Who are the policymakers?
Is there demand for ideas?
What is the policy process?
Build a reputation
Action-research
Pilot projects to generate legitimacy
Good communication
Establish credibility
Provide practical solutions
Establish legitimacy.
Present clear options
Use familiar narratives.
What is the current theory?
What are the narratives?
How divergent is it?
Get to know the others
Work through existing networks.
Build coalitions.
Build new policy networks.
Build partnerships.
Identify key networkers, mavens and salesmen.
Use informal contacts
Who are the stakeholders?
What networks exist?
Who are the connectors, mavens and salesmen?

22. Policy entrepreneurs
Storytellers
Networkers
Engineers
Fixers

23. Policy Engagement Tools
Overarching Tools
- The RAPID Framework
- Using the Framework
- The Entrepreneurship Questionnaire
Context Assessment Tools
- Stakeholder Analysis
- Forcefield Analysis
- Writeshops
- Policy Mapping
- Political Context Mapping
Communication Tools
- Communications Strategy
- SWOT analysis
- Message Design
- Making use of the media
Research Tools
- Case Studies
- Episode Studies
- Surveys
- Bibliometric Analysis
- Focus Group Discussion
Policy Influence Tools
- Influence Mapping & Power Mapping
- Lobbying and Advocacy
- Campaigning: A Simple Guide
- Competency self-assessment

24. Groundwater in India
to maximise impact of DFID forest/ ground water research project in India
Researchers, policy makers and activists
Used framework to analyse factors in water sector in India
Developed strategy for final phase:
Less research
More communication
Developing champions in regional and national government
Local, Regional & National advocacy campaign

25. SMEPOL Project Egypt Policy Process Mapping RAPID Framework
Stakeholder Analysis
Force-Field Analysis
SWOT
Action Planning

26. Policy Processes in DFID
Looking at internal policy processes – what works in DFID.
Small, informal workshop with 7 staff.
Participatory pair-wise ranking of factors influencing the success of 8 policy processes.
Worked quite well.
In DFID - agendas and processes rather than documents are key

27. Organisational Capacity
Knowledge, Attitudes and Practices:
The entrepreneurship questionnaire
Training & mentoring etc
Knowledge Management
Organisational development
Finance, admin & personnel systems
Strategic (action & business) planning
Fundraising & reporting
Building an organisational profile
Communications, Public Affairs and the Media
Struyk, 2002, Local Governance Institute, Open Society Network, Budapest

28. Learning & KM Learning before, during & after Collaboration Tools
Peer Assist
Reflective Inquiry
After-Action Review
Collaboration Tools
E-discussions
Shared workspaces
Information management tools
Intelligent search engines
Incentives

29. Increasing Research Use
Increasing the pull for evidence
Require the publication of the evidence base
Require spending bids to provide evidence base
Submit government analysis to external expert scrutiny
Provide open access to information
Facilitating better evidence use
Encourage better collaboration across analytical services
Co-locate policy makers and internal analysts
Integrate analytical staff at all stages
Link R&D strategies to departmental business plans
Cast external researchers more as partners than as contractors
Second more university staff into government
Train staff in evidence use
Source: Abstracted from PIU 2000, Bullock et al (2001)

30. UK Government “Tools” Overview and Checklist
1. Impact Assessment and Appraisal: guidance checklist for policy makers.
Strategy and Policy Evaluation
2. Strategy Survival Guide
3. Magenta Book: Guidance notes on Policy Evaluation
4. Green Book: Appraisal and evaluation in Central Government
5. Regulatory Impact Assessment (RIA)
Ensuring Key Perspectives are Included
6. Incorporating regional perspectives into policy making toolkit (Subnational)
7. International Comparisons in Policy Making Toolkit
8. Gender Impact Assessment: a framework for gender mainstreaming
9. Managing risks to the public: Appraisal Guidance
Testing Policy Ideas
10. Policy Pilots
Public-orientated Tools
11. Concern Assessment Tool
12. Community Engagement How to Guide
13. Connecting with Users and Citizens
Getting Better Advice and Evidence
14. Expert Advisory Bodies for Policymakers
15. Improving Standards of Qualitative Research

31. Regulatory Impact Assessment
Aims to improve causality between evidence and advice
A process that must be completed for all proposed policy changes:
Purpose / intended effect
Policy problem
Options & evidence
Impact & evidence
Results of consultation
Published
More at: office.gov.uk/regulation/ria/ria_guidance/index.asp

32. Assessing Qualitative Research
A framework developed by the Cabinet Office / National Centre for Social Research
Based on review of 29 existing frameworks (esp from medical/health)
Four principles. Research should be:
contributory;
defensible in design;
rigourous in conduct;
credible in claim.
18 Questions, with criteria
Recognises need for:
Policymakers to have necessary expertise
New approaches to research

33. Assessing Research Quality
How credible are the findings?
How has knowledge or understanding been extended by the research?
How well does the evaluation address its original aims and purpose?
How well is the scope for drawing wider inference explained?
How clear is the basis of evaluative appraisal?
How defensible is the research design?
How well defended are the sample design/target selection of cases/documents?
How well is the eventual sample composition and coverage described?
How well was the data collection carried out?
How well has the approach to and formulation of analysis been conveyed?
How well are the contexts of data sources retained and portrayed?
How well has diversity of perspective and content been explored?
How well has detail, depth and complexity of the data been conveyed?
How clear are the links between data, interpretation and conclusions - i.e how well can the route to any conclusions be seen?
How clear and coherent is the reporting?
How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the evaluation?
What evidence is there of attention to ethical issues?
How adequately has the research process been documented?
More at:

34. Conclusions From whichever side, you need Clarity of intent
A systematic approach
To establish the right incentives / culture
To establish the right systems
To spend more
To engage with all stakeholders
To produce the right products for the right people at the right time

35. Further Information ODI – www.odi.org.uk RAPID - www.odi.org.uk/rapid
Publications
Working Papers
Briefing Papers
Books
Case Studies
Workshops and Seminars
Tools and Toolkits

36. Mini-Workshops
Tools to understand the political and policy context and engage with policy makers. (Downstairs)
Research and project tools to generate evidence which will convince policymakers.
Ways in which local content reaches policy makers throughout the information chain.