ILRI Process and Partnership for Pro-Poor Policy Change Methods for Assessing Policy Impact Process and Partnerships for Pro-Poor Policy Change, Project Initiation Workshop 1 ILRI, 21st February 2005

ILRI Process and Partnership for Pro-Poor Policy Change Workshop Outline Introduction to the RAPID Framework and ILRI/ODI Project Case Study Approach Episode Study Approach Outcome Mapping Approach RAPID Outcome Assessment (ROA) Approach Lunch Practical Sessions

ILRI Process and Partnership for Pro-Poor Policy Change Workshop Purpose & Objectives Objectives By the end of the workshop, participants will: understand the Process and Partnership for Pro-Poor Policy Change projects purpose and general approach have the opportunity to contribute their own suggestions to improve the project; understand, and have had the chance to try out the three key methods which will be used in the project; assess the usefulness of the approaches in their own work. Purpose: To familiarise the participants with the general approach and specific methods to be used in the SDP case study

An introduction to the RAPID Framework and ILRI/ODI Project

ILRI Process and Partnership for Pro-Poor Policy Change Definitions Research: any systematic effort to increase the stock of knowledge Policy: a purposive course of action followed by an actor or set of actors –Agendas / policy horizons –Official statements documents –Patterns of spending –Implementation processes –Activities on the ground

ILRI Process and Partnership for Pro-Poor Policy Change Policy Processes - Identify a policy problem - Commission research - Assess the results - Select the best policy - Establish the policy framework - Implement the policy - The problem is solved

ILRI Process and Partnership for Pro-Poor Policy Change Reality… The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies 1 Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in Agricultural and Rural Development, Heineman Educational Books, London 2 – Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges, International Service for National Agricultural Research (ISNAR) Research Report No 21

ILRI Process and Partnership for Pro-Poor Policy Change Existing theory 1.Linear model 2.Percolation model, Weiss 3.Tipping point model, Gladwell 4.Context, evidence, links framework, ODI 5.Policy narratives, Roe 6.Systems model (NSI) 7.External forces, Lindquist 8.Room for manoeuvre, Clay & Schaffer 9.Street level bureaucrats, Lipsky 10.Policy as social experiments, Rondinelli 11.Policy Streams & Windows, Kingdon 12.Disjointed incrementalism, Lindquist 13.The tipping point, Gladwell 14.Crisis model, Kuhn 15. Framework of possible thought, Chomsky 16.Variables for Credibility, Beach 17.The source is as important as content, Gladwell 18.Linear model of communication, Shannon 19.Interactive model, 20.Simple and surprising stories, Communication Theory 21.Provide solutions, Marketing Theory I 22.Find the right packaging, Marketing II 23.Elicit a response, Kottler 24.Translation of technology, Volkow 25.Epistemic communities 26.Policy communities 27.Advocacy coalitions etc, Pross 28.Negotiation through networks, Sebattier 29.Shadow networks, Klickert 30.Chains of accountability, Fine 31.Communication for social change, Rockefeller 32.Wheels and webs, Chapman & Fisher

ILRI Process and Partnership for Pro-Poor Policy Change Existing theory – a short list Policy narratives, Roe Systems of Innovation Model, (NSI) Room for manoeuvre, Clay & Schaffer Street level bureaucrats, Lipsky Policy as social experiments, Rondene Policy streams and policy windows, Kingdon Disjointed Incrementalism, Lindblom Social Epidemics, Gladwell The RAPID Framework

ILRI Process and Partnership for Pro-Poor Policy Change An Analytical Framework The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc. The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc External Influences Socio-economic and cultural influences, donor policies etc The links between policy and research communities – networks, relationships, power, competing discourses, trust, knowledge etc.

ILRI Process and Partnership for Pro-Poor Policy Change Case Studies Sustainable Livelihoods: The Evolution of DFID Policy The PRSP Initiative: Research in Multilateral Policy Change The adoption of Ethical Principles in Humanitarian Aid post Rwanda Animal Health Care in Kenya: Evidence fails to influence Policy

ILRI Process and Partnership for Pro-Poor Policy Change A Practical Framework External Influences political context evidence links Politics and Policymaking Media, Advocacy, Networking Research, learning & thinking Scientific information exchange & validation Policy analysis, & research Campaigning, Lobbying

ILRI Process and Partnership for Pro-Poor Policy Change What you need to know The external environment: Who are the key actors? What is their agenda? How do they influence the political context? The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re- packaging? Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?

ILRI Process and Partnership for Pro-Poor Policy Change What researchers need to do What researchers need to know What researchers need to do How to do it Political Context: Evidence Links Who are the policymakers? Is there demand for ideas? What is the policy process? What is the current theory? What are the narratives? How divergent is it? Who are the stakeholders? What networks exist? Who are the connectors, mavens and salesmen? Get to know the policymakers. Identify friends and foes. Prepare for policy opportunities. Look out for policy windows. Work with them – seek commissions Strategic opportunism – prepare for known events + resources for others Establish credibility Provide practical solutions Establish legitimacy. Present clear options Use familiar narratives. Build a reputation Action-research Pilot projects to generate legitimacy Good communication Get to know the others Work through existing networks. Build coalitions. Build new policy networks. Build partnerships. Identify key networkers, mavens and salesmen. Use informal contacts

ILRI Process and Partnership for Pro-Poor Policy Change Policy entrepreneurs Storytellers Engineers Networkers Fixers

ILRI Process and Partnership for Pro-Poor Policy Change Practical Tools Overarching Tools - The RAPID Framework - Using the Framework - The Entrepreneurship Questionnaire Context Assessment Tools - Stakeholder Analysis - Forcefield Analysis - Writeshops - Policy Mapping - Political Context Mapping Communication Tools - Communications Strategy - SWOT analysis - Message Design - Making use of the media Research Tools - Case Studies - Episode Studies - Surveys - Bibliometric Analysis - Focus Group Discussion Policy Influence Tools - Influence Mapping & Power Mapping - Lobbying and Advocacy - Campaigning: A Simple Guide - Competency self-assessment

ILRI Process and Partnership for Pro-Poor Policy Change Practical Application Within ODI Workshops for researchers, policy makers and activists. Advice to a DFID forest/ground water research project in India: –Less research –More communication –Developing champions in regional and national government –Local, Regional & National advocacy campaign

ILRI Process and Partnership for Pro-Poor Policy Change Further Information / Resources ODI Working Papers Bridging Research and Policy Book Meeting series Monograph Tools for Policy Impact RAPID Briefing Paper

ILRI Process and Partnership for Pro-Poor Policy Change Yes, but: It this its role? Global Public Good Research vs Policy Advocacy Probably needs to do both: How? Understand the political context Get the evidence & package it well Strategic networking / lobbying / campaigning Collaboration…. Can ILRI do it?

ILRI Process and Partnership for Pro-Poor Policy Change ILRI International Livestock Research Institute The New DfID funded Project Process and partnership for pro-poor policy change

ILRI Process and Partnership for Pro-Poor Policy Change ILRI International Livestock Research Institute Project Leaders: ODI / ILRI Key collaborators:ECAPAPA Case study collaborators in Kenya: –MoLFD / KARI –Range of NGOs & other SDP partners

ILRI Process and Partnership for Pro-Poor Policy Change Why would I be interested? ILRI International Livestock Research Institute Not all research is expected or intended to lead to policy change, but there may be; –Specific cases where research is expected to; provide evidence for policy change identify potential policies (or impact of) influence the policy making process (advocacy) –Cases where speculative research becomes relevant because of changes in circumstance

ILRI Process and Partnership for Pro-Poor Policy Change The project … ILRI International Livestock Research Institute Ideas for methods and approaches Lessons learnt from earlier activities Identification of appropriate communication tools

ILRI Process and Partnership for Pro-Poor Policy Change What will we be doing? ILRI International Livestock Research Institute Three case studies in three DIFFERENT countries –A project considered to have influenced policy change –A stream of research addressing a particular policy area –A clear policy change; New policy statement New law Irrefutable change in way something is done

ILRI Process and Partnership for Pro-Poor Policy Change What will we be doing? ILRI International Livestock Research Institute Three case studies –SDP and impact on changed view of informal milk trade –????

ILRI Process and Partnership for Pro-Poor Policy Change ILRI International Livestock Research Institute Discussion: –Can ILRI hope to influence pro-poor policy through research? –Any good case studies?

Case Study Approach

ILRI Process and Partnership for Pro-Poor Policy Change What is a Case Study? Definition: " A systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" Bromley (1990)

ILRI Process and Partnership for Pro-Poor Policy Change Why is it useful? An ideal methodology when a holistic, in- depth investigation is needed Designed to bring out the details from the viewpoint of the participants by using multiple sources of data Goal : to describe as accurately as possible the fullest, most complete description of the case.

ILRI Process and Partnership for Pro-Poor Policy Change Types of Case Study Types of case studies: Exploratory, Explanatory, Descriptive (Yin, 1993) Stake (1995) included three others: Intrinsic - when the researcher has an interest in the case; Instrumental - when the case is used to understand more than what is obvious to the observer; Collective - when a group of cases is studied.

ILRI Process and Partnership for Pro-Poor Policy Change Issues The unit of analysis is a critical factor Typically a system of action rather than an individual or group of individuals Tend to be selective, focusing on one or two issues that are fundamental to understanding the system being examined Case studies are multi-perspectival analyses The researcher considers not just the voice and perspective of the actors, but also of the relevant groups of actors and the interaction between them They give a voice to the powerless and voiceless.

ILRI Process and Partnership for Pro-Poor Policy Change Triangulation Data source triangulation, when the researcher looks for the data to remain the same in different contexts; Investigator triangulation, when several investigators examine the same phenomenon; Theory triangulation, when investigators with different view points interpret the same results; and Methodological triangulation, when one approach is followed by another, to increase confidence in the interpretation.

ILRI Process and Partnership for Pro-Poor Policy Change Applications To explain complex causal links between research and policy To describe the real-life context in which policy has been influenced by research To describe the policy influencing process itself To explore those situations in which the policy intervention being evaluated has no clear set of outcomes.

ILRI Process and Partnership for Pro-Poor Policy Change Process 1.Design the case study protocol: –determine the required skills –develop and review the protocol 2.Conduct the case study: –prepare for data collection –distribute questionnaire –conduct interviews 3.Analyze case study evidence: –analytic strategy 4.Develop conclusions, recommendations, and implications based on the evidence

Episode Study Approach

ILRI Process and Partnership for Pro-Poor Policy Change What is an Episode Study A study that focuses on a clear policy change and tracks back to assess what impact research had among the variety of issues that led to the policy change.

ILRI Process and Partnership for Pro-Poor Policy Change an excellent way of investigating the influence of research on policy Can focus on a single episode or comparative episodes. What is the purpose? Tracking backwards from policy change to any particular research which influence policy

ILRI Process and Partnership for Pro-Poor Policy Change The process of working backwards in time gives a more realistic view of the broad range of factors – other than research – that influence policy Tracking forward probably overemphasizes the importance of research Advantage

ILRI Process and Partnership for Pro-Poor Policy Change Policy processes are complex, multi-layered and change over time Often difficult to isolate the impact of research from other factors Actors may re-write history Important to seek the views of a wide range of informed stakeholders The process of preparing an episode study is iterative Key facts and / or inconsistencies need to be cross-checked with key informants Issues

ILRI Process and Partnership for Pro-Poor Policy Change 1.Identify a clear policy change. 2.Identify key Research Questions (draw on RAPID framework) 3.Explore how and why those policy decisions and practices took place 4.Assess the relative role of research in that process by drawing on the framework. Process

ILRI Process and Partnership for Pro-Poor Policy Change Apply the RAPID Framework External Influences political context evidence links Politics and Policymaking Media, Advocacy, Networking Research, learning & thinking Scientific information exchange & validation Policy analysis, & research Campaigning, Lobbying

ILRI Process and Partnership for Pro-Poor Policy Change Key Questions The external environment: Who are the key actors? What is their agenda? How do they influence the political context? The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re- packaging? Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?

ILRI Process and Partnership for Pro-Poor Policy Change review of the literature; interviews with key actors; capturing the authors own experience; and discussions at workshops. Methods Steps 3 and 4 can be done through a variety of methods:

Episode Study Examples

ILRI Process and Partnership for Pro-Poor Policy Change ­Professionalisation of Public Services. ­Structural Adjustment Collapse ­Paravet projects emerge. ­ITDG projects. ­Privatisation. ­ITDG Paravet network. ­Rapid spread in North. ­KVB letter (January 1998). ­Multistakeholder WSs new policies. ­Still not approved / passed! 1970s 1980s 1990s 2000s ­Professionalisation of Public Services. ­Structural Adjustment ­Privatisation ­ITDG Paravet network and change of DVS. ­KVB letter (January 1998). ­Multistakeholder WSs new policies. ITDG projects – collaborative research. The Hubl Study Dr Kajume Paravets in Kenya International Research

ILRI Process and Partnership for Pro-Poor Policy Change PRSPs – Political Context Widespread awareness of a problem with international development policy in late 90s Failure of SAPs (and Asian financial crisis) Mounting public pressure for debt relief Stagnation of Comprehensive Development Framework idea Diverging agendas (UK – Poverty, US – Governance) WB/IMF Annual General Meeting, Sept 1999

ILRI Process and Partnership for Pro-Poor Policy Change PRSPs – Evidence Long-term academic research informing new focus on poverty, participation, ownership, aid effectiveness etc Applied policy research: –ESAF reviews –HIPC review –SPA Working Groups –NGO research on debt Ugandas PEAP

ILRI Process and Partnership for Pro-Poor Policy Change PRSPs – Links WB, IMF, SPA, Bilaterals, NGOs all involved Formal an informal networks None of the players was more than two handshakes away from any of the others

Outcome Mapping

ILRI Process and Partnership for Pro-Poor Policy Change What is it? an integrated PM&E tool a system to think holistically & strategically about how we intend to achieve result an approach that focuses on changes in the behaviour, relationships or actions of partners (as outcomes) a methodology that characterizes and assesses the programs contributions to the achievement of outcomes an approach for designing in relation to the broader development context but assessing within your sphere of influence

ILRI Process and Partnership for Pro-Poor Policy Change Focus: On Behavioural Change

ILRI Process and Partnership for Pro-Poor Policy Change For a program to tell its performance story in outcome terms by: –articulating its goals and designing its activities –designing a monitoring system for assessing internal performance and outcomes of partners – setting a use-oriented evaluation plan How can it be used?

ILRI Process and Partnership for Pro-Poor Policy Change Focussing on changes in partners behaviour, relationships, or actions allows a program to: –measure results within its sphere of influence –obtain feedback about its efforts in order to improve its performance –take credit for its contributions to the achievement of outcomes –show progress towards outcomes Why use it?

ILRI Process and Partnership for Pro-Poor Policy Change Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work Vision: the broad human, social and environmental betterment we desire Mission: how we intend to contribute towards the achievement of the vision Boundary partners: individuals, groups and organisations with whom we interact directly to effect change Outcome challenges: changes behaviours of the boundary partners as identified by the vision Terminology

ILRI Process and Partnership for Pro-Poor Policy Change The Three Stages

ILRI Process and Partnership for Pro-Poor Policy Change Intentional Study Design

ILRI Process and Partnership for Pro-Poor Policy Change Provides a framework for a continuous monitoring of the initiative as a tool to achieve its outcomes. The program uses progress markers, a set of graduated indicators of behavioural change, identified in the intentional design stage to clarify direction with its primary partners and to monitor outcomes Performance Monitoring

ILRI Process and Partnership for Pro-Poor Policy Change Helps identify the evaluation priorities assessing the strategy at greater depth than the performance monitoring stage Evaluation Planning

ILRI Process and Partnership for Pro-Poor Policy Change Main Elements

RAPID Outcome Assessment

ILRI Process and Partnership for Pro-Poor Policy Change What is it? A Visual Tool Combines the outcome mapping concept within a case study & episode study approach Systematic approach to collecting information about changes in behaviour of key project partners that contributed to the policy change Assessment of the contribution of the project (programme, strategy, etc.) to observed changes in behaviour –and ultimately to the policy change

ILRI Process and Partnership for Pro-Poor Policy Change Approach 1.Describe policy environment at end 2.Describe policy environment at the beginning 3.Identify the key policy actors 4.Identify key boundary partners 5.Describe boundary partner behaviour at end 6.Describe boundary partner behaviour at beginning 7.Describe changes in BP behaviour 8.Describe changes in project (strategic/opportunistic) 9.Describe external influences 10.Determine level of impact of changes in project 11.Determine level of impact of external influences 12.Check through external interviews 13.Write report

ILRI Process and Partnership for Pro-Poor Policy Change Sources & Outputs 1.Literature review - Project background, progress, (published) achievements 2.Participatory workshop with staff (and BP) –Gather detailed information –Identify issues for further investigation 3.Interviews with key informants to: –Triangulate the result of the workshop, –Fill the gaps of information –Clarify causality 4.Report Writing –Visual and Narrative

ILRI Process and Partnership for Pro-Poor Policy Change ROA Terms and Definitions Boundary partners: individuals, groups and organisations with whom we interact directly to effect change. Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work. Behaviours: the way we or our boundary partners do or think about things.

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 1.Describe the policy environment at the end of the project

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 2. Describe the policy environment at the beginning of the project

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 3/4. Identify key policy actors and boundary partners (that were influential at end)

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 5. Describe the behaviours of the boundary partners that contributed to the change in the policy environment or policy

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 6.Describe the behaviours of the boundary partners at the beginning of the project

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 7. Map the key changes in behaviour for each boundary partner from the start of the project

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 8. Map the key changes in the project including organisational changes, outputs and changes in behaviour during the same period.

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 9.Map the external influences including the actions f strategic partners and other exogenous factors during the same period

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 10. Determine the level of impact/influence of the project on the changes in behaviour of the boundary partners

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 11. Determine the level of impact/influence of external influences on the changes in behaviour of the boundary partners and the project

ILRI Process and Partnership for Pro-Poor Policy Change The key steps of the ROA framework 12. Refine conclusions with in-depth interviews and assess the real contribution of the project on the policy change