1. Monitoring and Evaluation of Influence Enrique Mendizabal June 2011

2. Outline The skeptical view Why monitor and evaluate? What can we monitor and evaluate? Types of influencing Methods and tools for each A suggestion Work on your own

3. The skeptics 3

4. (on think tanks) some experts say: From Braml (2004) Kent Weaver: it is hard to determine policy influence of one think tank in relation to others because they are all different, work on different aspects of a policy, influence in different ways and there are many other players involved. Nelson Poslky: there is no sense in looking for direct influence of think tanks’ activities because one can only ask these questions when one ignores the complexity of political processes. Some causal relationships may only be found in a few cases, but systematic explanations of this sort remain an illusion.

5. more Weidenbaum (2009) Andrew Rich: dollar for dollar, think tanks attract much more attention than any other organisation. John Hamre, of the Center for Strategic and International Studies: evaluating the influence of think tanks on a particular policy would be like determining who is the father of Anna Nicole Smith’s baby. As soon as the change happens every think tank who had something to say about it is quick to claim it for itself. David Frum, ofthe American Enterprise Institute: since one cannot measure influence, [no method] would do it. 5

6. even more Abelson (2006) Abelson: by focusing on the influence of think tanks on policy change one would overlook a great deal of other types of influence that is the source of think tanks’ value. Leslie Gelb, form the New York Times: influence is highly episodic, arbitrary and therefore difficult to predict. Abelson and Ricci: as the policy community becomes more open and complex it also becomes more difficult to say anything meaningful about the contribution of think tanks to it. 6

7. and more Puryear (1994) The most interesting and valuable contributions of think tanks [in Chile] may not be intellectual but psychological: the hundreds of events and seminars that Chilean think tanks organised throughout the 1980s helped to restore the mutual trust and understanding missing from Chilean politics and that had led to the rupture of democratic order. 7

8. And: Hoppe (2010) The search for ‘impact’ is driven by three assumptions: Unidirectional transfer of advice; Policymakers and experts performing different and exclusive tasks; All possible impact is desirable per se 8

9. In practice, however: Robust research into uptake and impact will be so costly and time consuming, that nobody would be able to afford it. Lessons drawn will be marginal and irrelevant for other situations. Indicators focusing on that can be measured would only lead to conformance and perverse conduct. 9

10. Why monitor and evaluate? 10

11. Why should you do M&E? To learn To manage better To get more funds To keep funders/clients happy (“what is the guarantee that doing this actually helps us?”)

12. Researchers vs research Do we want: 1)Our research/IE to be more influential? or 2) Policy to be based on evidence?

13. Visibility or substance? VisibilitySubstance Short term ‘relevant’ researchLong term research Focus on solutions for ‘agreed’ problems Engage with the definition of the problem Media exposureLobby, network, horse trading Briefing papers, Opinion piecesEstimates, costed proposals, policy options Website, Blogs, Facebook, etc.Academic publications, long reports Online communities with millions of hits Communities with the right people Delegations at high level global conferences Private meetings at Party conferences and private meetings while planning for the high level conferences Event focused influenceProblem focused influence Global Go-To-SurveyProspect magazine Think Tank of the year

14. What are we evaluating? –and some complications 14

15. 1.Strategy and M&E & Learning 1.Strategy and direction –are you doing the right things? 2.Management –are you doing what you planned to do? 3.Outputs – are outputs up to scratch? 4.Uptake – are people aware of your work? 5.Outcomes and impacts –are you having any impact?

16. How? Hovland (2009): 1.Strategy and direction: Logframes; Social Network Analysis; Impact Pathways; Modular Matrices 2.Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry 3.Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews 4.Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys 5.Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies 16

17. M&E of policy influence – what’s the problem? POLICY Influencing Activities Impact Evaluation

18. M&E of policy influence – what’s the problem? POLICY Influencing Activities And who does the influencing? The researchers Campaigners/third parties? Policymakers themselves?

19. Simplifying -What’s the solution? POLICY Influencing Activities Pathway of change

20. But, in reality POLICY Influencing Activities Pathway of change

21. And this is based on actual theory

22. And what are we aiming for? Changes in individuals, the process, or the policy environment Changes in discourse, attitude, process, content, behaviour Legitimation, monitoring or auditing… 22

23. Losing control Inputs ActivitiesOutputs Other Actors Project Team Outcome s Impact Outcomes Impact Outcomes Impact Policy changes

24. So we need to look for proxys “Government runs in part, on the basis of memos. If a SD or DD official, or an analyst at the CIEA or the NSC, has your study in front of him and open at the time he is writing his own memo to the secretary or the director or perhaps the president himself –if, in short, he is using your ideas and analysis at the time he writes his own memo- then you have influence. If your study is not open in front of him, or worse, you do not even know who the responsible official is, you do not have influence. It is as simple as that.” Howard Wiarda quoted by Abelson D. E., 2006 24

25. Types of influencing

26. What to measure and how 3 types of influencing approaches (there are more): Evidence and advice Public debate/education and advocacy Lobbying approaches

27. Methods and tools

28. Evidence and advice How?What to measureTools Consultancy Research and analysis, ‘good practice’ Evidence-based argument Providing advisory support Developing and piloting new policy approaches Outputs Evaluating research reports, policy briefs and websites Uptake and use Logs; new areas for citation analysis; user surveys Influence RAPID outcome assessment; Episode studies; Most Significant Change

29. Public debate/education and advocacy How?What to measureTools Public communications and campaigns University teaching ‘Public education’ Debate Messaging Advocacy Target audience attitudes, behaviour, etc Surveys, focus groups, direct responses, career paths Media attention Media tracking logs, media assessment Media framing and influence Framing analysis; coverage

30. Lobbying approaches How?What to measureTools Face-to-face meetings and discussions Relationships and trust Direct incentives and diplomacy Actors; relationships; policy process and institutions Recording meetings; tracking people; interviewing key informants; probing influence

31. Recommendations Develop a Theory of Change based on sound theory Integrate PME for policy influence Use a mix of tools to gather data Ask: What does it mean?

32. A suggestion

33. An example (based on ODI’s structure) 33

34. Develop your own

35. Exercise Describe the change you would like to see Describe the role you will play in it -your contribution to this change How will you know that you have fulfilled this role to the best of your abilities? How will you know whether you are doing the right thing? How will you incorporate lessons (that you and others have learned) into your planning and implementation?

36. Additional resources www.odi.org.uk/rapid www.ebpdn.org www.outcomemapping.ca www.onthinktanks.org