2. Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008

3. Agenda Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

4. The history of M&E in the international development sector is in four distinct phases 1960s to 1979: Early developments 1979-1984: Rapidly growing interest 1984 to 1988: M&E matures 1988 to the present: the crossroads

5. Wealth of tools, techniques and approaches are now available Logical framework analysis Results-based management Needs Assessments, Impact Assessments Ex ante and ex post assessments ZOPP GANTT Social Network Analysis Appreciative Enquiry Most Significant Change Outcome Mapping Many many more!!

6. For many organisations, evaluations are at the centre of a vicious circle... Increased competition Increased pressure to show results and impact Lack of professional norms and standards Poor learning and accountability Growing need for high profile fundraising and advocacy work...causing problems for M&E

7. Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes

8. Reflection, learning and analysis are threatened by existing agency cultures and processes Org. Learning Existing culture & process Evaluation Accountability

9. Evaluations, like other similar initiatives, often sit on top of existing organisational silos, inefficiencies and power imbalances, rather than resolving them Silos Evaluation KM

10. Agencies plough the same evaluation “field”, but stick to their own furrows

11. Understanding of the effectiveness and use of evaluations is weak at best… Where’s the data??!

12. Evaluations tend to be based on wish lists, not strategies, and therefore are often overloaded

13. Leadership and political buy-in to evaluation is rare and unreliable, with two common reactions

14. ...all of which means that (1) evaluation efforts resemble this iceberg... What is planned What actually happens

15. And (2) evaluators spend most of their time feeling like this... working against this...

16. Agenda Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

17. “Exploring the science of complexity” Primary aim was to explore the potential value of complexity science for those who work on change and reform initiatives within the aid sector Drew on scientific and experimental literature – physiology, physics, mathematics, public sector reform, sociology, economics, organisational theory, plus case studies, reports and evaluations from the aid sector Reviewed over 250 articles, books, reports and evaluations 10 peer reviewers, including 5 directors of leading research institutes Published February 2008 Available to download from www.odi.org.ukwww.odi.org.uk

18. A man was walking home one dark and foggy night. As he groped his way through the murk he nearly tripped over someone crawling around by a lamp post. “What are you doing?” asked the traveler. “I’m looking for my keys” replied the other. “Are you sure you lost them here?” asked the traveler. “I’m not sure at all,” came the reply, “but if I haven’t lost them near this lamp I don’t stand a chance of finding them.” A (well-known) story…

19. A closer inspection of the light under the lamp revealed… INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

20. Logical frameworks / results chains articulate a clear world view and theory of change: “The light under the lamp” The machine metaphor - universe as clockwork The future is knowable given enough data points Phenomena can be reduced to simple cause & effect relationships Dissecting discrete parts would reveal how the whole system worked; science was the search for the search for the basic building blocks The role of scientists, technologists & leaders was to predict and control - increasing levels of control (over nature, over people, over things) would improve processes, organisations, quality of life, entire human societies

21. Key Assumptions Assumptions about systems  Ordered  Reductionist - parts would reveal the whole Assumptions about how systems change  Linear relationships  influence as direct result of force from one object to another - p redictable  Simple cause & effect Assumptions about human actions  Rational choice  Behavior specified from top down  Certainty and “knowability”

22. ...Reality of aid is a little different...

23. But a new light is being turned on (slowly, unevenly, using a dimmer switch)…

24. Complexity science is a science of understanding change A loosely bound collection of ideas, principles and influences from a number of other bodies of knowledge, including  chaos theory  fractal geometry  cybernetics  complex adaptive systems  postmodernism  systems thinking Discovery of similar patterns, processes and relationships in a wide variety of phenomena  related to the nature and dynamics of change

25. From microscopic chemical reactions…

26. ...to the evolution of galaxies...

27. Complexity scientists use a range of ideas and concepts (familiar, challenging and baffling) to make distinctions between simple, complicated and complex systems and phenomena

28. Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Complicated Complex Simple The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipe notes the quantity and nature of “parts” needed Recipes produce standard products Certainty of same results every time

29. The claims of complexity scientists The complexity of real world systems is (usually) not recognised or acknowledged by scientists and policy makers Or, that if it is not acknowledged, they don’t deal with them Or, that if they do deal with them, they don’t do so effectively Or, that if they do deal with them effectively, it’s because they used an specific approach / framework...that is also available to you, dear client, at a reasonable daily rate plus a per diem [JOKE]

30. There have been diverse efforts to apply ideas to social, economic and political analysis and practice Arthur, Ormerod - Economics Stacey, Snowden - Organisations Jervis, Urry, Cutler - Intl relations De Mancha - History Gilchrist - Community development Education policy - Sanders and McCabe Health policy - Zimmerman Government reform - Chapman Strategic thinking – Saunders Evaluation - Williams

31. Complexity and systems approaches have already proved useful in re- thinking aid and development issues Uphoff, 1990s Chambers, 1997 Sellamna, 1999 IDRC, Outcome Mapping, 2001 Warner, 2001 Rihani, 2002 Lansing and Miller, 2003 Inclusive Aid, 2004 ECDPM, 2004-06 Eyben, 2006 Guijt, various Davies, Network Analysis, various

32. ` 10 key concepts and implications 8 Adaptive Agents 10 Co-Evolution 4 Non-Linearity 3 Emergence from Simple Rules 6 Phase space and attractors 5 Sensitivity to initial conditions 1 Interconnected and interdependent elements and dimensions 2 Feedback processes 7 Strange attractors and the edge of chaos 9 Self organisation Features of systems Dynamics of change Behaviours and relationships

33. Complex systems Collection of parts, which collectively have a range of dimensions Parts share an physical or symbolic environment / space Action by any part can affect the whole  E.g. individuals, families, communities, cities, markets, societies, populations, economies, nations, planets

34. Complex systems are interconnected and interdependent to different degrees Interconnectedness may occur between any elements, dimensions, systems and environments This interconnectedness leads to interdependence between the elements and the dimensions of a system, and gives rise to complex behaviour Complex systems can be tightly or loosely coupled, internally and with their environment, giving rise to different kinds of complex behaviours  Tightly coupled: Global FOREX markets  Loosely coupled: US University system, global construction industry Systems can be understood via mapping techniques, followed by analysis to understand the dynamics and interactions of the system

35. Foot and mouth disease: an example of failure caused by focusing on one part of the system and ignoring the links between sub-systems (biology, geography, economics) Economic rationalisation of abattoirs and EU subsidies increased the interconnectedness of herds to a critical point. Changes to foot and mouth reporting rules delayed the isolation of infectious animal The relationship between these actions and the epidemiology of F&M was not appreciated in advance where it mattered because the livestock industry was not viewed as a interconnected, interdependent system

36. Complexity also means that systems need to be understood at different scales Atom Molecule Cell Tissue Organs Organisms Communities

37. E.g. evaluating the effectiveness of child health programmes A.N. NGO Other NGOs Private Sector Community and Family Church Civil Society Developing Country Govmts Local partners

38. Example: evaluating resource flows in the humanitarian system

39. Issues of interconnectedness, interdependence and scale usually do not become apparent until a crisis... Foot and mouth disease, UK  Economics, cattle management, disease September 11 th  Globalisation and terrorism Climate change  Western consumerism and Southern disasters Credit crunch  US mortgage market mis-selling and the world economy Food prices  Biofuels and food consumption Vulnerability to natural disasters  Sichuan earthquakes and dams Human trafficking  Desire, economics and rights abuses

40. ...indicating that we have biases in the way we view the world... Three different kinds of problems have been identified, along with some common biases in dealing with them They are:  “Messes”  “Problems”  “Puzzles”

41. “Messes” are issues that do not have a well defined form or structure. NB not a value statement, but a description There is often not a clear understanding of the problem faced Messes often involve economic, technological, ethical and political issues  It has been suggested that all of the really important issues in the world start out as messes. For example, how was rising HIV/AIDS incidence in Brazil dealt with in the 1990s?  concerned money, technology, ethics, social relations, politics, gender relations, poverty  all of these dimensions of the problem had to be dealt with simultaneously, and as a whole

42. Many of the major problems we face are “messes”! Credit crunch  US mortgage market mis-selling and the world economy Food prices  Biofuels and Vulnerability to natural disasters  Sichuan earthquakes and dams Climate change  Western consumerism and Southern disasters September 11 th  Globalisation and terrorism Human trafficking  Sexual preferences and human rights abuses Arms trade  Economics and war Etc, etc, etc

43. “Problems” are issues that have a known or knowable form or structure The key dimensions and variables of a problem are known and the interaction of dimensions may also be understood, even if only partially. With problems, there is no single clear cut way of doing things  there are many alternative solutions, depending on the constraints faced  Expertise matters For example, dealing with the sewage system in a particular city may rely on amount of money available, technology, political stance of leaders, climatic conditions, urban development, the road system, and so on

44. Puzzles are well defined and well structured known problems with a specific “best” solution Solutions can be worked out and improved  Solutions are replicable - “best practices” are possible

45. Policy and traditional science is biased towards puzzle-solving Real-world, complex, messy nature of systems is frequently not recognised Simple puzzle-based solutions are applied to complex messes  E.g. Global War on Terror has been applied as the single best solution to the mess of terrorism

46. “Some of the greatest mistakes have been made when dealing with a mess, by not seeing its dimensions in their entirety, carving off a part, and dealing with this part as if it were a problem, and then solving it as if it were a puzzle, all the while ignoring the linkages and connections to other dimensions of the mess”

47. Or to put it another way: dividing a cow in half does not give you two smaller cows

48. Implications: analyse and deal with the reality of the system Multidimensionality, interdependence and interconnectedness of poverty and humanitarian crises (and responses to them) should be recognised by those designing, managing and evaluating aid interventions Analysis may need to be in line with historical research - not ‘did x cause y?’ but ‘what happened and why?’, building narratives about events and processes. The task of selection and synthesis of data becomes as important as analysis Different perspectives on what the system is need to be taken into account, especially when these perspectives differ as to the nature, interconnectedness and scale of the system Messes, problems and puzzles need to be identified and dealt with using appropriate approaches

49. Interconnectedness and interdependence gives rise to a range of phenomena and behaviours

50. ` To find out more, read the paper! 8 Adaptive Agents 10 Co-Evolution 4 Non-Linearity 3 Emergence from Simple Rules 6 Phase space and attractors 5 Sensitivity to initial conditions 1 Interconnected and interdependent elements and dimensions 2 Feedback processes 7 Strange attractors and the edge of chaos 9 Self organisation Features of systems Dynamics of change Behaviour of agents

51. Agenda Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary

52. Evaluation of Complex Systems is NOT new Educational systems, social initiatives and government interventions are complex social systems where effective evaluation is seen as a key process in measuring success But there is an increasing recognition that for evaluation of complex social systems to be more effective, evaluations may need to take into account the theoretical understanding of complex systems

53. Implications for evaluation are at four levels 1. Implications for evaluation methodologies and approaches 2. Implications for the focus of evaluations 3. Implications for the purpose and scope of evaluations 4. Implications for evaluations as a complex system in their own right

54. Implications for evaluation methodologies and approaches

55. Linear models dominate... If you deliver the product and / or service to the extent Intended, then your participants will benefit in certain ways If these benefits to participants are achieved, then certain changes in communities, organisations and systems might be expected to occur If you accomplish your planned activities, then hopefully you will deliver the amount of the product and / or service that you intended Certain resources are needed to operate your program If you have access to them, then you can use them to achieve your planned activities Outcomes / Purpose Impacts / Goal OutputsInputsActivities MONITORING EVALUATION

56. Many evaluation “results chains” visualize change as linear, based on multiple cause-effect logic models X Y Linear, Predictable Focused on the end result The program (X) gets the credit! Inputs Activities Outputs Outcomes Impact

57. When applied to development, “the results chain” can lead to Faulty thinking Misguided data collection Misleading reporting of results Gives a false sense of reality to senior managers and donors who are far from where change is taking place

58. Complexity science sees change as… Interconnected (multiple actors and factors) Non-linear (unexpected results occur) Incremental, cumulative, with tipping points Beyond the control of the project / programme Two-way (program also changes) Continuous (not limited to the life of the project)

59. Therefore Develop new theories of change, adapt existing theories of change to challenge assumptions of linearity Design evaluations to allow for interdependencies and interconnections in the system the program is trying to influence, and capture the resulting dynamics

60. Implications for evaluation focus

61. Focus of “traditional” evaluations Formal project / programme / organisation Environment is outside the organisation  evolves separately until programme is implemented Level of analysis is single or at most a few, relatively independent levels

62. Implications of complexity Features of systems: evaluate from perspective of multiple, nested levels of interconnected systems, study feedback between the organisation and its environment, look for emergent rather than planned change Dynamics and nature of change: Look for non-linearity, anticipate surprises and unexpected outcomes, analyse the system dynamics over time and frame the “space for possible change”, look for changes in conditions that facilitate systemic change, and how well matched the programme is to the wider system People, motivations and relationships: Study patterns of incentives and interactions among agents, study quality of relationships, study individuals and informal / shadow coalitions, vs. formal organisation, study co-evolution of organisation and environment

63. Implications for evaluation purpose and scope

64. M&E is seen as standing in contrast to creative dynamism of field work

65. Purpose and scope of traditional evaluations vs complexity-oriented evaluations TraditionalComplexity-oriented Measure success against predetermined goals Develop new measures and monitoring mechanisms as goals emerge & evolve Render definitive judgments of success or failure Provide feedback, generate learning, support direction or affirm changes in direction Aim to produce generalisable findings across time & space Aim to produce context-specific understandings that inform ongoing innovation Creates fear of failureSupports hunger for learning

66. Implications for evaluation as a complex system

67. Key Assumptions Assumptions about systems  Ordered  Reductionist - parts would reveal the whole Assumptions about how systems change  Linear relationships  influence as direct result of force from one object to another - p redictable  Simple cause & effect Assumptions about human actions  Rational choice  Behavior specified from top down  Certainty and “knowability”

68. Goals Results Activities Monitoring Evaluation Evaluations traditionally seen as a rational, technical, information-generating process...Reality is a little different...

69. Evaluation systems are themselves complex systems, with many interconnected parts and dimensions; no two evaluations are the same Focus of evaluation Policy / Guidelines Scope Project vs Policy Demand, goals Timing, quantity Preparation, TOR Management and team selection Methods and tools Engagement with stakeholders Dissemination of findings and utilisation Costs Quality maintenance mechanisms

70. “...To be effective an evaluation program must match the dynamics of the system to which it is applied....” Eoyang and Berkas (1998)

71. Implications of evaluations as a complex system TraditionalComplexity-oriented Position the evaluator outside to assure independence and objectivity Position evaluation as an internal, team function integrated into action and ongoing interpretive processes Accountability to control and locate blame for failures Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond strategically Accountability focused on and directed to external authorities and funders Accountability centred on fundamental values and commitments Evaluator controls the evaluation and determines the design based on the evaluator’s perspective about what is important Evaluator collaborates in the change effort to design a process that matches philosophically and organisationally

72. All systems can be placed on a spectrum between extremes of ordered and chaotic E.g. solids and gases  In solids, atoms are locked into place  In gases they tumble over one another at random In between the two extremes, at a phase transition, a phenomenon called the ‘edge of chaos’ occurs  This phenomenon describes systems behaviours where the evolution of the system never quite locks into place and never quite dissolve into turbulence either. In human organisations, the simplest example is of a system that is neither too centrally controlled (order) nor too bottom-up and therefore disorganised (chaos)

73. Physiology on the Edge of Chaos Healthy Heart – on the edge of chaos Severe Congestive Heart Failure – orderly Cardiac Arrhythmia, Atrial Fibrillation - chaotic

74. Dynamic adaptability is the key to system health Changelessness is a sign of death, transformation a sign of life. - Commentary on the I Ching

75. Evaluation at the edge of chaos?

76. Agenda Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Final points

77. Complexity science gives additional weight to calls for re-thinking The wider contexts of aid work The nature of the processes involved in aid work The dynamics of change involved in aid work The real influence of aid work The role of partner organisations and beneficiaries in aid work The tools and techniques for planning, monitoring, learning and evaluating aid work

78. Given the resistance to change in the power dynamics of aid, real world applications may continue to be “innovative”, “under the radar” and outside the mainstream of aid policy and practice...

79. There are a number of common criticisms of complexity... Theoretical: adds nothing new  E.g. the ideas of complexity science have all been identified elsewhere  But complexity brings them together Practical: doesn’t specify what should be done  E.g makes no specific recommendations as to how best to act in complex systems  See next slide Supports managerial “snake-oil” / “complexologists” / re-warmed ideas  E.g. is abused and misused  What isn’t? Political: emergence and self organisation support neo-liberal stances  Just because self-organisation happens, doesn’t mean it is good Credit crisis Rwandan genocide  Complexity is more centre-ground, for example, “edge of chaos” systems are seen as most robust and resilient because are the optimally combination of control and flexibility

80. ...and different perspectives on the value of complexity science Deep paradigmatic insights  Champions Interesting parallels and useful approaches, but not the only way to see things  Pragmatists Meaningless coincidences  Critics

81. Source of weakness is also the source of strength These ideas are not about “what you dos”, but about “how you dos”  Not “solutions for problems”, but “approaches to problems”  Tools for furthering understanding, for opening up new ways of seeing and thinking They point to the personal, professional, institutional, political mindsets, attitudes and conditions which need to be in place to work effectively in and with complex systems

82. Complexity concepts can support the intuition and navigation of practitioners INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

83. Four suggestions Develop collective intellectual openness to ask a new, potentially valuable, but challenging set of questions of our mission and their work Develop collective intellectual and methodological restraint to accept the limitations of a new and potentially valuable set of ideas  not misuse or abuse or let them become part of the ever-swinging pendulum of aid approaches Need to be humble and honest about the scope of what can be achieved through ‘outsider’ interventions, about the kinds of mistakes that are so often made, and about the reasons why such mistakes are repeated Need to develop the individual, institutional and political courage to face up to the implications of complexity

84. ...We can't solve problems by using the same kind of thinking we used when we created them.. Final points (1)

85. Everybody thinks to change the world; nobody thinks to change themselves Final points (2)

86. Thank you! Get in touch  b.ramalingam@alnap.org