v MEASURING IMPACT Kristy Muir Stephen Bennett ENACTUS November 2013

WHY MEASURE? Does your or other social enterprises really make a difference? How do you know? How can you prove it? Why bother? Purpose of social enterprise: balancing commercial strategy with social environmental or other public benefits Other benefits from measuring impact To understand your impact: determine the ‘merit, worth or significance’ of something (Scriven 1967) To determine the difference between model/enterprise/idea and implementation failure/success To inform and improve services/ideas/enterprises Measuring Impact 2

There are a range of ways to measure programs/policies/interventions and there are hierarchies of quality / 3 MEASUREMENT THEORIES Measuring Impact Definition "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of... Programs” (Rossi and Freeman, 1993) The systematic collection of information about the activities, characteristics, and outcomes of programs that specific people use to reduce uncertainties, improve effectiveness, and make decisions regarding what those programs are doing and affecting’ (York 2000) [our emphasis] Mullen et al. (2005)

4 MEASUREMENT THEORIES Measuring Impact Mullen et al. (2005) Outcome evaluations study the effects of the program on participants Process evaluations assess how the program works: establishment and implementation; facilitators and barriers to effective ways of working Formative and summative evaluation Formative: help businesses, policy makers and practitioners refine and develop the process through which the program is implemented Summative: measure the impact of the program Economic evaluation examples Social Return on investment Cost-benefit analysis Cost-effective analysis

Is your approach to measuring impact ethical? How have you incorporated research ethics – consent and data collection Impartial evaluator Operational challenges Data availability Attribution 5 ETHICS AND MORALS OF EVALUATION Measuring Impact

Social enterprise models Geographic location Diverse client demographics What data is available? Policy/legislative framework Social, cultural, political, religious Comparative programs Existing evidence of intervention 6 CONTEXT Measuring Impact What is best approach for collecting evidence based on the contextual factors for your organisation? What contexts do you need to consider in measuring your impact? (Coatsworth, 2002, reproduced in Lippman 2004)

Evaluators of complex social programs should: Develop a theory o E.g. marshmallow test & impulsive behaviour / impulse control Develop a hypothesis based on that theory (what might work for whom under what circumstances) o If we intervene early to teach children impulse control we can decrease the likelihood they will end up in gaol. Intervention will involve… Conduct research to test the hypothesis Make conclusions based on the hypothesis and the findings Feed back into the development of theory 7 THEORY TO PRACTICE Measuring Impact

What? A ‘program logic’ is a systematic, visual representation of the underlying assumptions of a planned program. A results logic illustrates why and how a program is presumed to work. These models present a sequential, interactive account of how inputs will lead to outputs, which will result in the desired outcomes. Designing a log frame/theory/hypothesis 1.What is the major outcome anticipated? Describe the problem your social enterprise is attempting to solve. What is your social mission? 2.What other goals or sub-outcomes anticipated or desired (which will help work towards the major outcome)? What are the near and long term desired results? 3.What’s the theory behind why the program will work? What broad systems and structures [or programs] are in place to achieve these outcomes? What are the successful strategies that have helped other organisations achieve the outcomes you desire? What other evidence is available? 4.What outputs from these systems and structures might contribute to the outcomes? 8 LOGIC FRAME APPROACH Measuring Impact

9 A LOGIC MODEL Measuring Impact Inputs/ Resources ActivitiesOutputsOutcomesImpact What goes inWhat happensImmediate resultsShort and long term results Effects on root causes – sustained change Resources Equipment Knowledge/ expertise What does your organisation/ enterprise do? How much activity occurs? For each activity what are the results? For example: Improved living conditions Increased income For example: Change in poverty Change in social norm & attitudes W.K. Kellog Foundation (2004), Ebrahim & Rangan (2010) Logic model evaluation method Map appropriate methods against the program logic What information could or needs to be collected Client segmentation What population level information is available

Lead to [MAJOR OUTCOME] Lead to [OUTCOMES] Which in turn will result in [OUTPUTS] 123 And through... [WHAT OTHER ASSUMPTIONS ARE IMPORTANT RE PROGRAM STRUCTURE] By providing... [WHAT AND HOW] Supports... [WHO is being supported] Model [WHAT]

For example…

What is the business model for your social enterprise? - Purpose - Objectives and strategy & goals - Theory/ research - Finance model How will you measure the social impact of your social enterprise? 12 WHAT DOES THIS MEAN FOR ME? Measuring Impact

13 Q&A Measuring Impact