1. Theories, impacts, & evaluation Simon Pringle, SQW Ltd

3. Objectives for this Session 4+1! 1.Deepening our general understanding of the role of ‘Theory’ in evaluation 2.What Theory Based Impact Evaluation involves > When & how to do it, generally > Specific approaches to TBIE > Practical fieldwork methods 3.Where, & when, Counterfactual Impact Evaluation makes sense > The ‘necessary’ conditions to undertake CIE > Pros & cons vs TBIE approaches 2

4. Objectives for this Session 4.Suggestions for evaluating the ‘added value’ of cooperation – Strategic Added Value (SAV) > Key aspects of SAV > Challenges in characterising Mix of chalk/talk, exercises, & discussion Regular Q&A For you to have fun & enjoyment in to the process My promise – KISS!... & copies of slides to follow 3

5. About SQW 33 years old ~40 staff, €4.2m T/O, 3 offices (cf ~90, €12m T/O, 6 offices in 2011) Working primarily for Public Sector > EC, ADB, World Bank, nation states > UK Government - BIS, DECC, DEFRA, HMT - & the Devolved Administrations > City States, wider place & sector partnerships, etc.. > Innovation, Place, Children & Young People 4

6. About SQW Recent assignments > WEFO - evaluation of ERDF Business Support in Wales (CIE) > DETI - evaluation of the Regional Selective Assistance Programme in Northern Ireland (CIE) > Interreg 4B - review of management structures & programme procedures for NWE INTERREG IVB > Interreg 4B - evaluation of results & achievements of the 2007/14 InterReg IVB Programme ('Capitalisation' Study, TBIE) > DG Res & Inno - Supply & Demand Side Policies for Innovation 5

7. 6

8. Let’s start at the very beginning...

9. Talking Terms Activities > things that are done [within programme/project's timeframe] Outputs > direct measures of an Activity that can be counted [within programme/project's timeframe] – e.g. no attendees at workshop, no businesses advised Results > subsequent effects caused by the Outputs, which can again be measured, ideally) [within/after programme/project] –↑ understanding of new policy, ↑ knowledge/confidence among SMEs Impacts (aka ‘long term effects’) > wider economic, social or other effects that can be credibly attributed to an intervention [generally after programme/ project] 8

10. Talking Terms Monitoring... > Is the systematic, regular collection & occasional analysis of information to identify (& possibly measure) change over time Evaluation... > Is the analysis of the effectiveness & direction of an activity, & involves making judgments about progress & impact. Main differences between monitoring & evaluation > Timing & frequency of observations > Types of questions asked >... when monitoring & evaluation are integrated as project management tools, the line between the two becomes can be blurred (After Vernooy et al 2003) 9

11. So, why bother with evaluation? Because of... Compliance  Emphasis in new 2014 -2020 period on ‘results orientation’ > Attempt to shift Programmes from being ‘busy’ to being ‘productive’  Explicit requirement in the Regulations > Common Provision Regulation (CPR), ETC Regulation etc 10

12. So, why bother with evaluation? Because of... Conscience  Are programmes/project's doing the right things & doing these things right  Identifying learning to enable real-time change & improvement  Establishing evaluation (& underpinning monitoring) as a core behaviour & part of our legacy 11

13. Stakeholders Typically, 3 groups 1.Those being evaluated – Justification/Legitimacy – Learning at operational level 2.Sponsor(s) & audience (s) for the evaluation – Accountability – Resource allocation – Learning at policy level 3.Those performing the evaluation – Professional & academic interest/advancement Forget about stakeholders, their interactions & their various interests at your peril! 12

14. Differing levels & scopes System(s) &/or markets Policy(s) Programme portfolio Programme, ranging from > Multi-measure, multi-partner > Single-measure, single-partner Individual Project > As part of programme implementation (ex- ante, selection of funded projects) > As part of Programme evaluation to understand effects, E 3 etc.. Same core thinking applies... but approach approaches/methods vary 13

15. Evaluation is not new Formally recognised since the 1920s Jacob Adler, Manchester Institute of Innovation Research 1.“Part of the general practice of science > Career progression, editorial judgment, award of financial grants > Allocation of funds beyond projects (basic funding), attractiveness of institutions 2.Broader requirement for evaluation of publicly- funded activities at various levels > Legal requirements: monitor, audit > Management: Economy, efficiency & effectiveness > Policy: the intended effects, cost-benefit, Value for Money > Society: the overall benefits, along many dimensions” 14

16. The Pantheon of Thinkers 15 After Marvin Alkin in ‘Evaluation Roots’

17. The centrality of ‘theory’ in evaluation A theory is a collection of assumptions & hypotheses about how a programme/project's /intervention will work > Design, appraisal, implementation, monitoring &/or evaluation Often depicted in the form of a logic model (a.k.a. intervention logic, logic chain, log frame, causal chain) > Specifies inputs & activities, & results expected > May also make explicit key contextual factors, assumptions, mechanisms, & risks Matters because theory provides the ‘constant’ against which: > Appraisal, delivery & evaluation are undertaken > Partners & delivers can act, proceed, & be held accountable 16

18. Logic model - the ‘Building Blocks’ 17 What is broken? Is there a Rationale? Objectives (1 0 s & SOs) Inputs Outputs Results Impacts Conditions indicators Activities

19. Example: a simple logic model 18 E.g. start-up support, linked to other services; engagement of businesses; development of mentoring scheme Activities E.g. public & private sector expenditure; time inputs of volunteer business mentors Inputs E.g. numbers of individuals starting a business; numbers of businesses assisted Outputs E.g. Effects on individual & business behaviour; effects on business performance; effects on entrepreneurial culture (Outcomes link to Objectives/ Rationale) Results (short & long-term) E.g. increase start-up & survival rates by x; increase performance of businesses with growth potential by x Objectives E.g. low start-up rates; constrained growth of businesses E.g. due to area- based factors around culture; lack of appreciation of benefits of support Conditions & rationale Feedback to Conditions A business support programme

20. Example: a more complicated logic model 19

21. Different formats of logic model 20 D Impacts Changes in the contextual conditions that gave rise to the policy intervention C Results Effects on the behaviour, capacity & performance of the people, communities, businesses & organisations B Targets & key performance measures Activities, outputs & outcomes Theory of change Why & how will the interventions tackle the problems? Assumptions What factors must exist for success Inputs Resources – people, time, materials, funds – dedicated to the design & delivery of interventions Activities & processes The services provided & mechanisms supporting Gross outputs Direct effects Net outputs Adjusted through additionality A Contextual conditions & problems in the relevant policy domain/spatial area Policy context – aims & objectives of the interventions 1. Analysis of contextual conditions 2. Appraisal of strategic priorities & options 3. Targeting & monitoring 4. Process evaluation 5. Impact evaluation, synthesis & learning

22. Different formats of logic model 21 Contextual conditions & rationale Aims & objectivesInputsActivitiesOutputsResults Theory of change - StrategyTheory of change - DeliveryTheory of change - Benefits

23. To summarise Evaluation  Matters!  Is not new... & continues to develop as a science/art  ‘Theory’ thinking is increasingly centre-stage ... And very, very satisfying to undertake! 22

24. Q&A 1

25. 2. The importance of ‘n’ in determining evaluation methods 24

26. Large & small ‘n’ considerations in evaluation In the ideal world of evaluation...  All intervention populations (i.e. ‘n’) are large – QED, statistical tests of significance hold  The beneficiary (or ‘treatment’) group, the intervention (or treatment), & the wider context are highly homogeneous  Budgets, & political/other constraints are not an issue for sample sizes, and/or the use of comparison groups  In these circumstances, CIE is the way to go... 25

27. Large & small ‘n’ considerations in evaluation But the world is very rarely ideal – ‘n’ is often small - & CIE very frequently unfeasible Hence, role of Theory-Based Impact Evaluation (TBIE) approaches in evaluation  Yes, TBIE does not provide a quantifiable counterfactual  Yes, TBIE does not provide statistically hard & fast numbers  Yes, TBIE has limitations in deriving robust cost/benefit numbers ... But done well, can make very meaningful causal interferences to evaluate impact 26

28. Large & small ‘n’ considerations in evaluation 27 TYPICAL APPROACH & PURPOSE Intensive analysis of two or more contrasting cases to understand key factors that influence results Intensive analysis of a single case (community, organisation, event etc..) to understand how & why results are generated Collection of data on a large number of cases at a single point in time to detect patterns of association Same as cross- sectional, but data collected on at least two occasions to allow insights into the time order of variables Develop the counterfactua l ex post facto by taking advantage of ‘natural’ experiments, cut-off points or statistical techniques Design an intervention with otherwise identical treatment & non- treatment groups to isolate its effects Quantitative, counterfactual Typically qualitative Quasi- experimental Cross- sectional Longitudinal Case study Comparative Experimental Typically quantitative, no counterfactual Theory-based approaches Small ‘n’ - TBIE Methods Large ‘n’ - CIE Methods

29. Large & small ‘n’ considerations in evaluation 28 Quantitative, counterfactual Typically qualitative Quasi- experimental Cross- sectional Longitudinal Case study TYPICAL APPROACH & PURPOSE Comparative Experimental Intensive analysis of two or more contrasting cases to understand key factors that influence results Intensive analysis of a single case (community, organisation, event etc..) to understand how & why results are generated Collection of data on a large number of cases at a single point in time to detect patterns of association Same as cross- sectional, but data collected on at least two occasions to allow insights into the time order of variables Develop the counterfactua l ex post facto by taking advantage of ‘natural’ experiments, cut-off points or statistical techniques Design an intervention with otherwise identical treatment & non- treatment groups to isolate its effects Typically quantitative, no counterfactual Theory-based approaches Small ‘n’ - TBIE Methods Large ‘n’ - CIE Methods

30. Small ‘n’/TBIE methods – careful what you wish for! 29 Realist Evaluation Contribution Analysis Policy Scientific Approach Strategic Assessment Approach Prospective Evaluation Synthesis Elicitation Methods General Elimination Methodology (aka Modus Operandi Method) Process Tracing Most Significant Change Success Case Method Qualitative Comparative Analysis Outcome Process Mapping Theory of Change Method for Impact Assessment of Programmes/Projects

31. Types of small ‘n’ approaches Group 1  Theory-based methods to determine causes of observed effects & how ‘additional’ observed outcomes occurred e.g. > Theory of Change > Realist Evaluation > Contribution Analysis > General Elimination Methodology (aka The Modus Operandi Method) > Process Tracing Variations on the ‘Theory’ tune 30

32. Types of small ‘n’ approaches Group 2  Factors perceived to have been important in producing change, with a strong emphasis on stakeholder views > Most Significant Change > Success Case Methods > Outcome Mapping > Method for Impact Assessment of Programmes & Projects Important to understand that Group 1 & 2 methods often intermingled in real-world evaluation studies (i.e. ‘mixed methods’) 31

33. 3 x Group 1 approaches 32

34. 1A. ‘Vanilla’ Theory of Change Takes the logic chain for the intervention...... & develops this in to a predictive & explanatory depiction of what should happen through the intervention Evaluation explores each step of the ToC to understand whether theoretically predicted changes occurred as expected, &/or as result of other external factors 33

35. So, Logic model... 34

36. ... & on to Theory of change 35 “Logic models are descriptive. Theory of change [...] models are explanatory & predictive.” Qualitative Research & Evaluation Methods, Patton, 2001

37. 1B. Realist Evaluation After Pawson & Tilley (1997) > Sceptical about (quasi) experimental approaches > “Where several evaluations of similar interventions in different contexts exist, the most usual finding is that the results vary.” Key concept: context + mechanisms = outcomes > What Context Mechanism Outcome Configurations (CMOCs) appear the most successful > By doing so, RE seeks to understand “what works, how, in which conditions & for whom” 36

38. 1B. Realist Evaluation So, in practice? > Realist Evaluation is not a method but a way of thinking, so realist design can be incorporated within almost any evaluation Typical steps: 1.Theory & hypothesis formulation Carry out research to establish the prevailing Middle Range Theory (MRT) Map out a series of conjectural mini-theories or CMOCs 2.Data collection Includes quantitative & qualitative research Aim is to refine, refute or demonstrate how CMOCs have operated in practice 3.Data analysis & conclusions How mechanisms have operated in programme contexts to generate results – which CMO configurations were substantiated, which were invalidated, & which need to be revised 37

39. 1C. Contribution Analysis Developed Mayne (1999) to address the problem of attribution: » Are observed results due to programme activities rather than other factors? » Conceived as an alternative to experimental designs, when these are not feasible CA sets out to verify the theory of change, but also takes into consideration other factors Causality/contribution is inferred ‘beyond reasonable doubt’ by assessing factors 38 1. The programme is based on a reasoned theory of change; the assumptions behind why the programme is expected to work are sound & plausible 2. The activities of the programme were implemented 3. The theory of change is verified by evidence: the chain of expected results occurred 4. Other factors were assessed & were either shown not to have made a significant contribution, or if they did, their relative contribution was recognised

40. 1C. Contribution Analysis So, in practice  Develop a theory of change > Including underlying assumptions, risks to it, & other factors that may influence outcomes  Gather the existing evidence on the theory of change > Need evidence on programme activities & results… > …but also on assumptions & other influencing factors > Methods can include surveys, case studies, process tracking, etc..  Assemble & assess the contribution story, & challenges to it > Any weaknesses point to where additional data or information would be useful.  Seek out additional evidence  Revise & strengthen the contribution story 39

41. Break for Coffee 40

42. And 3 x Group 2 approaches 41

43. 2A. Most Significant Change (MSC) After Davies & Dart (2005) Participatory process involving sequential collection of stories of significant change which have occurred as a result of intervention Linked process of sifting by stakeholders to select, discuss, & crystallise most significant changes Typically, “looking back over the last XX, what do you think the MSC in XX or YY has been” Done well, can generate useful information for the specification & subsequent assessment of a Theory of Change 42

44. 2B. Success Case Method (SCM) After Brinkerhoff (2003) Narrative technique using naturalistic enquiry & case study analysis Intended to be quick/simple Focus deliberately on very best & very worst results of intervention, & role of contextual factors in driving this “Searches out & surfaces successes, bringing them to light in persuasive & compelling stories so that they can be weighed... provided as motivating & concrete examples to others, & learned from so that we have a better understanding of why things worked & why they did not” 43

45. 2C. Qualitative Comparative Analysis Case-based method which identifies different combinations of factors that are critical to a given result, in a given context Not yet widely used in evaluation, provides an innovative way of testing programme theories of change Qualitative data/evidence on potentially relevant causal factors is turned into a quantitative score that can be compared across cases – Crisp set QCA: cases coded “0” or “1” – Multi-value QCA: allows for some intermediate values (e.g. 0.33 or 0.5) – Fuzzy set QCA: allows for coding on a continuous scale anywhere between 0 & 1 44

46. Qualitative Comparative Analysis Example of a multi-value QCA data table 45 CasesFactor AFactor BFactor COutcome Existence of space for dialogue between state & civil society Capacities of key civil society actors to engage with state Horizontal coordination between key civil society actors Stronger national & regional policy making & implementatio n Country 11011 Country 200.51 Country 30100 0 = no or weak evidence to support 0.5 = some evidence to support 1 = strong evidence to support

47. OK, Simon says... Let’s be honest > Evaluation in an EU-funding context still maturing > Many of you personally coming new to this area... >... & many of you want to ‘do’ rather, than ‘review’! > There’s a ferocious market of external ‘evaluators’ > Very significant risk of ‘running’ before ‘walking’ So, 4 guiding principles to go forward with 1.Be pragmatic – 85% of something is better than 100% of nothing... & avoid being overly academic! 2.Be developmental – Rome wasn’t build in a day, & evaluative capability needs to build & evolve 3.Become intelligent consumers – think what you are doing, or buying, evaluation-wise 4.Do commit to making time for evaluation things, so building your knowledge 46

48. Q&A 2 47

49. 3. So, to EuroGungHo 48

50. Exercise 1: logic chain & theory of change Purpose  Develop Logic Chain & Theory of Change for EuroHungHo Context  EuroHungHo – a fabricated project!  ‘Improving existing & developing new innovation support services, with a focus on the sectors of special interest to the Programme Area’  8 countries  5 sectors of special interest  Identify/developing R&D projects, pilots/prototypes, demonstrators You’ve already seen the script... 49

51. Exercise 1: logic chain & theory of change Task  Using template, develop 1.Logic chain (descriptive) 2.Theory of Change (explanatory & predictive) Logistics  Work as tables  Self-selected: > Chair > Rapporteur for feedback  I will float around...  You have until 1205 ... Plenary to 12.30 Off you go! 50

52. Exercise 1: logic chain & theory of change Plenary Feedback  Our good friend EuroGungHo  2 tasks > Logic Chain > Theory of Change 51

53. Break for lunch – back at 1320 prompt 52

54. Theories, impacts, & evaluation Simon Pringle, SQW Ltd

55. Reprising this morning Why evaluation matters The development of ‘Theory’ thinking The ‘n’ thing 2 groups of theory-based approaches Putting the thinking in to practice 54

56. 4. So far so good, what about some evaluation techniques? 55

57. Technique 1: Contextual & Documentary Review ‘What did we think we were doing?’ Desk-based review of > The problem/challenge faced (context – data) > The case for intervention (rationale – arguments) > Our practical commitment (objectives, inputs) > Progress so far (activities, outputs, & processes) > Knowing what we know now: – How logically consistent is all of this? – Do we need to change track? Sources: secondary data (local, national, European) original programme documents, application forms, appraisals, approvals, monitoring data & reports etc. 56

58. Technique 2 - One-to-one Consultation [As an informed viewer] ‘What are you observing about the intervention?’ Detailed consultations with key stakeholders  Policymakers/funders  Adjacent programmes  Delivery bodies Modes: face-to-face – telecom – postal – online Useful for scoping the issues & for cross-checking messages from elsewhere in study 57

59. Technique 3 – Surveys [As someone who is impacted] ‘What has your experience of this intervention been’ 2 groups  Beneficiaries – intended or otherwise  Non-Beneficiaries – typically those who were ruled out Modes: Face-to-face – telecom – postal – online Typically, self-reported view & observation Prone to  Last event bias  Memory decay Questionnaire design & analyse-ability a key challenge 58

60. Technique 4 – One-to-Many Consultation [As informed viewers] ‘What are you observing about the intervention?’ Similar to one-to-one consultations, but multi- rather than bi-lateral Efficient to setup/deliver... but prone to  Superficiality  Herd effects  Loudest voices  & tend to be primarily qualitative in observation Often useful to calibrate headlines from 1to1 consultations & surveys 59

61. Technique 5 – Case Studies What has worked well & less well Provide deep-dives into specific aspects of the intrevnetion  Process  Impact  Learning Typically, done face-to-face – so, resource intensive Judgement required to establish rounded view Can be hard to secure consensus amongst consultees Can be difficult to synthesise findings across case study authors 60

62. Technique 6 – Learning Diaries Real-time recording of intervention experiences Avoids memory decay & last-event bias Does require discipline on part of participants to maintain diary Need recording interval that makes sense – related to speed of changes happening/progress being achieved Helpful to frame wider consultation/survey work 61

63. 5. EuroGungHo Returns... 62

64. Exercise 2 – Impact Evaluation Plan Back to our good friend EuroGungho...... we now have a ToC we understand... So, what might an Impact Evaluation Plan for EuroGungho look like? Using presented methods & techniques etc.: > ‘What, where & how’ of an outline impact evaluation plan – Which theory based approach? – What mix of techniques to progress, & sequencing? – Do as a simple block diagram – template provided > What pre-requisites > Timing of impact evaluation activity – When, & why? > Resourcing – What cost to undertake - €s, internal vs external? 63

65. Exercise 2 – Impact Evaluation Plan Logistics  Again, work as tables  Self-select > Chair > Rapporteur for feedback  I’ll be on hand...  You have until 14.30!  Plenary 14.30- 15.00 Off you go! 64

66. Exercise 2 – Impact Evaluation Plan Plenary Feedback  Our good friend EuroGungHo  Evaluation Plan 1.‘What, where & how’ of an outline evaluation plan 2.What pre-requisites 3.Timing of impact evaluation activity 4.Resourcing 65

67. 6. Where, & when, Counterfactual Impact Evaluation makes sense 66

68. Large & small ‘n’ considerations in evaluation 67 Quantitative, counterfactual Typically qualitative Quasi- experimental Cross- sectional Longitudinal Case study TYPICAL APPROACH & PURPOSE Comparative Experimental Intensive analysis of two or more contrasting cases to understand key factors that influence results Intensive analysis of a single case (community, organisation, event etc..) to understand how & why results are generated Collection of data on a large number of cases at a single point in time to detect patterns of association Same as cross- sectional, but data collected on at least two occasions to allow insights into the time order of variables Develop the counterfactua l ex post facto by taking advantage of ‘natural’ experiments, cut-off points or statistical techniques Design an intervention with otherwise identical treatment & non- treatment groups to isolate its effects Typically quantitative, no counterfactual Theory-based approaches Small ‘n’ - TBIE Methods Large ‘n’ - CIE Methods

69. Reminder: when is CIE feasible? CIE relies on statistical tests of significance between treatment & comparison groups  This requires a large “n” (sample size) CIE is a serious option where: 1.The total population is large 2.The treatment group, the treatment itself, and/or the wider context, is homogeneous 3.The intervention affects very defined segments 4.Budgetary, political or other constraints allow a sufficient sample size or use of comparison group(s) 68

70. Practicalities Theory-based thinking still holds...... but CIE activity used to deliver the statistically robust impact assessment Methods used similar to TBIE  Doc review, consultations, case studies, learning diaries etc.... But quantitative impact story delivered by formal CIE analysis CIE, & its underpinning statistics, are specialist areas  Largely delivered by academics  Not always easy to work with  Demanding to follow what is going on – ‘Black Box’ 69

71. 70

72. So, TBIE vs CIE? TBIECIE Pros Pros Cons Cons 71

73. Suitability of CIE for Interreg programmes SDP’s personal view  Limited because > Width & range of projects, plus cross-border issues, makes homogeneity hard to achieve > Sophistication of method out of step with programme evaluation capacities, which are still building > Cost to undertake effectively may be prohibitive  So, time is not yet right... but something to think about for the future, & to be alert to as you evaluate consultancy submissions 72

74. Conclusions Theory-based impact evaluation cannot rival the rigour with which well-designed counterfactual impact evaluation addresses issues of attribution However, done ‘right’, TBIE can tackle attribution & provide evidence to back up causal claims White & Phillips have identified the following “common steps for causal inference in small ‘n’ cases”: 1.Set out the attribution question(s) 2.Set out the programme’s theory of change 3.Develop an evaluation plan for data collection & analysis 4.Identify alternative causal hypotheses 5.Use evidence to verify the causal chain 73

75. Further reading European Commission: Evalsed Sourcebook – Methods & Techniques Rogers, P (2008): Using Programme Theory to Evaluate Complicated & Complex Aspects of Interventions. Evaluation, Vol 14(1): 29 – 48 White, H & Phillips, D (2012): Addressing attribution of cause & effect in small n impact evaluations: towards an integrated framework. 3ie Working Paper 15 (You already have!) Westhorp, G (2014): Realist Impact Evaluation – an Introduction. ODI.org/methodslab Mayne, J: Contribution analysis (2008): An approach to exploring cause & effect. ILAC Brief 16 Baptist, C & Befani, B (2015): Qualitative Comparative Analysis – A Rigorous Qualitative Method for Assessing Impact 74

76. 7. Suggestions for evaluating the added value of cooperation – Strategic Added Value

77. Suggestions for evaluating the added value of cooperation – SAV Cooperation a key element of our programmes – part of the Cohesion agenda Our interventions primarily about delivery... but changing attitudes & behaviours also important strategically Important to think carefully about these strategic intents Beware – outcomes & results are harder to assess & prove than for ‘straight’ project delivery 76

78. Analytical Framework for assessing SAV – the English Experience 5 aspects of SAV  Strategic leadership & catalyst: Articulating & communicating development needs in the programme area, opportunities & solutions to partners & solutions to partners & stakeholders in the programme area & elsewhere  Strategic influence: Carrying-out or stimulating activity that defines the distinctive roles of partners, gets them to commit to shared strategic objectives & to behave & allocate their resources accordingly 77

79. Analytical Framework for assessing SAV – the English Experience  Leverage: Providing/securing financial & other incentives to mobilise partner & stakeholder resources – equipment & people, as well as funding  Synergy: Using organisational capacity, knowledge & expertise to improve information exchange & knowledge transfer & coordination &/or integration of the design & delivery of interventions between partners 78

80. Analytical Framework for assessing SAV – the English Experience  Finally, Engagement: Setting-up the mechanisms & incentives for the more effective & deliberative engagement of stakeholders in the design & delivery of programme emphases 79

81. Contact Simon Pringle SQW Ltd springle@sqw.co.uk +44 7915 650 732 80