1. Concepts and Application Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH Impact Monitoring in Value Chain Promotion Heike Höffler, GTZ Kenya First sequence of thoughts, prepared for the Training Course Promoting Promoting Value Chains for Agribusiness Development in Africa held in Nairobi, 4 th - 7 th April 2005

2. 2 Impact Monitoring: Concepts and Application Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 2 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 1 Practice: PSDA Impact Monitoring System 3 Heike Höffler Kenya

3. 3 Impact Monitoring: Concepts and Application Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 2 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 1 Practice: PSDA Impact Monitoring System 3 Heike Höffler Kenya

4. 4 Monitoring is an ongoing activity! Projects and Programmes need to follow-up their progress Successes and failures need to be looked at while projects and programmes are running Monitoring is an internalised process of team communication, continuously undertaken while implementing, whereas Evaluation is an act of stopping implementation to reflect past activities (but drawing from information from monitoring). 1 Heike Höffler Kenya

5. 5 The World of Impact Monitoring Impact ChainImpact ModelImpact IndicatorsOutputsOutcomeOutreachBenefitsPerformance MonitoringInput MonitoringActivity MonitoringOutput MonitoringImplementation MonitoringImpact MonitoringImpact AssessmentProcess MonitoringResult-based MonitoringEvaluation Heike Höffler Kenya

6. 6 The World of Impact Monitoring Impact ChainImpact ModelImpact IndicatorsOutputsOutcomeOutreachBenefitsPerformance MonitoringInput MonitoringActivity MonitoringOutput MonitoringImplementation MonitoringImpact MonitoringImpact AssessmentProcess MonitoringResult-based MonitoringEvaluation Heike Höffler Kenya

7. 7 The World of Impact Monitoring Performance MonitoringImpact MonitoringImpact Assessment Input Monitoring Activity Monitoring Implementation Monitoring Process Monitoring Output Monitoring Impact Chain Impact Model Impact Indicators Result-based Monitoring OutputsImpactsBenefits Outreach Evaluation Outcome Heike Höffler Kenya

8. 8 What to monitor in projects – Performance & Impact Monitoring 1 Performance Observing outputs against planned activities Providing information for project management Day-to-day activity To trigger short-term adjustments in operation Impact Focussing on effect of outputs: impacts! Observing the direct benefit of outputs Strategic steering of implementation To self-evaluate whether activities contribute to objectives Heike Höffler Kenya

9. 9 Impacts are...... changes that have a causal - or at least a plausible - link to a project/programme... a change of circumstances as a consequence of an intervention, it can be intended or unintended, positive or negative.... there: from the first moment of intervention and they continue to occur all the time.... rather the result of social interaction than a straight-forward interventions... the result of complex interactions and thus, a complex matter to deal with! 1 Heike Höffler Kenya

10. 10 Why concentrating on Impacts? Broad international discussion late 90ies Criticism about the efficiency of development cooperation Criticism about the Monitoring and Evaluation system of GTZ 1 BMZ is a contractor, GTZ is an agent New structure of the project documents Changes in political dialogue Changes in Report obligations AURA Heike Höffler Kenya

11. 11 1 The GTZ Impact Model (I) Impact Orientation has become a principle of GTZs corporate development. quality at entry quality at exit what has been done what has changed Heike Höffler Kenya

12. ProjectUse of Output Direct Benefit Indirect Benefit Engel, P., The Social Organization of Innovation, 1997 Model of Interaction 1 Heike Höffler Kenya

13. GTZ Input Use of Outputs Poverty Alleviation Direct Benefit (Goal) Outputs Indirect Benefit Activities The GTZ Impact Model (II) Attribution? 1 Partner Heike Höffler Kenya Observation !

14. 14 Illustration: which of the 100 Bricks did we donate?

15. GTZ Input Use of Outputs Poverty Alleviation Direct Benefit (Goal) Outputs Indirect Benefit Activities The GTZ Impact Model (II) 1 Partner diff. Impact Levels Heike Höffler Kenya

16. 16 Increase in rural employment & income Indirect Benefits Direct Benefit (Goal) Use of Output Output Activities Inputs Producers can access international markets Market information is used to change production standards Presentation of Research to stakeholders and publishing Market research for a strategic agricultural product Advisory Services in the Ministry of Agriculture The GTZ Impact Model (III) 1 Attribution Gap Pro-poor Rural Economomic Growth Heike Höffler Kenya

17. 17 Impact Models form the essential methodological part in project design! Project reports are reduced to the information relevant for political intervention, i.e. –Changes in risk –Changes in assumptions and: IMPACTS! i.e. –Which changes can be observed? –Which impacts can be plausibly attributed to the project? –Are there unintended impacts? Consequences for GTZ Projects 1 Changes in Monitoring! Heike Höffler Kenya

18. 18

19. 19 Theory: Six Steps (GTZ) Step 1:Identify the System Boundaries Step 2: Agree on Purpose and Procedures for Results-based Monitoring Step 3:Agree on Results Hypotheses Step 4:Review Indicators and Define Milestones Step 5:Conduct Data Survey Step 6:Using Monitoring Results 1 Heike Höffler Kenya GTZ 2004: Result-based Monitoring: Guidelines for Technical Cooperation Projects and Programmes

20. 20 Theory: Six Steps (Herweg/Steiner) Step 1:Involvement of Stakeholders and Information Management Step 2: Review of Problem Analysis Step 3:Formulation of Impact Hypotheses Step 4:Selection of impact Indicators Step 5:Development and Application of Impact Monitoring Methods Step 6:Impact Assessment & Follow-up 1 Heike Höffler Kenya Herweg, K & Steiner, K. 2002: Impact Monitoring and Assessment, Vol. I & II

21. 21 Theory: Seven Steps (MAPP, GDI) Step 1:Preparing a Life Line Step 2: Preparing a Trend Analysis Step 3:Cross-checking with other sources Step 4:Compiling an Intervention List Step 5:Developing the Influence Matrix (connecting trends and interventions) Step 6:Developing the Impact Profile Step 7:Attribution of Impacts to MDGs 1 Heike Höffler Kenya DIE 2004: Briefing Paper: Impact Analysis of Development Cooperation is Feasible

22. 22 Impact Monitoring: Concepts and Application Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 2 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 1 Practice: PSDA Impact Monitoring System 3 Heike Höffler Kenya

23. 23 Impact Models: A chain of changes What do we expect to happen in value chain promotion? 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction C H A N G E! Intervention; i.e. extension Trans- formation TradeConsumption Specific Inputs Heike Höffler Kenya

24. 24 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation Trade 2 Defining the Impact Hypothesis (I) Consumption Use of Output Farmer group gets EuroGap certified Output Activities Training on EuropGap Compliance and Certification Inputs Advisory Services for Extension Providers Production Direct Benefit (Goal) Small-scale producers access international markets Farmer Groups are setting up an Internal Control System Heike Höffler Kenya

25. 25 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation Trade 2 Defining the Impact Hypothesis (II) Consumption Use of Output Output Activities Inputs Production Direct Benefit (Goal) Heike Höffler Kenya

26. 26 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation Trade 2 Defining the Impact Hypothesis (III) Consumption Use of Output Output Activities Inputs Production Direct Benefit (Goal) Heike Höffler Kenya

27. 27 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation Trade 2 Defining the Impact Hypothesis (IV) Consumption Use of Output Output Activities Inputs Production Direct Benefit (Goal) Heike Höffler Kenya

28. 28 Defining the Impact Hypothesis (V) 2 Use of Output Output Activities Inputs Direct Benefit (Goal) Each intervention undertaken along a value chain should follow an impact model; thus: each value chain to be promoted needs a concept of impact hypothesis. All impact hypotheses should be derived from the project goal; thus need to be derived from the impact indicators! Heike Höffler Kenya

29. 29 Impact Hypotheses – Food for Thought: 2 Pre-formulation of impact chains can lead to mono-causal conclusions. Linear following of impact chains can fade out the complex interaction of interventions and impacts. The higher the impact level, the more a context oriented approach is needed, taking into account development trends. Danger of mono-causality ! Heike Höffler Kenya

30. 30 Defining the Scope of Influence (I) Specific single interventions 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction Heike Höffler Kenya

31. 31 Defining the Scope of Influence (I) Specific single interventions 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction Heike Höffler Kenya

32. 32 Defining the Scope of Influence (II) Chain promotion at all stages 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction Heike Höffler Kenya

33. 33 macro meso micro Defining the Scope of Influence (III) Full chain promotion, all stages, all levels 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction Subsector-specific Technical Agencies Subsector-specific BDS providers AssociationsLocal Government, Providers of Utilities / Infrastructure National Government (Line Ministries) & Public Agencies Heike Höffler Kenya

34. 34 Impact Hypothesis - Conclusion 2 Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Specific Inputs Trans- formation TradeConsumptionProduction Subsector-specific Technical Agencies Subsector-specific BDS providers AssociationsLocal Government, Providers of Utilities / Infrastructure National Government (Line Ministries) & Public Agencies Heike Höffler Kenya 2

35. 35 2 Benchmarks, Milestones & Indicators Benchmark Data & Information The state of the art at given point in time (i.e. programme start) Milestones Desired output at a given point in time for (programme) achievements Indicators Determinants for checking whether activities led to the expected output (performance) or expected implications (impact). Heike Höffler Kenya

36. 36 2 Benchmarks, Milestones & Indicators (I) Indicators Heike Höffler Kenya 0 t Benchmark Data Programme level Value chain level Milestones Progress Reports Evaluation

37. 37 Specific Inputs Trans- formation Trade 2 Benchmarks in Value Chains ConsumptionProduction Usage of Inputs Usage of services Outreach: How many are involved? Employment Gross margins Regional distribution Number of processors Degree of vertical concentra- tion and integration Market Concentra- tion Entry Barriers Price Volatility Seasonality Trade margins Quality Food Safety Product Diversity Heike Höffler Kenya

38. 38 Specific Inputs Trans- formation Trade From Benchmarks to Milestones ConsumptionProduction Output Activities Inputs 3 processing firms Low process quality High costs of processing In 18 Month: 2 more firms established At least 2 certified under ISO xxx Cost-effectiveness of at least 2 firms improved by x % Milestones serve as process indicators. They are directly derived from programme indicators and the impact hypothesis. 2 Heike Höffler Kenya

39. 39 Specific Inputs Trans- formation Trade Benchmarks & Milestones in Value Chains ConsumptionProduction Try to build these couples per stage in a chain that you are targeting with your project. Not all Benchmarks need to be quantitative data; qualitative description also serves the purpose. Market Analyses need to take the need for benchmarks into account (into the TORs!) We might not be the first to undertake this – keep looking for partners in gathering data! 2 Heike Höffler Kenya

40. 40 2 From Milestones to Indicators – or was it the other way round? Output Activities Inputs Use of Output Direct Benefit (Goal) Production Heike Höffler Kenya Several Milestones in a chain Single Milestone Programme Indicator(s) Value chain Indicator(s)

41. 41 Specific Inputs Trans- formation Trade Benchmarks, Milestones & Indicators (II) ConsumptionProduction Output Activities Inputs 3 processing firms Low process quality High costs of procesing In 18 Months: 2 more firms established At least 2 certified under ISO xxx Cost-effectiveness of at least 2 firms improved by x %... 2 Heike Höffler Kenya Use of Output Direct Benefit (Goal) Impact Indicator, Component 3, Phase 2 The number of profitably operating private processors doubles (2002: 6)

42. 42

43. 43 Example: Employment Customers Market 2 Customers Market 1 (Research Institute) Retailers A Retailers B Industrial SMEs Importers Large processing companies 3 2 1 Primary Producers 1 SM Primary Producers 2 n = 400n = 450n = 20 n = 6000 n = 40 n = 2400 n = 20 n = 200 3 Heike Höffler Kenya The number of primary producers doubles.

44. 44 100,00 Specific Inputs Production Trans- formation Trade Con- sumption Example: Value Added Specific Input providers Primary producers Traders Final Con- sumers Logistics centres, Industry Sales Prices / unit 60,0025,00 Cost / unit Income / unit Distribution in % - of value-added - of income 23,00 2,00 27,00 8,00 30,00 10,00 25 % 10 % 35 % 40 % 50 % 3 Heike Höffler Kenya The value added at producer level increases from 25% to 35%.

45. 45 Example: Better Business Relations Heike Höffler Kenya Specific Inputs Production Trans- formation Trade Con- sumption Specific Input providers Primary producers Traders Final Con- sumers Key informant interviews show that wholesale traders are more satisfied with the quality/quanity/reliability of produce delivered by small scale farmers. Interviews show that small cale farmers have improved relationship to traders and trust them.

46. 46 2 Indicators in Value Chains Some examples: Customer satisfaction Repeated customers Number of enterprises demanding a service Satisfaction with last service purchased Percentage of women-owned enterprises Heike Höffler Kenya There is increasing interest in, and use of, private sector tools for performance and impact measurement; i.e. consumer market research tools for measuring changes in markets and private sector business tools for measuring service provider performance.

47. 47 Methods of Measuring Impacts 2 Heike Höffler Kenya Market Studies Key Informant Interviews Point of Leverage – comparisons (before – after) Participatory Monitoring Time-series etc... Specific Inputs Production Trans- formation Trade Con- sumption Where to measure in the chain? What data – qualitative or quantitative? Which degree of participation?

48. 48 Example: Spider Web Diagrammes 2 Heike Höffler Kenya

49. 49 E-Val 2 Heike Höffler Kenya GTZ tool to catch different perceptions of projects and programmes. There are only subjective opinions to changes and impacts Triangulation of opinions: GTZ, partners, and target group

50. 50 Using Impact Monitoring Systems 2 Heike Höffler Kenya The objective of Impact Monitoring was.... Focussing on effect of outputs: impacts! Observing the direct benefit of outputs Strategic steering of implementation To self-evaluate whether activities contribute to objectives

51. 51 Using Impact Monitoring Systems 2 Heike Höffler Kenya SteeringObserve direct BenefitSelf-evaluation Steering Committee Reports Annual Progress Reports Participation! Discussion with Stakeholders Finetuning Communication Team Meetings Partner Meetings Other donors Private Sector

52. 52 Using Impact Monitoring Systems 2 Heike Höffler Kenya More ideas – lets be creative! Replanning Communication of Results Newsletter Business Fora Publications Go- No-go- Decisions Accountability

53. 53 Its better to be approximately right than precisely wrong! ( John Maynard Keynes ) Please email your comments & questions to heike.hoeffler@gtz.deheike.hoeffler@gtz.de Heike Höffler Kenya

54. 54 Impact Monitoring: Concepts and Application Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 2 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 1 Practice: PSDA Impact Monitoring System 3 Heike Höffler Kenya