1. The Most Significant Change Technique (MSC) Dr Jessica Dart Clear Horizon

2. MSC –Form of qualitative, participatory M&E –Based on ‘stories’ of significant change –Developed by Davies 1996 - Bangladesh –Now used in numerous development programs and in the public sector

3. Qualitative vs quantitative monitoring Quantitative Focus on measurement Closed questions About ‘proving’ Easy to aggregate Deductive Static Goal displacement can be a problem Qualitative Focus on questioning Open questions About learning Hard to aggregate Inductive Dynamic Goal displacement is not an issue

4. Goal displacement Creaming Not about learning Don’t tell you what you don’t know you need to know Limitations of indicator based monitoring

5. Qualitative monitoring Can be used in conjunction with conventional output monitoring Is usually more aimed at learning than accountability

6. Why stories? People tell stories naturally - indigenous Stories can deal with complexity and context People remember stories Stories can carry hard messages /undiscussables But stories not known for accuracy/truth

7. Use of stories in MSC Collection of stories + systematic, collective interpretation = storytelling can be effectively harnessed for participatory evaluation Because interpretations tell another story & process has beneficial outcomes for evaluation utilisation

8. Overview of MSC 1. Determine sorts of change to monitor 2. Collect stories 3. Review & filter stories regularly 4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

9. Overview of MSC 1. Determine sorts of change to monitor 2. Collect stories 3. Review & filter stories regularly 4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

10. Example Target 10 Dairy Extension Project Four regions in Victoria, 50 staff 1999-2000 trail of the approach Still continues today

11. Step 1- Selection of ‘domains’ of change 3 broad ‘domains’ of changes to be monitored at the project level: Changes in on-farm practice Changes in farmer-decision making skills Changes in profitability Any other type of change * Not precisely defined

12. Step 2 - Collect stories “During the last month, in your opinion, what do you think was the most significant change that took place as a result of the project?” The respondent (farmer, extension worker or industry rep) answers in 2 parts: 1) descriptive 2) explanatory

13. Step 3 – Review & filtering process The stories were reviewed by: –The regional committees (every 2-3 months) –Statewide Executive (every 2-3 months) The stories are reviewed using a facilitated process at the state and funder levels

14. Funder meeting State meetings Region 1Region 2 Region 3 Region 4 Story tellers feedback flow of stories

15. Step 4 - Collate & review selected stories 1.In total 134 stories were collected - 80% from extension staff 2.A booklet containing 24 selected stories accompanied by the selection criteria & comments 3.Purchasers provide feedback to project after reviewing booklet

16. Step 5 – Monitor process & verify stories A database was developed to keep track of all stories Secondary analysis at end of reporting period In this case selected stories were not verified Storytellers were asked to check final stories in report

17. Impact of MSC Staff gained more fully shared vision Process boosted their morale Process saw farmers, staff, collaborators sitting together and interpreting qualitative data & casting evaluative judgements Project committees became better at conceptualising impact

18. Use of stories In addition to reporting, stories were used: –To improve planning –To help explain a point to a farmer –To recruit new participants –To help explain a point to another member of staff –For PR

19. MSC vs quantitative monitoring MSC Focus on measurement Closed questions Project out About ‘proving’ Deductive Static Inclusive Central tendencies Qualitative Focus on questioning Open questions Context in About learning Inductive Dynamic Selective Outer edges of experience

20. Goals Goal-based evaluation From the viewpoint of the Participants change Extent to which they were achieved From the view point of the program staff + consultation Program outContext in From the viewpoint of the Participants Program

21. Purpose of MSC in M&E  Primary purpose to facilitate improvement by:  focusing direction of work towards explicitly valued directions  eg. what do we really want to achieve and how will we produce more of it?  Contributes to summative evaluation:  Information about unexpected outcomes  Performance information concerning very best success stories  Can inform criteria used to judge projects

22. MSC Creates space for stakeholders to reflect, to make sense of complex changes Provides dialogue to help make sense of each other’s values Facilitates dynamic dialogue ie. “what do we really want to achieve and how will we produce more of it?” Excellent for participatory programs with diverse, complex outcomes, & multiple stakeholders