The Most Significant Change Technique (MSC) Dr Jessica Dart Clear Horizon

MSC –Form of qualitative, participatory M&E –Based on ‘stories’ of significant change –Developed by Davies Bangladesh –Now used in numerous development programs and in the public sector

Qualitative vs quantitative monitoring Quantitative Focus on measurement Closed questions About ‘proving’ Easy to aggregate Deductive Static Goal displacement can be a problem Qualitative Focus on questioning Open questions About learning Hard to aggregate Inductive Dynamic Goal displacement is not an issue

Goal displacement Creaming Not about learning Don’t tell you what you don’t know you need to know Limitations of indicator based monitoring

Qualitative monitoring Can be used in conjunction with conventional output monitoring Is usually more aimed at learning than accountability

Why stories? People tell stories naturally - indigenous Stories can deal with complexity and context People remember stories Stories can carry hard messages /undiscussables But stories not known for accuracy/truth

Use of stories in MSC Collection of stories + systematic, collective interpretation = storytelling can be effectively harnessed for participatory evaluation Because interpretations tell another story & process has beneficial outcomes for evaluation utilisation

Overview of MSC 1. Determine sorts of change to monitor 2. Collect stories 3. Review & filter stories regularly 4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

Overview of MSC 1. Determine sorts of change to monitor 2. Collect stories 3. Review & filter stories regularly 4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

Example Target 10 Dairy Extension Project Four regions in Victoria, 50 staff trail of the approach Still continues today

Step 1- Selection of ‘domains’ of change 3 broad ‘domains’ of changes to be monitored at the project level: Changes in on-farm practice Changes in farmer-decision making skills Changes in profitability Any other type of change * Not precisely defined

Step 2 - Collect stories “During the last month, in your opinion, what do you think was the most significant change that took place as a result of the project?” The respondent (farmer, extension worker or industry rep) answers in 2 parts: 1) descriptive 2) explanatory

Step 3 – Review & filtering process The stories were reviewed by: –The regional committees (every 2-3 months) –Statewide Executive (every 2-3 months) The stories are reviewed using a facilitated process at the state and funder levels

Funder meeting State meetings Region 1Region 2 Region 3 Region 4 Story tellers feedback flow of stories

Step 4 - Collate & review selected stories 1.In total 134 stories were collected - 80% from extension staff 2.A booklet containing 24 selected stories accompanied by the selection criteria & comments 3.Purchasers provide feedback to project after reviewing booklet

Step 5 – Monitor process & verify stories A database was developed to keep track of all stories Secondary analysis at end of reporting period In this case selected stories were not verified Storytellers were asked to check final stories in report

Impact of MSC Staff gained more fully shared vision Process boosted their morale Process saw farmers, staff, collaborators sitting together and interpreting qualitative data & casting evaluative judgements Project committees became better at conceptualising impact

Use of stories In addition to reporting, stories were used: –To improve planning –To help explain a point to a farmer –To recruit new participants –To help explain a point to another member of staff –For PR

MSC vs quantitative monitoring MSC Focus on measurement Closed questions Project out About ‘proving’ Deductive Static Inclusive Central tendencies Qualitative Focus on questioning Open questions Context in About learning Inductive Dynamic Selective Outer edges of experience

Goals Goal-based evaluation From the viewpoint of the Participants change Extent to which they were achieved From the view point of the program staff + consultation Program outContext in From the viewpoint of the Participants Program

Purpose of MSC in M&E  Primary purpose to facilitate improvement by:  focusing direction of work towards explicitly valued directions  eg. what do we really want to achieve and how will we produce more of it?  Contributes to summative evaluation:  Information about unexpected outcomes  Performance information concerning very best success stories  Can inform criteria used to judge projects

MSC Creates space for stakeholders to reflect, to make sense of complex changes Provides dialogue to help make sense of each other’s values Facilitates dynamic dialogue ie. “what do we really want to achieve and how will we produce more of it?” Excellent for participatory programs with diverse, complex outcomes, & multiple stakeholders