Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Using Qualitative Methods in Impact Evaluation Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1,

Similar presentations


Presentation on theme: "Introduction to Using Qualitative Methods in Impact Evaluation Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1,"— Presentation transcript:

1 Introduction to Using Qualitative Methods in Impact Evaluation Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1, 2010

2 Outline of Presentation  What are qualitative methods?  Why include qualitative methods in impact evaluation?  How can qualitative methods can be used in your impact evaluations?  Challenges to effectively using qualitative methods in impact evaluation

3 Multiple Reasons for Using Participatory Methods  This morning’s session emphasized social justice and accountability as reasons to utilize participatory methods in impact evaluation  There are are also methodological justifications for integrating participatory assessment and other qualitative measures with randomized impact evaluation

4 What Are Qualitative Methods?  Qualitative methods are a diverse set of research tools that are primarily concerned with the meaning of social actions, statements, and documents  Both alone and in combination with randomized evaluation and sample surveys, these methods provide leverage with respect to:  Describing and categorizing social contexts  Interpreting the social behavior, actions, and outcomes  Tracing the specific processes and mechanisms whereby programs lead to effects

5 The Diversity of Qualitative Methods  Participatory and Action-Based Research  Interviewing and Discussion  Content Analysis of Documents and Records

6 The Diversity of Qualitative Methods  Participatory and Action-Based Research  Systematic Social Observation  Drama and Role Playing  Maps, Models, and Diagrams  Ranking and Scoring  Interviewing and Discussion  Content Analysis of Documents and Records

7 The Diversity of Qualitative Methods  Participatory and Action-Based Research  Interviewing and Discussion  In-Depth, Semi-Structured Interviews  Focus Group Discussions  Oral Histories of Individuals, Communities, or Projects  Content Analysis of Documents and Records

8 The Diversity of Qualitative Methods  Participatory and Action-Based Research  Interviewing and Discussion  Content Analysis of Documents and Records  Diaries or Personal Time/Financial Records  Archives and Government Records  Newspapers and Civil Society Reports

9 Analyzing Qualitative Data: The Role of Narrative  Despite the differences between these methods of collecting qualitative data, the common principle is using close observation to understand the meaning of what people say and do  As a result, the writing of field notes and systematic the aggregation of notes into narrative case studies are a central skill in analyzing qualitative data

10 Combining Qualitative and Quantitative Methods  Qualitative methods can be linked to randomized evaluation  Before Implementation:  Use interviews, participation, and observation to design contextually relevant surveys and behavioral activities  Use observation or participation to design interventions that can test the interaction of different program components  After Implementation:  Aggregating and coding the content of interviews, behavioral observation, and newspapers to cross-check survey data  Analyzing process to understand the lack of statistically significant differences ▪ Implementation failure vs. design failure?

11 Homework: Qualitative Methods and Your Projects When you meet with your clinic groups this afternoon, consider this question: What types of qualitative methods would add value to the evaluation of your project? Focus on three things: 1) Identify each question you want to answer 2) Choose a set of qualitative methods to integrate into your evaluation design for each question and how/when you would use each method 3) Why these methods add value to your research design

12 Example 1: Integrating Local Context into Program Design  The Study: Evaluation of a waste management program in North-Central Kenya (Sheely 2010)  Specific Qualitative Tools Used: Long-Term Observation, Oral Histories, Archival Research  Specific Quantitative Tools Used: Randomized Assignment of Villages to Different Mixes of Enforcement Institutions  Implication for Impact Evaluations: Close analysis of local context can create opportunities for randomization and ideas for program design

13 Example 2: Using Local Knowledge to Assess Measurement Validity  The Study: Assessment of the effect of cultural framing on behavior in trust games in Maasai communities (Cronk 2007)  Specific Qualitative Tools Used: In-Depth Interviewing  Specific Quantitative Tools Used: Behavioral Economics Games in the Field  Implication for Impact Evaluations: Accurately measuring outcomes such as social cohesion may require careful attention to specific local culture practices

14 Why Include Qualitative Methods in Impact Evaluation?  Systematically including local context in program design  Understanding and preventing potential sources of Measurement Error  Cross-Checking and Interpreting Findings  Understanding causal mechanisms linking programs, context, and outcomes

15 Qualitative Methods: Methodological Limitations  Small, Nonrepresentative samples  Researcher/Facilitator Bias  Possibility of Contamination of Control Groups when combining with Randomized Evaluation  Lack of standards for comparing qualitative and quantitative findings

16 Qualitative Methods: Practical Limitations  Time and Cost of Qualitative Research  Understanding and Documenting Context Requires Time to Develop Mutual Trust Between Researcher and Community  Increased Staffing and Training due to more diverse skill sets  Coordination and Management Problems Associated With Larger Teams

17 Conclusions: Mixing Qualitative and Quantitative Methods  Qualitative Methods are a diverse set of tools and skills that require training, practice, and careful research design  Qualitative methods can help to better understand why a given program did or did not have an effect  Helping to develop better treatments and outcome measures for randomized evaluations  Using local knowledge and cultural context to interpret the meaning of social behavior and process of intervention  Ideal mix of qualitative and quantiative methods depends on the questions you want to answer and the local context

18 REFERENCES AND SUGGESTIONS FOR FURTHER READING Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2 nd revised edition) Thousand Oaks, CA: Sage Publications Barron, Patrick, Rachael Diprose and Michael Woolcock (2010) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press (in press) Chambers, Robert Whose Reality Counts: Putting the First Last, ITDG Publishing. Cronk, Lee “The influence of cultural framing on play in the trust game: a Maasai example” Evolution and Human Behavior. Kanbur, Ravi, ed Q-Squared? Qualitative and Quantitative Methods of Poverty Appraisal Malan, N “On the Relationship between participatory research and participatory development.” Africanus 2: Ravallion, Martin “How Can Qualitative Methods Help in Measuring Poverty,” Qual-Quant: Qualitative and Quantitative Poverty Appraisal- Complementarities, Tensions and the Way Forward, Permanent Black Publishers, Rodrik, Dani “The New Development Economics: We Shall Experiment, but How Shall we Learn?” Paper Prepared for the Brookings Development Conference, May Sheely, Ryan “Community Governance, Collective Action, and the Maintenance of Local Public Goods: Qualitative and Experimental Evidence from Rural Kenya.” Better Governance for Better Health Conference, Stanford University. April 26-27, Van der Riet, Mary “Participatory Research ant he Philosophy of Social Science.” Qualitative Inquiry 14 (4): White, Howard “Combining Quantitative and Qualitative Approaches in Poverty Analysis,” World Development. Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14


Download ppt "Introduction to Using Qualitative Methods in Impact Evaluation Ryan Sheely Kennedy School of Government Harvard University DIME Workshop, Dubai June 1,"

Similar presentations


Ads by Google