1. Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual Metrics Conference June 23-24, 2015

2. Overview  What you can expect  What we’re going to do  What APEP does

3. Starting point: I want to assess my impact, but I can’t run a large-scale RCT

4. When?  Random assignment isn’t possible  The intervention is changing over time  Impacts are not adequately captured by quantitative measures  Causal pathways are complex  Small-scale projects  We are not hypothesis testing

5. Alternatives? Quantitative  Surveys  Documents/records Qualitative  Interviews  Focus groups  Site visits  Open-ended survey questions  Document review  Participatory methods

6. Can we still say something meaningful about impact?

7. Absolutely!

8. How do I optimize my ability to collect meaningful and useful data on impact?

9. Prioritize Who is your audience?

10. Decisions, decisions, decisions…

11. Am I looking at my impact on SGBs or my impact on the people the SGBs serve? ANDE member does X Impact on SGB Impact on individuals, communities, etc. SGB does Y

12. Example:

13. Root Capital provides loans to Freshco Freshco is able to produce more high- yield, drought- resistant maize seed Farmers have increased yield (more food, increased income, surplus grain for local population) Greater access to high-quality seed for farmers

14. Am I looking at short-term or long-term impacts?

15. Am I in hypothesis testing mode or exploratory mode? Did I have X impact? What impacts did I have? How or why did I have an impact? vs.

16. Make the most of your measures

17. #1. Measure outcomes, not just outputs OutputsOutcomes Your products/services to SGBs. Impact.

18. #2. Not just “whether” but also “how” and “why” Did I have an impact? Whether How Why How did my activities contribute to a positive impact? Why didn’t I have an impact?

19. Vote with your You’ve worked with quantitative methods You’ve worked with qualitative methods

20. #3. Consider qualitative and quantitative Combine if possible! Quantitative Qualitative

21. #4. Sample with purpose

22. How big is big enough? Sample size calculator: http://enterprise- development.org/page/ calculator Sample size calculator: http://enterprise- development.org/page/ calculator

23. #5. Scout external factors

24. #6. Try for two (or more) Time 1Time 2 T1T2 T3 T4 T5

25. #7. Optimize questionnaires  Is this clear?  If everyone answers “yes,” what will that tell me? Ask good questions Be mindful of burden Beware of branching  Too many questions can lead to fatigue and skipping  Sub-groups may be too small to analyze

26. and share  Measuring outcomes rather than outputs  Measuring “how” or “why”  Combining qualitative and quantitative  Sampling  External influences  Collecting data at multiple points  Questionnaires

27. Ok, I’ve collected my data. Now what?

28. Data analysis: Things to check

29. Your sample  What’s my response rate?  Anybody under/over- represented?  Does the pre match the post?  Are there missing data?

30. Analysis Does it make sense to calculate statistical significance? Should you disaggregate? What’s the appropriate way to analyze your qualitative data? Is that calculation meaningful?

31. Assess your limits Attribution vs. Contribution Counterfactuals Generalizability

32. What to do when…

33. Group Exercise