1. Study Design for Quantitative Evaluation ©2015 The Water Institute

2. Objectives To understand the strengths and limitations of basic quantitative study designs To understand main sources of error resulting from poor study design To know how to control or minimize these sources of error. ©2015 The Water Institute

3. A possible scenario You get a memo from headquarters asking how can you show, quickly, that your new water supply programs are “sustainably functional”, and “better than average” How would you design a study to do this? ©2015 The Water Institute

4. Let’s look at some ways to do this….. 1. Visit all the systems your program is built 2. Assess what fraction is working well 3.Compare with overall fraction of water supplies in your area that are working ©2015 The Water Institute

5. CROSS-SECTIONAL STUDIES ©2015 The Water Institute

6. Cross-sectional Studies 200020102020 Time Assess your completed systems and access to water ©2015 The Water Institute

7. What are you comparing? An external evaluator from HQ comes along and raises questions: – How old are your systems? – How old are the systems with which you are comparing your systems? – How were the communities in which you worked selected? – Are they similar, or different, from your comparison group? ©2015 The Water Institute

8. Control Groups The first question (system age) is relatively easy to answer by choosing your comparison group carefully. The second one (innate differences between intervention and control groups) is much more difficult, and is a fundamental limit to cross- sectional studies, ©2015 The Water Institute

9. Cross-Sectional Studies Strengths – Simple – Fast…snapshot at one point of time Weaknesses – Association, not causation – Can mislead, if not thought through ©2015 The Water Institute

10. A second chance to solve the problem Fortunately, headquarters is now concerned about the long term answer to these questions, and is seeking advice on how to do this better. They want to know how well your “training of water committees” program works to improve access. If you had more money and time available, what could you do? ©2015 The Water Institute

11. COHORT STUDIES ©2015 The Water Institute

12. More detailed studies Instead of only looking at one time, you could follow a group of your projects, and a group of other projects over time. These are called “cohort” studies, and the groups that you follow over time are called “cohorts”. ©2015 The Water Institute

13. Cohort Studies Measure the intervention and the outcome at the start Measure the outcome the same way at a later date Measure other relevant variables as needed to test hypotheses ©2015 The Water Institute

14. Cohort Design 200020102020 Time Assess training for water committees and access to water Assess access to water ©2015 The Water Institute

15. Cohort Studies Strengths Tells a story over time Cause and effect are clearer Weaknesses Takes more time, money ©2015 The Water Institute

16. The need for controls 2000 20102020 Time Assess water committees that you have trained and access to water Assess access to water 2000 20102020 Time Assess water committees that have not received your training and access to water Assess access to water Intervention Control ©2015 The Water Institute

17. A better example 2000 20102020 Time Assess well drillers that you have trained and access to water Assess access to water 2000 20102020 Time Assess well drillers that have not received your training and access to water Assess access to water Intervention Control ©2015 The Water Institute

18. Intervention and control groups What are you comparing with what? – Do you want to compare with other similar programs? – Do you want to compare with communities with no programs? – Do you want to compare with national average, or local average? Your intervention and control groups define what you can say from your monitoring/evaluation ©2015 The Water Institute

19. Greater insight costs money! Simple cross-sectional study – One group, one time snapshot Simple cohort study – One group, followed over time Controlled cohort study – Intervention and control groups followed over time ©2015 The Water Institute

20. CONFOUNDING & CONFUSION 20 Source: Simonkneebone.com ©2015 The Water Institute

21. Some sources of error Selected populations or samples reflect unintentional bias or restriction Control groups reflect unintended bias or are inappropriate Ecological fallacy. Trends between groups are falsely assigned to individual differences within the groups Confounding. Statistical associations between factors, causes and effects that do not reflect the causal chain Thinking about these problems when designing your evaluation is much better than waiting until after all the data are collected! 21 ©2015 The Water Institute

22. A warning from a health study The example which follows is from a health studies of WaSH, but issues/approaches are the same for measuring “health of water systems” as well as “the health of people” Experience shows health effects from WaSH require are difficult to measure, and erroneous conclusions are easily drawn! Here is a story of two villages in Africa, one with mostly piped water, another using hand-dug wells… 22 ©2015 The Water Institute

23. Compare two communities’ water services VillagePiped Water (# of villagers) Dug Home (# of villagers) % Piped Water% diarrhea (children <5) Namabengo2167076%7% Mkongo10013443%32% Researchers noted that only 7% of Namabengo’s children had diarrhea in one week, while 32% in Mkongo had diarrhea… and Namabengo has more people with better service than Mkongo. Does the better water supply in Namabengo make the difference? ©2015 The Water Institute

24. Not if you look carefully! VillagePiped Water (children with diarrhea /# of villagers) Dug Home (children with diarrhea /# of villagers) Avg. diarrhea (children <5 years of age /total population) Namabengo15/216 (7%)5/70 (7%)20/286 (7%) Mkongo37/100 (37%)39/134 (29%) 76/234 (32%) Children < 5 years with diarrhea during previous week Source: Prag JB et al. (1983) Water Master Plan for Iringa, Ruvuma and Mbeya Regions, Tanzania Vol. 13 Ch. 11 ©2015 The Water Institute

25. Ecological Fallacy “ Ecological fallacies” occur when an association between groups is held to apply to individuals within the groups. – While a village with more piped water had less disease, the individuals using piped water in each village were not significantly more healthy. 25 ©2015 The Water Institute

26. Typhoid Fever & Telephone Poles An epidemiologist once showed: – as the number of telephone poles increased in the US, typhoid fever decreased. 1 – We are also sure that traffic deaths also increased (from accidents NOT involving telephone poles) Did the telephone poles reduce the typhoid, or increase the traffic deaths? _____________________________________________ 1 Kawata, K. (1978) Of Typhoid Fever and Telephone Poles: Deceptive data on the effect of water supply and privies on health in tropical countries Prog. Wat. Tech. Vol 11, Nos 1 – 2, pp 37-43. 26 ©2015 The Water Institute

27. A causal model 27 Fewer typhoid cases Economic Growth Better Water More Telephone Poles More traffic deaths More Cars Causal factor Irrelevant Causal chain Non-causal Association Better Education Better Sanitation ©2015 The Water Institute

28. Confounding A “confounding variable” is: – Statistically associated with a cause – Statistically associated with the effect of interest – NOT on the causal chain between the cause and the effect of interest Confounding variables matter, because they are often mistaken for causes, or distort findings ©2015 The Water Institute

29. Confounding and Control Groups In one cross-sectional study, sanitation is highly correlated with low diarrhea People with sanitation had less diarrhea than people without sanitation Question: What kinds of people invest in sanitation? 29 ©2015 The Water Institute

30. What to do about confounding? Identify likely confounders, and measure them! – Can then often control for them statistically Randomize between control and intervention at the outset of a cohort study – Random means “rigorously random” by statistical methods, it does not mean “haphazard” 30 ©2015 The Water Institute

31. PROCESS OF STUDY DESIGN ©2015 The Water Institute

32. Basic stages of study design Define questions to study Identify your statistical support! Define intervention (cause) and outcome (effect) to study Identify other factors that influence the outcome to be measured…”map” them Determine whether x-sectional or longitudinal study Define populations of interest (including controls if needed) Define sampling strategy of populations to minimize bias Define methods of data collection Define methods of analysis -- The rest is implementation….made MUCH easier by good design! 32 ©2015 The Water Institute

33. RECAP 33 ©2015 The Water Institute

34. Study designs Examples have been from “health studies”, but logic and issues are the same to study “healthy water supplies” or “healthy sanitation programs”. Cross-sectional studies are a snapshot in time  Relatively quick and cheap  Can’t establish cause and effect, only association  Good to generate ideas Cohort studies occur over time  Take more time, and more money  Can investigate causation,  For outcome assessment, need control groups 34 ©2015 The Water Institute

35. Process of study design An iterative process, as constraints become apparent (e.g. on sampling, timing, etc.) – Plan the data analysis from the beginning…so that you KNOW you will collect enough of the righ data to answer your question! 35 ©2015 The Water Institute