Survey Training Pack Session 18 – Checking Data Analysis

Introduction Data analysis – done internally or outsourced – needs to be checked The data analysis plan is the starting point of your check – It specifies how indicators have been broken down into sub- indicators and what variables need to be created at the stage of analysis Using the data analysis plan, ideally you should check that the syntax developed by the analyst is coherent – In SPSS, to generate the syntax, the analyst must use the “paste” function before executing any command

(Re)visiting a few concepts Confidence intervals – estimated range of values that is likely (generally 95%) to include the unknown/desired parameter in the population Precision – width of the confidence interval – The wider it is, the less precise it is, and sometimes it may mean that the findings of the study are inconclusive Hypothesis testing – expected effect (“hypothesis”) can be discerned and plausibly shown, and is not owing to chance (or sampling) – Tiny effects can be ‘significant’

Unit of analysis Based on anecdotal evidence, IRM adoption can be limited to a specific proportion of the entire land cultivated with rice: – Trainees whose entire rice land was cultivated using IRM practices – Trainees who applied IRM practices on a portion of their entire rice cultivated land – Trainees who did not apply IRM practices Bearing the above in mind, what issue is apparent from Table 11 in the rice analysis (page 14)? How do you think Table 11 should be have been laid out?

How do you catch errors? The best tool to catch errors is using your programme knowledge and COMMON SENSE : – Sound understanding of the programmatic context and information needs Percentage calculation – understanding what is meaningful: – What question are you trying to answer? Stating the responses you expect is a useful way of isolating which percentage is useful to calculate. – What numerator and denominator do you want to use? How were the percentages calculated in Table 35? How would you report this information?

How do you catch errors? Denominator – many mistakes are made with respect to the denominator: – Valid responses versus overall denominator – issues with missing data – Knowing what questions in your tool allow you to identify the denominator (SPSS syntax) – But also understanding the link between different tables How would you check the denominator specified (i.e. the absolute value quoted) in Table 18 is the correct one? And Table 22?

Summary Data Analysis Plan is the starting point of your check When examining output tables, verify that – The correct unit of analysis is being used in each table – The correct denominator is being used in each table – That percentages have been calculated in a way which you find meaningful – Confidence intervals have been generated where required – The right statistical tests have been performed where required Your programmatic knowledge and your common sense are your best tools in helping you check data analysis