Presentation on theme: "Monitoring and Evaluation Use of statistics"— Presentation transcript:
1Monitoring and Evaluation Use of statistics 28/03/2017Monitoring and Evaluation Use of statisticsModule 5Session 9
2Use of Statistics in Monitoring and Evaluation 28/03/2017Use of Statistics in Monitoring and EvaluationWhy consider statistics in M&E?Where are the entry points for using stats?Improving the quality of indicatorsEnhancing monitoring of indicatorsInforming evaluations
31. Why use statistics in M&E? 28/03/20171. Why use statistics in M&E?“When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure it, then you cannot express it in numbers, your knowledge is of the meager and unsatisfactory kind.” – Lord Kelvin (British Physicist)
42. Entry points Planning – informing the selection of indicators 28/03/20172. Entry pointsPlanning – informing the selection of indicatorsMonitoring – measuring progress against quantitative indicator targetsEvaluation – using statistics from different sources to inform on achievement in context
53. Planning – enhancing the quality of indicators 28/03/20173. Planning – enhancing the quality of indicatorsDefinition:Indicators are signposts of change along the path to development. Indicators are what we observe in order to verify whether – or to what extent – it is true that progress is being made towards our goals, which define what we want to achieve. Indicators make it possible to demonstrate results. Indicators can also help in producing results by providing a reference point for monitoring, decision-making, stakeholder consultations and evaluation. In particular, indicators can help to:Measure progress and achievements;Clarify consistency between activities, outputs, outcomes and goals;Ensure legitimacy and accountability to all stakeholders by demonstrating progress;Assess project and staff performance.
628/03/2017Indicator qualitiesGood indicators have the following five characteristics:Numeric. While not always more objective, numerical precision lends itself better to an agreement over the future interpretation of data. On the other hand, factual indicators provide only a very crude “measurement” due to their limited scale (mostly yes/no). For a set of factual indicators, no monitoring system is needed, since the status is mostly known by stakeholders (e.g.: law passed by parliament: yes or no). If it not possible to avoid factual indicators due to the nature of the project, factual indicators should be at least be supplemented by numeric indicators in a comprehensive set.Objective. An indicator which involves a subjective judgement by somebody is not objective. For a good indicator, there has to be a general agreement over interpretation of data.
7Indicator qualities (2) 28/03/2017Indicator qualities (2)Specific. The indicator needs to be as specific as possible in terms of quantity, quality, time, location, target groups, baseline, targets etc.Relevant. The indicator needs to relate directly to the respective output, outcome or impact. In other words, a good indicator is a relevant “measure” for the objective.Feasible. Even if an indicator fulfils all other criteria, it is not useful if the data collection for the indicator is not feasible. First, data for the indicator needs to be easily available.
9Description of indicator 28/03/2017Description of indicatorPrecise definition:The definition must be detailed enough to ensure that different people at different times, given the task of collecting dataFor a given indicator, would collect identical types of data. Potentially ambiguous terms (for example: small farmers, poor households, disadvantaged groups) need to be clearly defined (for example: farmers with < 1 hectare of land, households below national poverty line).Unit of measure:Define the precise parameter used to describe the magnitude or size of the indicator (for example: number of individuals, percentage, shillings, hectares, cumulative, average, etc.).Disaggregated by:Identify how data will be separated to improve the breadth of understanding of results reported (for example: gender, district, urban/rural, etc.).
10Baseline and Targets Baseline: 28/03/2017Baseline and TargetsBaseline:The baseline is the value of the indicator prior to an action. The baseline value establishes the starting point from which change can be measured.Benchmarks:Benchmarks are values of the indicator while an action is still ongoing. Benchmarks are therefore intermediate targets.Target:The targets is the expected value of the indicator after an action.
114. Thinking about monitoring 28/03/20174. Thinking about monitoringPlanning for data acquisitionAttributes of good dataTiming of relevant data collection exercises
12Planning for data acquisition 28/03/2017Planning for data acquisitionData collection method and timing:Describe exactly and in detail how and when you will collect the data. Identify what methods and instruments you will use.Note any tool or survey required to collect the data. Attach data forms when necessary. Examples of data collection methods are secondary data, surveys, expert judgments, etc.).Data source:The data source is the entity from which the data are obtained (e.g. a government department, an NGO, other donors, etc.).Estimated cost of data acquisition:Provide a rough estimate of what it will cost to collect and analyze the data.Individual responsible and location of data storage:Describe who will take the lead for collecting this indicator. Describe how data will be stored over time and in what format.
13Data quality issues Known data limitations and significance: 28/03/2017Data quality issuesKnown data limitations and significance:Identify where data may be weak or limited.Identify actions taken or planned to address data limitations:
14Attributes of ‘good’ data 28/03/2017Attributes of ‘good’ dataThe source of data should be known. Typically someone at the agency that collected the data should be available to clarify and explain details;The reason for which quantitative data was collected should be known and documented. This is important because it helps to understand if for example there were any biases in response or if you need to account for interviewer/response biases/errors;Codified responses should be carefully documented at some place. Frequently responses are ranks or ordinal responses;
15Attributes of ‘good’ data (2) 28/03/2017Attributes of ‘good’ data (2)The date of collection, scale, frequency, sampling unit, enumeration units, selection process and coverage of data should be known. Thus the number (scale) of households (sampling unit) covered on a certain date, should be known. The number of times it is collected (frequency) and what region of the country it covers (coverage) should be known. If data is collected on livestock then the units that this is recorded in (thousands or millions) should be known. If the data does not include all the observations as initially planned, it is good to know the reason for this;The method used to identify respondents should be known. There are many methods to collect data and identify respondents. These can be random sample over a country, or a village, or variations thereof. Thus for example if data was collected randomly or to reflect proportionate representation of ethnic groups or in an interview format – then all of this should be known. Data can via a rapid rural assessment. The method for identifying respondents helps to know the extent to which data can be scaled-down or disaggregated.
16Attributes of ‘good’ data (3) 28/03/2017Attributes of ‘good’ data (3)Spatially explicit data that is either derived from satellite images or ground collected are also another type of data. Documentation for these data is also extremely important. Additional knowledge about scale and format of data is critical.Errors or faults in data collection (there are always some) should be known well.People working with the data should be very familiar with the software they use to extract data, transform data and construct data. Frequently a lot of errors in analyses occur because researchers are not familiar with the way their software handles data.
17Timing of data collection exercises 28/03/2017Timing of data collection exercisesMonitoring indicators require updating quarterly or biannually.Identify which reliable instruments may provide some useful data for district level monitoring (e.g. consumer price index in urban centres; biomass data; migration statistics etc)
185. Informing Evaluations 28/03/20175. Informing EvaluationsWhat type of data do evaluations require?What sources of data exist?How to access and manage that data?
19Evaluation data requirements 28/03/2017Evaluation data requirementsEvaluations ask the question of not only ‘what’ has and has not been achieved, but also ‘why’ it has or has not been achieved.The ‘what’ question requires answers to often quantitative indicators. For example, the PEAP evaluation needs to address the question of whether poverty headcount has gone down as per the target.Equally, an agricultural project evaluation might ask ‘what’ has changed against indicators such as:Increased yield/acreExpansion of acreageShare of high value crops in total production
20Evaluation data requirements (2) 28/03/2017Evaluation data requirements (2)The ‘why’ question of evaluation requires a broader investigation into data:Explanatory data: What else has been occurring that may have affected the intervention (e.g. reduction in international coffee prices; heavily floods in area etc)Trend data: what has occurred over time in key indicatorsMixing quantitative and qualitative data: using qualitative approaches (e.g. focus groups) to better understand possible cause and effect relationships
21What sources of data exist? 28/03/2017What sources of data exist?Best sources of quantitative data are:Surveys (representative at regional level)Demographic health surveyHousehold budget surveyNational household surveySero behavioural surveyNational service delivery surveyInformal cross-border surveyOther Evaluations and ReviewsCensuses (population, agriculture, etc)
2228/03/2017“Without data, all you are is just another person with an opinion” - UnknownSource: Statistical Literacy Project, Bureau of Development Policy, UNDP
23How to access that information 28/03/2017How to access that informationSurvey data sets are available in reports, and raw at UBOSPartner evaluation reports are available online, e.g. World BankN.B. Often necessary to visit key offices to identify and access key data sets