Challenges and solutions for the setting up of comprehensive M&E systems for Social Safety Nets International Social Protection Conference, Arusha Valentina Barca, 17 December 2014 Based on a Paper by Ramlatu Attah, Valentina Barca, Ian MacAuslan, Luca Pellerano, Luigi Peter Ragno, Nils Riemenschneider and Claire Simon

Introduction and approach One of the aspects of SSNs that is often given insufficient attention is the establishment of a comprehensive, strong and functioning programme Monitoring and Evaluation (M&E) system, that goes beyond the undertaking of a few specific ad hoc external studies to provide systematic and continuous information for internal improvement and external accountability. This paper: – develops a Conceptual Framework describing the supply and demand side factors that underpin the establishment of such a system – applies this framework to OPM’s experience supporting the design and implementation of M&E systems for SSNs in four countries:  Ghana - LEAP CT programme  Kenya - NSNP which harmonises, strengthens and scales up five existing CT programmes  Mozambique - four different SSNs, including the country’s major cash benefit PSSB  Moldova - ‘Ajutor Social’, a means-tested cash benefit December 2014© 2014 Oxford Policy Management Ltd2

3 Ability of the system to provide reliable, quality information Whether there is actual demand for and use of M&E info to support core programme/government activities at all levels

Supply: indicators and targets (a) Defining indicators based on information needs: in the four study countries a similar demand-driven approach was used to develop the content of the M&E system. With little differences, this included assessing information needs based on three distinct factors: – The programme’s objectives, Theory of Change and Results Framework; – The needs of different actors and stakeholders through field visits to local levels of programme administration, consultations with Ministry staff and (where necessary) consultations with other ministries that were engaged in the programme; – The functioning of key programme processes: for example, in Ghana, Mozambique and Moldova, business processes for each programme being monitored were traced so as to make sure that each aspect of programme implementation could be assessed within the M&E framework and linked to Service Standards. December 2014© 2014 Oxford Policy Management Ltd4

Risk! Overload… December 2014© 2014 Oxford Policy Management Ltd5

Supply: indicators and targets (b) Prioritising, refining and organising indicators as an iterative process: – To prioritise indicators, further consultations and workshops were organised with a range of stakeholders (the main users of the M&E system); – Refining indicators involved a careful assessment of each indicator to make sure it fulfilled the so-called ‘CREAM’ criteria and could be actually applied in practice. In Ghana and Mozambique, this involved mapping each indicator back to its constituting formula (numerator and denominator) and potential data source, as well as defining how often that indicator would be collected and by whom. – Once this process was completed, indicators were organised based on their use. In Moldova, and Ghana an important distinction was made between those focusing on programme operations, and those focusing on results. December 2014© 2014 Oxford Policy Management Ltd6

Supply: data sources In the four case study countries, potential data sources were evaluated and selected based on three main criteria: 1. Staging monitoring and evaluation components: the exact mix of monitoring versus evaluation activities needed based on the level of ‘maturity’ of the programme (monitoring up-front, evaluation further down). 2. Building on and combining existing data sources: internal and external  Programme MIS  Internal standard periodic reports  Ad-hoc qualitative studies  Existing national household surveys  Other administrative sources e.g. line-Ministry’s staff database, the Bank’s payments database, the MoE or MoH’s MIS etc  Externally contracted impact evaluations  Beneficiary Surveys or Citizen Perception Surveys 3. Minimising the burden of data collection and reporting: designed to be generated as an integral part of programme administration rather than as an additional task December 2014© 2014 Oxford Policy Management Ltd7

Supply: institutional arrangements “The formal and informal processes, procedures, rules, and mechanisms that bring monitoring activities into a coherent framework” (Bedi et al., 2006). Best practice: – Ensuring the institutional arrangements of the M&E system reflected the overall institutional structure of the programme. In Ghana, this meant ensuring that reporting lines reflected the existing LEAP administrative system, the incentive structure within that system, as well as taking account of the government’s objective of greater devolution of power to the districts. – Working as much as possible with existing systems, staff and processes and helping to improve them. In all four countries existing systems were enhanced, with a strong focus on Capacity Development. – Acknowledging that that building institutional arrangements with new actors for M&E purposes requires time, dedication and in some cases legal frameworks or memoranda of understanding. For example, in Moldova new questions on receipt of the Ajutor Social were added to the Household Budget Survey thanks to long negotiations with the Statistics Bureau. December 2014© 2014 Oxford Policy Management Ltd8

Demand: macro – national policy environment An ‘enabling’ national policy significantly improved the chances of stakeholders being actively involved in the design and use of the M&E system. On one side, this meant having a policy framework that is performance-oriented (as in Ghana and Kenya). On the other side, an enabling environment was fostered where there was a strong policy focus on SSNs, which was the case in all four case study countries. Designing integrated M&E systems was potentially easier in contexts where there was an overall institutional culture that fosters linkages between different ministries (accounting to each other) and has actors focused on planning (e.g. National Planning Commission). The role of donors pressing for accountability and attaching conditions to aid (e.g. World Bank Performance for Results (PfR) loan in Kenya) also had large impact on the overall value attached to M&E… A similar role could also be played by Civil Society (holding government to account or not), but this was not often the case in the four countries. Example of Moldova. December 2014© 2014 Oxford Policy Management Ltd9

Demand: meso – implementing agency, central and local Factors affecting use of information generated from the M&E system included: – The level of autonomy in decision-making and the incentives within the institution – How much the agency actually backed the development of an M&E system in the first place – The existence of a culture of benchmarking performance across different locations and learning from best practice across the country – The presence of a strong connection and liaison between central and decentralised levels based on mutual feedback, knowledge of differences across different local branches, and continuous engagement – Existence of Standard Service Agreements that helped to transparently frame objectives in terms of service delivery and were taken seriously by the implementing agency December 2014© 2014 Oxford Policy Management Ltd10

Demand: micro – individuals Staff at national and local levels will be more likely to embrace M&E data if they understand the potential usefulness of it – but need to overcome Catch 22! The ‘fear’ surrounding M&E can be reduced by shifting the focus from ‘controlling’ to ‘learning’ and building forums for local and central level administrators to compare and contrast their experiences and learn from each other. Reducing the capacity constraints of staff involved in M&E was the most difficult challenge in the four study countries as it required large investments for recruitment and training. December 2014© 2014 Oxford Policy Management Ltd11

Thank you