TRENDS IN DEVELOPMENT OF NATIONAL MONITORING AND EVALUATION SYSTEMS FOR AIDS RESPONSE Kevin Kelly Inaugural SAMEA Conference March 2007, Johannesburg

Background Massive growth in multisectoral responses and social responses to AIDS? Massive growth in multisectoral responses and social responses to AIDS? How do we know what is being done and how we are doing? How do we know what is being done and how we are doing? How can we plan to develop and improve effectiveness of programmes? How can we plan to develop and improve effectiveness of programmes?

How far are we from this?

The three ones One AIDS action framework that provides the basis for co-ordinating the work of all partners One AIDS action framework that provides the basis for co-ordinating the work of all partners One national AIDS coordinating authority with a broad-based multisectoral mandate One national AIDS coordinating authority with a broad-based multisectoral mandate One country level monitoring and evaluation system One country level monitoring and evaluation system

The third one At global-level alignment around indicators linked to the United Nations Declaration of Commitment on HIV/AIDS. At global-level alignment around indicators linked to the United Nations Declaration of Commitment on HIV/AIDS. At country level, adoption of a core national monitoring and evaluation system for analyzing country performance of the national AIDS action framework. At country level, adoption of a core national monitoring and evaluation system for analyzing country performance of the national AIDS action framework. At programme level, investment in human capacity and infrastructure to meet national monitoring and evaluation needs. At programme level, investment in human capacity and infrastructure to meet national monitoring and evaluation needs.

Principles - GAMET Link the HIV M&E Road Map to the costing of the national HIV strategic plan of the country. Develop a multi-year M&E Road Map that covers the same time period as the national HIV strategic plan. Involve ALL HIV implementers at all levels.

Steps – GAMET (illustrative) Step 1. Advocate for a national HIV M&E Road Map to be developed and used. Step 2. Carry out a national HIV M&E capacity assessment. Step 3. Obtain copies of HIV M&E work plans and budgets from all relevant stakeholders: the NAC, development partners, national NGOs, government ministries, academic institutions, the National Bureau of Statistics. Step 4. Ensure that the M&E technical working group has wide representation from all stakeholders mentioned in the Principles for Developing an M&E Road Map (on the previous page). Step 5. Develop a draft HIV M&E Road Map (without costing) in a collaborative manner. Step 6. Build consensus around the activities to be included in the national HIV M&E Road Map.

Step 7. Submit the national HIV M&E Road Map for approval to the M&E technical working group (its membership should include all relevant development partners). Step 8. Cost the HIV M&E Road Map. Step 10. Prioritize the Road Map activities and ensure that all priority activities are either funded from the NAC’s own government resources, or from secured development partner resources. Step 11. Secure and assign resources for each Road Map activity. Step 13. Update and finalize the membership of the HIV M&E technical working group (TWG). Step 15. Review and update the M&E Road Map once a year, before the NAC budget and work plan for the following year are approved.

The proto-type 1. INPUT INDICATORS 2. OUTPUT INDICATORS 3. OUTCOME INDICATORS 4. IMPACT INDICATORS LEVEL PROVINCIAL - Routine surveillance - Periodic research PROVINCIAL - Periodic measurement through research - Not linked to implementers ALL IMPLEMENTERS Output monitoring and reporting PROGRAMME LEVEL -Internal input monitoring POMS* ECAC ↔ PMEU Tracking progress made in achievement of provincial objectives EVALUATION MONITORING *Provincial Output Monitoring System

Common problems in such frameworks Answering the ‘so what’ question Lack of human resources for HIV M&E Skill levels in HIV M&E at all levels Lack of funding for HIV M&E Lack of appreciation for HIV M&E M&E systems driven by epidemiologists –lack of systemic focus Inadequate Use of Computer Support for M&E M&E systems driven by epidemiologists –lack of systemic focus Inadequate Institutional M&E Capacity Inadequate M&E Coordination Inadequate M&E Systematization Lack of data use

M&E process Documen- ted M&E strategy 29% Documen- ted key indicators 69% Use of monitoring tools 48% Charting monitoring data 19% Flow from M&E strategy to processed monitoring information: NGOs/FBOs Capturing monitoring data on computer 29%

Problems in current developments Homogenisation not harmonisation Homogenisation not harmonisation National priorities may be contested or too narrowly service-oriented for some National priorities may be contested or too narrowly service-oriented for some Non-funded or separately funded programmes Non-funded or separately funded programmes Value of reporting comes downstream but we want cooperation upstream Value of reporting comes downstream but we want cooperation upstream Decentralised and horizontal access does not seem to be priority Decentralised and horizontal access does not seem to be priority

SUMMARY REFLECTIONS Has been rapid adoption of need for M&E training (with emphasis on monitoring) because ‘fundability’ has been directly connected to ‘evaluability’ Has been rapid adoption of need for M&E training (with emphasis on monitoring) because ‘fundability’ has been directly connected to ‘evaluability’ Concerns with flow of information from periphery to centre, rather than use of information for programme development throughout Concerns with flow of information from periphery to centre, rather than use of information for programme development throughout National concern with key indicators is oriented around reporting to funders rather than programme management and development National concern with key indicators is oriented around reporting to funders rather than programme management and development International indicator sets do not capture many programme outputs, especially in those areas of intervention closely associated with social life (e.g. impact mitigation interventions), as opposed to public health interventions and concerns International indicator sets do not capture many programme outputs, especially in those areas of intervention closely associated with social life (e.g. impact mitigation interventions), as opposed to public health interventions and concerns What about programme reviews, formative evaluation, participatory evaluation? International M&E frameworks in the AIDS field are largely audit-oriented and say little about how M&E can serve programme development. What about programme reviews, formative evaluation, participatory evaluation? International M&E frameworks in the AIDS field are largely audit-oriented and say little about how M&E can serve programme development.

Alternatives and solutions Build monitoring as part of a bottom-up framework for M&EVALUATION Build monitoring as part of a bottom-up framework for M&EVALUATION Start with showing organisations the value of M&E in general through showing the value of methodological reflection Start with showing organisations the value of M&E in general through showing the value of methodological reflection Use umbrella and sectoral organisations as point of entry for capacity development Use umbrella and sectoral organisations as point of entry for capacity development