1. Building New Institutional Capacity in M&E: The Experience of National AIDS Coordinating Authority V F Kemerer Getting To Results IDEAS Global Assembly Johannesburg, South Africa 18 March 2009

2. Overview  Background  Use of Assessment Tools for Strategic Planning  NACA Specific Capacity Building Interventions  Measurement and Determination of Effectiveness and Outcomes

3. Background System - Assessments, development of operational plans, and implementation of selected interventions Organization - Reinforce key management systems and use of proven leadership practices through a cadre of tools Individual - On-the-job mentoring & coaching, workshops, M&E Master’s Programs, distance learning

4. Background National Agencies USG & PEPFAR GFATM UN Family Misc Bi- & Multi- Laterals Coordinate the National HIV DMRS Lead National M&E System Development NAC M&E UNIT CB through TA using various tools & strategies to establish and/or improve performance

5. Strategic Vision: Build effective and sustainable M&E systems to provide comprehensive, high- quality information Reduce constraints to data use and increase ownership of information Technical capacity Poor understanding of existing data Organizational cultural

6. Use of Assessment Tools for Strategic Planning  SI Assessment  MESST  DQA/RDQA

7. SI Assessment Activities  Map national HIV output indicators and identify data flow  Outline challenges in the DMRS  Assess M&E structures  Stakeholder Analysis – National M&E Framework  Stakeholder Analysis –HIV researchers, organisations, and research

8. Linking SI Capacity and SI Planning with SI Products Timely, high quality strategic information products Unified SI Strategic Planning and Coordinated SI Support SI Systems Assessments Ongoing Monitoring & Feedback Building of National SI Systems

9. Use of Assessment Tools for Strategic Planning  SI Assessment  MESST  DQA/RDQA

10. 10 A new approach to M&E Assessments Donor-drivenCountry-driven Focused exclusively on capacities of specific silos Expanded to integration in the wider M&E Systems General approach More systematic approach (based on a checklist of items) Strictly Assessment Tool Broader Assessment, Management, Planning & Capacity Building Tool FROM …TO …

11. 11 How does the M&E Strengthening Tool work? Step 1: Diagnosis Step 3: Action Plan Development of a Costed Action plan with: ■Identification of Strengths and Weaknesses ■Defining Strengthening Measures: - Responsibility - Timeline - Funding Implication - T/A Needs Diagnosis through completion of a Checklist: ■Answers to Statements/ Questions ■Provision of Comments/Expla nations Step 2: Analysis A Summary Dashboard will be automatically generated : ■Displays the overall distribution of answers for each of the checklists’ sections ■Provides a more detailed analysis of the specific gaps that helps respondents complete the ACTION PLAN

12. 12 Expected Benefits ■Better identify M&E capacity gaps and corresponding strengthening measures, including through Technical Assistance (TA) ■Ensure that such investments contribute to the strengthening of the National Systems (avoiding parallel reporting systems) ■Guide investments in M&E (within the recommended range of 5%-10% of the overall Budget) ■Enhance the quality of programmatic data to improve program management

13. Use of Assessment Tools for Strategic Planning  SI Assessment  DDIU Tools  MESST  DQA/RDQA

14. 14 DQA METHODOLOGY – 2 Protocols ■The methodology for the DQA includes two (2) protocols: Data Verifications (Protocol 1) Quantitative comparison of recounted to reported data and review of timeliness, completeness and availability of reports. 1 Assessment of Data Management Systems (Protocol 2) Qualitative assessment of the strengths and weaknesses of the data-collection and reporting system. 2

15. 15 PROTOCOL 2: Assessment of Data Management and Reporting Systems M&E Management Unit Service Delivery Sites / Organizations Intermediate Aggregation levels (eg. District, Region) 3.Assess Data Management and Reporting Systems (Part 1) ■PURPOSE: Identify potential risks to data quality created by the data- management and reporting systems at: -the M&E Management Unit; -the Service Delivery Points; -any Intermediary Aggregation Level (District or Region). ■The DQA assesses both (1) the design; and (2) the implementation of the data- management and reporting systems. ■The assessment covers 8 functional areas (HR, Training, Data Management Processes, etc.)

16. 16 Objective of the Data Quality Assessment Tool The Data-Quality Assessment (DQA) Tool is designed to:  assess the ability of data-management systems to collect manage, and report quality data.  verify the quality of reported data for key indicators at selected sites; and The DQA Tool is not intended to assess the entire M&E system; it relates and focuses more in depth into component 10 (i.e. Supportive supervision and data auditing) of the “Organizing Framework for a Functional National HIV M&E System”. !

17. NACA Specific Capacity Building Interventions MESST RDQA TA for DQ SI Assessment M&E Road Map 1/4ly Mentoring (SHAPMoS)

18. CB INTERVENTIONS

19. SI Assessment M&E Road Map Implementation  Incorporation of Data Flow Findings into SHAPMoS  Incorporation of the Institutional Inventory List into:  National Programmatic Database based at the NACA  Supporting primary efforts of WHO’s P-SAM Exercise  Participation in the Launch of the National HIV M&E Strategy

20. SI Assessment M&E Road Map Implementation  ToT M&E Basic Concepts  4 quarterly mentoring & coaching visits to Regional M&E Officers  Support to the NACA to produce QSCR

21. Institutional Capacity Strengthened – Regional M&E Officers  Internal monitoring systems developed and in place  Trace and verification conducted in each Region at SDPs  Regional QSCRs produced  Information products developed for feedback to SDPs

22. CB INTERVENTIONS

23. Timeline: TA to Malawi 20072008 0102030405060708091011120102030405 DataQuality Plan for MESST MESST:Malaria MESST:HIV Discussions with NAC: Next Steps Plan for DQ T/A M&E Plan Revision

24. RDQA Tool To Operationalize Statement of Work  Support finalization of the Draft Data Quality Protocol  Assist to roll-out the data quality protocol  Conduct data quality training(s) for key implementing agencies Draft Services & Indicators Matrix Draft register(s) to be used by all CBO SDPs Draft clear instructions Draft DQ policies Draft data steps performed at each level of the DMRS Draft guidelines for quality controls Draft written back-up procedures Data Quality Training in 5 Zones

25. Institutional Capacity to Support Data Quality Developed  Data quality plan written that includes guidance, reporting forms, and indicator matrix  Key stakeholders at national and sub-national levels oriented to – and reviewed – new data quality documents  NACA M&E Officers trained to train on data quality  Key staff in all 5 Zones have been trained by NACA on data quality

26. Conclusions  NACA M&E Unit institutional capacity strengthened at National and Sub-National levels  Data reporting more timely, with increasingly accurate, reliable and complete data  Foundation for effective data use laid  Documented systems establish institutional memory

27. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GPO-A-00-03- 00003-00 and is implemented by the Carolina Population Center at the University of North Carolina in partnership with Futures Group, John Snow, Inc., Management Sciences for Health, ORC Macro International, and Tulane University. Visit us online at http://www.cpc.unc.edu/measure.