1. Global Trends in HIV/AIDS Monitoring and Evaluation Deborah Rugg, PhD Associate Director for Monitoring and Evaluation HHS/US Centers for Disease Control and Prevention (CDC) Global AIDS Program

2. Overview  Background on HIV/AIDS M&E  HIV/AIDS M&E at the National Level  Trends in Global HIV/AIDS M&E –The need for collaboration –UNGASS reporting –Monitoring and Evaluation Reference Group (MERG) –The "3 Ones" Principle  Background on HIV/AIDS M&E  HIV/AIDS M&E at the National Level  Trends in Global HIV/AIDS M&E –The need for collaboration –UNGASS reporting –Monitoring and Evaluation Reference Group (MERG) –The "3 Ones" Principle

3. BACKGROUND ON HIV/AIDS M&E

4. Monitoring and Evaluation for Program Improvement Program Improvement Program Improvement Reporting/ Accountability Reporting/ Accountability Share Data with Partners Share Data with Partners

5. What is the problem? ● Situation Analysis and Surveillance What are the contributing factors? ● Determinants Research What interventions and resources are needed? ● Needs, Resource, Response Analysis & Input Monitoring What interventions can work (efficacy & effectiveness)? ● Special studies, Operations res., Formative res. & Research synthesis Are we implementing the program as planned? ● Outputs Monitoring What are we doing? ● Process Monitoring & Evaluation, Quality Assessments Are interventions working/making a difference? ● Outcome Evaluation Studies Are collective efforts being implemented on a large enough scale to impact the epidemic (coverage; impact)? ● Surveys & Surveillance Understanding Potential Responses Monitoring & Evaluating National Programs Determining Collective Effectiveness ACTIVITIES OUTPUTS INPUTS OUTCOMES & IMPACTS MONITORING Problem Identification A Public Health Questions Approach to Unifying SI/M&E Are we doing the right things? Are we doing them right? Are we doing them on a large enough scale?

6. Most Some Few* All Monitoring and Evaluation Pipeline Input /Output Monitoring Process Evaluation Outcome Monitoring/ Evaluation Impact Monitoring/ Evaluation Levels of Monitoring & Evaluation Effort # of Projects # of Projects * Supplemented with impact indicators from surveillance data. Strategic Planning for M&E: Setting Realistic Expectations

7. Project Level Indicators Country Level Indicators Global Level Indicators M&E Indicator Pyramid: Levels of Indicators

8. HIV/AIDS M&E AT THE NATIONAL LEVEL

9. Basic Outline for a National M&E Plan  Introduction- overview of programs or interventions  Background Information: e.g., M&E resources—financial, human, other; roles and responsibilities  Logic Model/ Results Framework –Problem statement –Expected outcomes/impacts –Indicators –Multi-year targets (measurable objectives)  Operational definitions, sources, frequency of indicator data, method of verification/validation  Inclusion of plans for special evaluation studies  Introduction- overview of programs or interventions  Background Information: e.g., M&E resources—financial, human, other; roles and responsibilities  Logic Model/ Results Framework –Problem statement –Expected outcomes/impacts –Indicators –Multi-year targets (measurable objectives)  Operational definitions, sources, frequency of indicator data, method of verification/validation  Inclusion of plans for special evaluation studies

10. Multi-agency M&E Logic Model Intermediate Outcomes Long-Term Impacts OutputsShort-Term Outcomes Other inputs USG inputs World Bank inputs Government inputs Program NAC/NAP Program Adapted from Milstein & Kreuter. A Summary Outline of Logic Models: What are They and What Can They Do for Planning and Evaluation? CDC 2000

11. Fundamentals of M&E Planning at the National Level  National governments must believe in the value of M&E  Donors’ / development partners’ external assistance efforts are aligned with overall national or local strategies  Donor / partner funding is part of overall development funding  Effective coordination mechanisms between partners are essential  Transparency, trust and consultation between partners are essential  National governments must believe in the value of M&E  Donors’ / development partners’ external assistance efforts are aligned with overall national or local strategies  Donor / partner funding is part of overall development funding  Effective coordination mechanisms between partners are essential  Transparency, trust and consultation between partners are essential

12. TRENDS IN GLOBAL HIV/AIDS M&E –Current status of HIV/AIDS M&E –UNGASS reporting –Monitoring and Evaluation Reference Group (MERG) –The "3 Ones" Principle TRENDS IN GLOBAL HIV/AIDS M&E –Current status of HIV/AIDS M&E –UNGASS reporting –Monitoring and Evaluation Reference Group (MERG) –The "3 Ones" Principle

13.  Long-term approach to development planning and funding by key players such as government, donors  Linking of short, medium and long-term frameworks and strategies - including to budgets  M&E systems need to encompass much more complex frameworks and environments  Donor fatigue too many reports, too many terms, too little feedback, too little ownership of interventions  We spend lots of time on indicators, but Evaluation is often neglected; we need to strengthen Evaluation to better understand our programs  Inadequate analysis of results / understanding of what we are actually doing and what is working or is not working –, synthesis of what we learn from M&E, and adapting program practice accordingly  Need for harmonized and yet manageable M&E data systems  Long-term approach to development planning and funding by key players such as government, donors  Linking of short, medium and long-term frameworks and strategies - including to budgets  M&E systems need to encompass much more complex frameworks and environments  Donor fatigue too many reports, too many terms, too little feedback, too little ownership of interventions  We spend lots of time on indicators, but Evaluation is often neglected; we need to strengthen Evaluation to better understand our programs  Inadequate analysis of results / understanding of what we are actually doing and what is working or is not working –, synthesis of what we learn from M&E, and adapting program practice accordingly  Need for harmonized and yet manageable M&E data systems Current status and Challenges of HIV/AIDS M&E

14.  Prior to this UNGASS report, we only compared country data on HIV prevalence  Now there are standardized indicators for policies, funding, services, coverage and risk reduction  Data coming directly from over 100 countries  70% of reports involved civil society, 50% involved people living with HIV/AIDS  Prior to this UNGASS report, we only compared country data on HIV prevalence  Now there are standardized indicators for policies, funding, services, coverage and risk reduction  Data coming directly from over 100 countries  70% of reports involved civil society, 50% involved people living with HIV/AIDS United Nations General Assembly Special Session on AIDS (UNGASS)* *UNGASS Information courtesy Paul De Lay, Director for Evaluation, UNAIDS/Geneva (2004)

15. 2003 UNGASS Survey  103 countries responded out of 189  Progress seen in political commitment, improved policies, prevention efforts  Insufficient progress in human rights, human capacity, financial resources  103 countries responded out of 189  Progress seen in political commitment, improved policies, prevention efforts  Insufficient progress in human rights, human capacity, financial resources

17. Challenges/Issues  Indicators are mainly for generalized epidemics  No indicators for blood safety and infections in hospitals  Indicators for IDU and behavior change in youth need improvement  No indicators for sex workers  Few countries could report on quality of STI treatment  Indicators are mainly for generalized epidemics  No indicators for blood safety and infections in hospitals  Indicators for IDU and behavior change in youth need improvement  No indicators for sex workers  Few countries could report on quality of STI treatment

18.  UNAIDS established and international M&E standards setting group  Members from all UN co-sponsors and international agencies  Develops M&E strategy guidelines and international indicators as well as coordinates international M&E Technical Assistance activities  Meets annually (with sub-committees meeting more frequently) and generates M&E documents and other reports available on the UNAIDS website  Involved in monitoring the implementation of the “ Three Ones Principles”  UNAIDS established and international M&E standards setting group  Members from all UN co-sponsors and international agencies  Develops M&E strategy guidelines and international indicators as well as coordinates international M&E Technical Assistance activities  Meets annually (with sub-committees meeting more frequently) and generates M&E documents and other reports available on the UNAIDS website  Involved in monitoring the implementation of the “ Three Ones Principles” Monitoring and Evaluation Reference Group (MERG)

19. The ‘Three Ones’ Key Principles 1.One agreed HIV and AIDS action framework 2.One national AIDS coordinating authority 3.One agreed monitoring and evaluation framework 1.One agreed HIV and AIDS action framework 2.One national AIDS coordinating authority 3.One agreed monitoring and evaluation framework

20. The Third “One” Why is it better?  Data based on national needs rather than individual donors  Production of high quality, relevant, accurate and timely data  Submission of reports to international bodies under a unified global effort  Efficient and effective use of data and resources  Allows synthesis of data from multiple sources  Greater transparency, coordination and communication among different partners.  Data based on national needs rather than individual donors  Production of high quality, relevant, accurate and timely data  Submission of reports to international bodies under a unified global effort  Efficient and effective use of data and resources  Allows synthesis of data from multiple sources  Greater transparency, coordination and communication among different partners.

21. Principles for agreement  One M&E unit which coordinates M&E activities  One multisectoral M&E plan  One national set of standardized indicators  One national level information system  Strategic information flow from sub-national to national level  Harmonized M&E capacity building  Collective responsibility and collective achievement (attribution)  One M&E unit which coordinates M&E activities  One multisectoral M&E plan  One national set of standardized indicators  One national level information system  Strategic information flow from sub-national to national level  Harmonized M&E capacity building  Collective responsibility and collective achievement (attribution)

22. Why isn’t available data used better?  Data collection is fragmented  No single unit is responsible for compiling, analyzing and presenting data in a cohesive whole  No budgets for analyzing and presenting data  Underestimate skills and cost needed to present data effectively  Most M&E persons do not know how to use data well  Data collection is fragmented  No single unit is responsible for compiling, analyzing and presenting data in a cohesive whole  No budgets for analyzing and presenting data  Underestimate skills and cost needed to present data effectively  Most M&E persons do not know how to use data well

23. Key categories of information  Biologic surveillance  Policy environment  Behavioral surveillance  Resource flows data Tracking commodities  Provision of prevention and treatment services and the coverage of these services  Mortality and morbidity data  General health service performance  Evaluation research  Biologic surveillance  Policy environment  Behavioral surveillance  Resource flows data Tracking commodities  Provision of prevention and treatment services and the coverage of these services  Mortality and morbidity data  General health service performance  Evaluation research

24. Challenges for Monitoring and Evaluating ART  Need for short-term indicators (first two years of program implementation) –3-6 month intervals –Including equity of access –Survival at 6, 12, and 24 month time periods  Need to monitor long-term sustainability –Resource flow tracking (sources, cost per unit services, costs per commodities) –Staffing patterns –Facilities capacity –Systems infrastructure capacity  Need to monitor long-term impact –ARV resistance patterns –5 to 10 year survival and quality of life –System wide impact –Impact on incidence –Impact on economic productivity and social sector services  Need for short-term indicators (first two years of program implementation) –3-6 month intervals –Including equity of access –Survival at 6, 12, and 24 month time periods  Need to monitor long-term sustainability –Resource flow tracking (sources, cost per unit services, costs per commodities) –Staffing patterns –Facilities capacity –Systems infrastructure capacity  Need to monitor long-term impact –ARV resistance patterns –5 to 10 year survival and quality of life –System wide impact –Impact on incidence –Impact on economic productivity and social sector services

25. We need to mainstream M&E at all levels- district, state, provincial, national, and global. We need to do this in a credible way that includes primary users and focuses on outcomes that are meaningful to people on the front lines. Conclusions

26. In building on each other’s strengths, we must identify incentives and opportunities for collaboration, with the fundamental consensus that working together in a harmonized manner is better than going at it alone. Conclusions