Presentation is loading. Please wait.

Presentation is loading. Please wait.

M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007.

Similar presentations


Presentation on theme: "M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007."— Presentation transcript:

1 M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007

2 ORIENTATION 2008 UNGASS Country Progress Report Benefits of UNGASS for M&E strengthening - Benefits of UNGASS for M&E strengthening - UNGASS Guidelines Purpose and use Purpose and use Core indicators Core indicators Data sources & data vetting Data sources & data vetting Reporting process Reporting process

3 INTRODUCTION Where does UNGASS fit in M&E system? Where does UNGASS fit in M&E system? UNGASS Reporting Timetable UNGASS Reporting Timetable Use of 2006 UNGASS data Use of 2006 UNGASS data –Global use –Country use

4 WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Benefits of UNGASS reporting for M&E strengthening: Brings partners together around a core set of indicators Helps identify data gaps & data quality issues Stimulates the establishment of a centralised database of HIV indicator data Provides a de facto bi-annual report on the epidemic and the response

5 Program Improvement Program Improvement Reporting/ Accountability [UNGASS, other] Reporting/ Accountability [UNGASS, other] Share Data with Partners Share Data with Partners WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM?

6 UNGASS REPORTING TIMETABLE REPORT REQUIRED? 2003Yes 2004No 2005Interim report from a subset of countries 2006Yes 2007No 2008Yes 2009No 2010Yes

7 GLOBAL USE OF 2006 UNGASS DATA Global Progress Report Report to the Secretary General (as required in the Declaration of Commitment) Used for tracking the Millennium Development Goals

8 COUNTRY USE OF 2006 UNGASS DATA 1. National report on the epidemic and response informs programmatic decisions guides reviews of the National Strategic Plan informs resource mobilisation efforts (e.g. GFATM proposal development) is used as an advocacy tool is used to report progress on Declaration of Commitment to Senior Government Officials

9 COUNTRY USE OF 2006 UNGASS DATA 2. M&E system strengthening convenes relevant partners to address a particular M&E need identifies data gaps assists in planning data collection efforts identifies data weaknesses helps mobilise resources for system strengthening stimulates evaluation questions

10 LESSONS LEARNED FROM 2006 UNGASS REPORTING LESSONS LEARNED FROM 2006 UNGASS REPORTING Challenges in 2006 reporting Challenges in 2006 reporting 2006 reporting from MENA 2006 reporting from MENA Lessons learned from 2006 reporting Lessons learned from 2006 reporting –UNAIDS/Geneva-level –Country-level

11 CHALLENGES IN 2006 REPORTING 137 countries submitted a Country Progress Report 10% reported on all indicators relevant to epidemic 51% used CRIS to report indicator data Some basic data quality issues including –incorrect indicator or incorrect indicator calculation –incomplete data (e.g. only 20% of indicators reported included the required disaggregation) –illogical values –discrepancies between data in CRIS and Report

12 Percent Range and Median of Indicators Reported by Countries by Region - UNGASS 2006 Report

13 IMPORTANCE OF DISAGGREGATED DATA

14 The following inputs are paramount for improving the quality of the data obtained from countries: –Revise UNGASS indicators, where necessary –Provide clear and specific Guidelines –Provide user-friendly, flexible CRIS software –Provide training on UNGASS indicators & reporting process –Provide technical support, where needed LESSONS LEARNED FROM 2006 REPORTING UNAIDS/Geneva-level

15 LESSONS LEARNED FROM 2006 REPORTING Country-level Ensure correct understanding of indicator definition/operands Report all required indicators Identify and document data sources Provide all data and information needed for each indicator Cross-validate data (comparison with previous years, comparison with alternate sources, etc.) Use CRIS for indicator reporting Double-check for data entry errors Submit timely

16 LESSONS LEARNED FROM 2006 REPORTING 1.Address data gaps early 2.Seek to enhance data quality 3.Follow all the steps in the reporting process 4.Double-check accuracy and obtain consensus on all indicator values at country level before submission LESSONS LEARNED FROM 2006 REPORTING Country-level

17 ORIENTATION ON THE UNGASS GUIDELINES Purpose Purpose Target audience Target audience Contents Contents Principles and process of Guidelines revision Principles and process of Guidelines revision Use of Guidelines Use of Guidelines

18

19 PURPOSE Provide essential information on construction of core indicators for reporting Ensure transparency of the reporting process in-country Ensure consistency of information across countries for global progress analysis

20 TARGET AUDIENCES Government sector –including the National AIDS Committee, Ministries (e.g. Ministry of Health, Ministry of Finance, Ministry of Education) Nongovernmental sector –including nongovernmental organizations, faith-based organizations, people living with HIV, private sector UN co-sponsors and donor agencies

21 CONTENTS OVERVIEW Introduction p Implementation at National Level p Core Indicators p Appendices p

22 APPENDICES Changes to Core Indicators 2.Country Progress Report template 3.Methodology used for the coverage survey 4.Consultation/preparation process for the Country Progress Report 5.Reporting schedule for Core Indicators 6.National Funding Matrix National Composite Policy Index (NCPI) Sample check-list for Country Progress Report 9.Selected bibliography

23 PRINCIPLES OF GUIDELINES REVISION Consensus-driven process Transparency Consistency Comprehensiveness Quality Integration within the National M&E system Feasibility

24 Process of Guidelines Revision Debriefing process 2006 Indicator performance M&E Reference Group review Technical working groups for each indicator PROCESS OF GUIDELINES REVISION

25 HOW TO USE THE GUIDELINES Familiarize yourself with the current Guidelines & the changes since the previous round Disseminate the Guidelines and use them to develop a collaborative in-country process Ensure that the correct definitions are used for construction of the indicators & refer to Further Information, as needed Use all Appendices Ask for clarification / assistance, as needed

26 CORE INDICATORS Overview of Core UNGASS Indicators Overview of Core UNGASS Indicators Which UNGASS indicators to report Which UNGASS indicators to report Changes since the 2005 UNGASS Guidelines Changes since the 2005 UNGASS Guidelines Core UNGASS Indicators Core UNGASS Indicators

27 WHAT ARE CORE UNGASS INDICATORS? Quantitative variables which provide simple and reliable ways of measuring progress towards achieving the Declaration of Commitment on HIV/AIDS

28 CORE UNGASS INDICATORS National Indicators (N=25) Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact Global Indicators (N=4)

29 CORE UNGASS INDICATORS: NATIONAL INDICATORS Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact

30 CORE UNGASS INDICATORS: NATIONAL INDICATORS National indicators are important for two reasons: 1.They are used to evaluate the effectiveness of the national response 2.They are used to provide information on regional and global trends

31 Millennium Development Goals (MDG) There are 8 MDGs: –Goal 6: Combat HIV/AIDS, Malaria and other diseases Target: Have halted by 2015 and begun to reverse the spread of HIV/AIDS UNGASS indicators are used to monitor progress MILLENIUM DEVELOPMENT GOALS (MDGs)

32 UNGASS Indicators to monitor MDG-6 UNGASS Indicators to monitor MDG-6 Four of the national indicators are also 12) Current school attendance among orphans and among non-orphans aged ) Percentage of young women and men aged who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission 17) Percentage of women and men aged 15–49 who had more than one sexual partner in the past 12 months who report the use of a condom during their last sexual intercourse 22) Percentage of young women and men aged 15–24 who are HIV infected

33 Global Indicators 1)Amount of bilateral and multilateral financial flows (commitments and disbursements) for the benefit of low- and middle-income countries 3)Percentage of transnational companies that are present in developing countries and that have workplace HIV policies and programmes 4) Percentage of international organizations which have workplace HIV policies and programmes 2)Amount of public funds for research and development of preventive HIV vaccines and microbicides

34 Which UNGASS indicators should countries report? No distinction between Generalized Epidemic Indicator Set & Concentrated /Low-prevalence Epidemic Indicator Set Most national indicators are applicable for all countries Countries are expected to "know their epidemic" Which UNGASS indicators should countries report? Which UNGASS indicators should countries report?

35 Which UNGASS indicators should countries report? When countries choose not to report on a particular indicator, an explanation needs to be provided: Either, The indicator is not applicable to the epidemic The indicator is applicable, but no data is available Which UNGASS indicators should countries report? (continued) Which UNGASS indicators should countries report? (continued)

36 The indicators have to be constructed including the required disaggregations, as outlined in the current UNGASS guidelines

37 Changes since the 2005 UNGASS Guidelines Why changes? To improve the comprehensiveness and quality of data What changes? See Appendix 1

38 Principles used in revisions changes are based on –input received from partners –an analysis of indicator performance in the 2005 reporting round –new programmatic developments every effort was taken to minimize changes every effort was taken to ensure that most countries would be able to collect the data or obtain it from already existing data sources Changes since the 2005 UNGASS Guidelines (continued)

39 Indicators added: 6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV 7) Percentage of women and men aged who received an HIV test in the last 12 months and who know their results Changes since the 2005 UNGASS Guidelines (continued)

40 Indicators removed: -Percentage of large enterprises/companies which have HIV/AIDS workplace policies and programmes -Percentage of women and men with sexually transmitted infections at health care facilities who are appropriately diagnosed, treated and counselled

41 Changes since the 2005 UNGASS Guidelines (continued) Indicator definition changed 1) AIDS spending 3) Blood Safety 5) Prevention of Mother-to-Child Transmission 9)Most-at-risk Populations: Prevention Programmes 11) Life Skills-based HIV Education in Schools 16) Higher-risk Sex 17) Condom Use During Higher-risk Sex

42 Changes since the 2005 UNGASS Guidelines (continued) Age range expanded 4) HIV Treatment: Antiretroviral Therapy Composite indicator divided into its components 20) Injecting Drug Users: Condom Use 21) Injecting Drug Users: Safe Injecting Practices Added questions 2) National Composite Policy Index

43 CORE UNGASS INDICATORS: NATIONAL INDICATORS CORE UNGASS INDICATORS: NATIONAL INDICATORS National Commitment and Action 1)Domestic and International AIDS spending by categories and financing sources 2) National Composite Policy Index (NCPI) Questionnaire divided into two sections: Part A (Government) Strategic plan; Political support; Prevention; Treatment, care & support; Monitoring & Evaluation Part B (Non-government) Human rights; Civil society involvement; Prevention; Treatment, care & support

44 CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes 3) Percentage of donated blood units screened for HIV in a quality-assured manner 4)Percentage of adults and children with advanced HIV infection receiving antiretroviral therapy 5) Percentage of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission 6)Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV

45 CORE UNGASS INDICATORS: NATIONAL INDICATORS CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 7)Percentage of women and men aged who received an HIV test in the last 12 months and who know their results 8)Percentage of most-at-risk populations who received an HIV test in the last 12 months and who know their results 9)Percentage of most-at-risk populations reached with HIV prevention programmes

46 CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 10) Percentage of orphaned and vulnerable children aged 0-17 whose households received free basic external support in caring for the child 11) Percentage of schools that provided life-skills based HIV education within the last academic year

47 CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour 12) Current school attendance among orphans and non-orphans aged 10–14* 13) Percentage of young women and men aged 15–24 who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission* 14) Percentage of most-at-risk populations who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission * Millennium Development Goals indicator

48 CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued) 15) Percentage of young women and men aged who have had sexual intercourse before the age of 15 16) Percentage of women and men aged 15–49 who have had sexual intercourse with more than one partner in the last 12 months 17) Percentage of women and men aged 15–49 who had more than one sexual partner in the past 12 months reporting the use of a condom during their last sexual intercourse* * Millennium Development Goals indicator

49 CORE UNGASS INDICATORS: NATIONAL INDICATORS CORE UNGASS INDICATORS: NATIONAL INDICATORS 19) Percentage of men reporting the use of a condom the last time they had anal sex with a male partner 18) Percentage of female and male sex workers reporting the use of a condom with their most recent client 20) Percentage of injecting drug users reporting the use of a condom the last time they had sexual intercourse 21) Percentage of injecting drug users reporting the use of sterile injecting equipment the last time they injected Knowledge and Behaviour (continued)

50 CORE UNGASS INDICATORS: NATIONAL INDICATORS Impact 22) Percentage of young women and men aged 15–24 who are HIV infected* 23) Percentage of most-at-risk populations who are HIV infected 24) Percentage of adults and children with HIV known to be on treatment 12 months after initiation of antiretroviral therapy 25) Percentage of infants born to HIV-infected mothers who are infected * Millennium Development Goals indicator

51 DATA PROCESS FOR UNGASS REPORTING What steps to include in the data process What steps to include in the data process Why is the data process important Why is the data process important

52 DATA PROCESS FOR UNGASS REPORTING Data collection, vetting, and analysis process including: Identification of relevant tools / sources for data collection for each indicator Timeline for data collection in line with other data collection efforts, including those via funding agencies (e.g. GFATM, PEPFAR) Reporting timeline for facility-based data for national level aggregation Data vetting workshops to reach consensus on the correct value for each indicator

53 DATA PROCESS FOR UNGASS REPORTING (continued) Protocols for data processing and management including: Basic data cleaning and validation One database for analysis and reporting purposes Analysis of relevant data in coordination with partners from government, civil society and the international community - See Appendix 8 Sample Checklist -

54 5. Use of Data 4. Consensus on Data 3. Understanding Data Have data and related information been collected in a timely manner and collated from multiple sources for each indicator? 2. Gathering Data Has a data process for UNGASS reporting with roles and responsibilities been established and shared with all relevant partners? 1. Planning Have the characteristics and quality of the available data been assessed and strengths and limitations understood? Has a collaborative consultation with relevant partners to reach consensus on indicator values taken place and been documented? Have a comprehensive data analysis and findings been included in the Country Progress Report, endorsed by all relevant stakeholders? DATA PROCESS FOR UNGASS REPORTING: the WHAT

55 Completeness, integrity, credibility, and consensus of indicator data is paramount for successful monitoring of the HIV epidemic & for guiding program planning, advocacy, and policy A clear and comprehensive data process, involving all relevant stakeholders, will help ensure steps are taken to obtain the best possible data DATA PROCESS FOR UNGASS REPORTING: the WHY

56 DATA CONSIDERATIONS Data sources Data sources Numerators and Denominators Numerators and Denominators Data disaggregation Data disaggregation Recency and representativeness of data Recency and representativeness of data

57 DATA SOURCES Population-based surveys (e.g. DHS, AIS, MICS) Special surveys (e.g. BSS, 2 nd generation surveillance surveys, school surveys, health facility surveys) Programme monitoring (e.g. patient tracking, Health Information System) Key informants Desk review HIV surveillance / estimates

58 NUMERATORS AND DENOMINATORS Most indicators have a numerator and denominator to calculate a percentage –Follow the instructions exactly It is important that the data collection period is consistent for all the information relevant to a specific indicators numerator and denominator

59 DATA DISAGGREGATION Countries are expected to submit –total values AND –all disaggregated values as per the instructions in the Guidelines Data collected through facilities, programme monitoring, and surveys is often disaggregated; the challenge is to ensure the disaggregated data is also available at national level

60 DATA DISAGGREGATION (continued) Why important? Allows for more effective tracking of resources and programmatic responses Allows for monitoring trends in specified sub- populations Allows for monitoring equity of access to services

61 IN-COUNTRY REPORTING PROCESS In-country reporting process In-country reporting process Country Progress Report Format Country Progress Report Format Submission Process Submission Process Role of UNAIDS/Geneva Role of UNAIDS/Geneva

62 RECENCY & REPRESENTATIVENESS Which data to submit? Ideally, most recent data available nationally representative data But, if data is not representative, most recent survey data which has been reviewed and endorsed by technical experts within the country

63 STEPS FOR IN-COUNTRY UNGASS REPORTING PROCESS Submission Preparation Planning Implementation

64 ROLES & RESPONSIBILITIES The overall responsibility of preparing and submitting the Country Progress Report lies with national governments (NAC or equivalent) UNGASS should be routine and integrated in the countrys M&E plans, budgets and systems

65 PREPARATION 1. Assess country readiness 2. Identify stakeholders 3. Identify funding 4. Identify relevant data tools / sources Preparation

66 ASSESS COUNTRY READINESS Countries are expected to know their epidemic to determine which of the UNGASS indicators are applicable to the countrys epidemiological context to know availability of data Preparation

67 IDENTIFY STAKEHOLDERS Individuals and organizations who are affected by HIV and/or are involved in the HIV response: Government (including NAC, Ministry of Health, Ministry of Education; Ministry of Finance) Civil society groups (including nongovernmental organizations, people living with HIV) Private sector Preparation

68 IMPORTANCE OF CIVIL SOCIETY Strategic and tactical expertise Quantitative and qualitative data Valuable perspective on the data Preparation

69 HOW AND WHEN TO INCLUDE CIVIL SOCIETY Involvement in all phases of the report preparation: Identify relevant civil society organizations Assign roles and responsibility Plan with them data collection and analysis Invite them to participate in workshops Share drafts with them for review and comments Disseminate final report to them Preparation

70 IDENTIFY SOURCE OF FUNDING What sources are available? No separate funding available for UNGASS reporting, but should be integrated in M&E budgets (government, GFTAM, World bank) Preparation

71 DATA SOURCES Population-based surveys (e.g. DHS, AIS, MICS) Special surveys (e.g. BSS, 2 nd generation surveillance surveys, school surveys, health facility surveys) Programme monitoring (e.g. patient tracking, Health Information System) Key informants Desk review HIV surveillance / estimates Preparation

72 DATA CONSIDERATIONS Recency and representativeness of data Requirements for disaggregation Appropriate data process Preparation

73 PLANNING Develop a plan for data collection / collation, analysis and report writing Designate a coordinator Assign roles and responsibilities for each indicator Assign realistic timelines and milestones Secure funding for the entire process (collecting, collating, analysing and reporting the data) Planning

74 PLANNING (continued) Disseminate the plan Use the plan as a checklist to monitor progress against the timeline and milestones Planning

75 ACTION PLAN TEMPLATE Steps Activities Timeline Budget Responsible person Stakeholders involved Planning

76 IMPLEMENTATION Establish a process for data collecting / collating, vetting and analysis Organise workshops for vetting and triangulation of data Complete the appropriate data forms Draft the Country Progress Report Disseminate it for review and comment Collate comments from stakeholders, including government agencies and civil society Implementation

77 Organize a national consultation workshop to reach consensus with stakeholders on the final report Enter agreed data into CRIS (or equivalent data management system) Validate the indicator data against the report narrative (consistency check) Finalize and submit the report to UNAIDS Identify focal point for communication between UNAIDS Secretariat and country IMPLEMENTATION (continued) Implementation

78 MONITOR PROGRESS & DEBRIEF Monitor progress throughout the implementation process using the action plan Plan a debriefing meeting –to assess success of report (timeliness, comprehensiveness, quality) –to agree actions to undertake to strengthen the national M&E system

79 OVERVIEW OF THE REPORTING PROCESS Identify data needs, data sources, stakeholders, fundsEstablish plan for data collection, analysis and reporting writingSecure funds, collect/collate and analyse data, complete data formsDraft Country Progress Report, share draft with stakeholders Enter data in CRIS Validate the narrative report against the data in CRIS Reach consensus with stakeholders on final Report & submit timely

80 FORMAT OF THE COUNTRY PROGRESS REPORT Narrative part (see Appendix 2) –Structured to ensure consistent information across countries –Provide an in-depth analysis of data regarding the HIV epidemic and response Submission of report

81 UNGASS COUNTRY PROGRESS REPORT [Country Name] Reporting period: January 2006–December 2007 Submission date: […] Submission date: […] I.Table of Contents II.Status at a glance III.Overview of the AIDS epidemic IV.National response to the AIDS epidemic V. Best practices VI.Major challenges and remedial actions VII.Support from countrys development partners VIII.Monitoring and evaluation environment

82 ANNEXES ANNEX 1 Consultation/preparation process for the Country Progress Report ANNEX 2 National Composite Policy Index Questionnaire Submission of report

83 COUNTRY PROGRESS REPORT Indicator Data COUNTRY PROGRESS REPORT Indicator Data

84 BENEFITS OF USING THE COUNTRY RESPONSE INFORMATION SYSTEM (CRIS) CRIS is a monitoring system which greatly facilitates UNGASS reporting: It supports all six UN languages [no need for translation] It uses the current Guidelines [no confusion] It uses the correct indicator definitions [no confusion] It facilitates compilation and management of all UNGASS data [saves time] Submission of report

85 BENEFITS OF USING CRIS (continued) It has built-in validity checks [avoids logical errors] It stores available data from population-based surveys [saves time] It facilitates trend analysis as previous UNGASS data is stored [saves time] Data is seamlessly integrated into the global database [no transcription errors, no delays] Submission of report

86 BENEFITS OF USING CRIS (continued) In addition to use for UNGASS reporting and reporting to donors CRIS can be used for the management of all monitoring information at the national and sub-national levels Submission of report

87 SUBMISSION PROCESS What? Country Progress Report consisting of –Narrative section (including Annex 1, Annex 2) –CRIS data file (or excel data file, see template on CD) When? –15 January 2008 if not using CRIS –31 January 2008 if using CRIS [preferred] How? to Submission of report

88 WHAT HAPPENS AT UNAIDS/GENEVA? UNAIDS and its partners are responsible for calculating the global indicators A process for collating, analysing and reporting data in coordination with co-sponsors and global partners, including civil society occurs

89 WHAT HAPPENS AT UNAIDS/GENEVA? Country Report Findings + Global indicators = Report on the Global AIDS Epidemic

90 WHAT HAPPENS AT UNAIDS/GENEVA? The Report on the Global AIDS Epidemic is prepared by May 2008 and submitted to the United Nations General Assembly The Country Progress Reports and the Global Report will be available on the UNAIDS website at

91 BENEFITS OF UNGASS REPORTING Provides answers to important questions: What is the status of the epidemic in the country? What are the basic trends in HIV transmission and service coverage? What are the main obstacles to accessing HIV prevention, care and treatment services? What is the quality of services being delivered?

92 BENEFITS OF UNGASS REPORTING (continued) Provides answers to important questions: Are services being delivered equitably and effectively? What exacerbates these problems? (e.g. policies, laws, resources, politics, customs, organizations, individuals) Who can change this situation? (e.g. elected leaders, bureaucrats, religious leaders, community leaders, traditional leaders, donors, international organizations, NGOs) What are these people/organizations currently doing to address the problems?

93 BENEFITS OF UNGASS REPORTING (continued) This information can be used to evaluate the effectiveness of the national response analyse linkages between policy, implementation of programmes, behavioural change and HIV prevalence identify gaps in programmes, personnel, technology identify data gaps and data quality issues strengthen the National M&E system

94 SUMMARY Ownership & participation Consultation & consensus Data vetting, validation, triangulation Timeliness: 31 January 2008 or 15 January 2008 if not submitted in CRIS Focus on systems and process


Download ppt "M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007."

Similar presentations


Ads by Google